Java Ninja Chronicles By Norris Shelton

Things I learned in the pursuit of code

Tomcat access logs are not enabled in Spring Boot by default. The access logs can be a very useful troubleshooting tool. The logs can be enabled by adding a property to the application.properties in the src/main/resources directory.

# Enable access log.
server.tomcat.accesslog.enabled=true

This will enable Tomcat Access Logging. The logs will go to the current temporary directory where the existing Tomcat work directory is located. The logs will appear under a directory named logs in a file that follows the normal Tomcat Access logs, such as access_log.2019-02-04.log.

It is a good idea to specify the directory that you want to contain the Tomcat logs, etc instead of relying on them being sent to a temporary directory. You can specify the Tomcat Basedir by adding the following to the application.properties.

# Tomcat base directory, which will be at the same level as the 
# Spring Boot jar. If not specified, a temporary directory
# is used.
server.tomcat.basedir=tomcat

The directory with the specified name will be created at the same directory level as the .jar that contains the Spring Boot application. If you are running the application as part of your IDE, then the specified directory will appear in the root of your source code.

An alternative way to specify the properties is to specify them as JVM parameters. An example of both of these would be

-Dserver.tomcat.basedir=tomcat -Dserver.tomcat.accesslog.enabled=true

Tomcat Access Logs can be easily enabled in a Spring Boot application by setting the appropriate property. This allows easy troubleshooting and if you set the Tomcat basedir, you can also access Tomcat’s work directory.

Additional information about configuring Spring Boot Tomcat can be found at https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#howto-configure-accesslogs

February 5th, 2019

Posted In: Javaninja

Tags: , , , ,

Leave a Comment

We’ve all been there. Hot deploys are so convenient, but they consume memory via ClassLoader leakage. You eventually run out of memory and your server dies.

java.lang.OutOfMemoryError: PermGen space

Tomcat leak prevention

Tomcat helped us a lot when it came out with it’s memory leak prevention. This has been our mainstay tool since Tomcat 6. At times, even this wasn’t enough to prevent dreaded permgen errors. http://wiki.apache.org/tomcat/MemoryLeakProtection

We have all seen a log message similar to the following, thanks to Tomcat:

02-Dec-2015 14:50:41.197 WARNING [RMI TCP Connection(6)-127.0.0.1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [bonus-services] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

Java 8 says goodbye to PermGen

Java 8 helped a lot by removing the limits of permgen by opening up all of the meta space. This wasn’t as much a fix as it was to make the memory container much larger. Even this has it’s limits.

java.lang.OutOfMemoryError: Metadata space

Mattias Jiderhamn’s ClassLoader Leak Prevention library

Enter the https://github.com/mjiderhamn/classloader-leak-prevention library. This does many of the things that Tomcat does along with some others. It’s uses an Apache 2 license. There is a Servlet Context Listener that listens for the context creation and context destruction. This allows it to perform it’s work, all for your benefit.

It is very easy to integrate into your webapp. The first step is to add the Maven dependency.

<dependency>
    <groupId>se.jiderhamn</groupId>
    <artifactId>classloader-leak-prevention</artifactId>
    <version>1.15.2</version>
</dependency>

Then you need to add the listener as the first listener in your web.xml.

<listener>
    <description>https://github.com/mjiderhamn/classloader-leak-prevention</description>
    <listener-class>se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor</listener-class>
</listener>

When your webapp starts up, you will see the following as logging:

ClassLoaderLeakPreventor: Settings for se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor (CL: 0xcb6c98f):
ClassLoaderLeakPreventor:   stopThreads = true
ClassLoaderLeakPreventor:   stopTimerThreads = true
ClassLoaderLeakPreventor:   executeShutdownHooks = true
ClassLoaderLeakPreventor:   threadWaitMs = 5000 ms
ClassLoaderLeakPreventor:   shutdownHookWaitMs = 10000 ms
ClassLoaderLeakPreventor: Initializing context by loading some known offenders with system classloader

When you hot deploy your webapp, you will see logging similar to the following:

ClassLoaderLeakPreventor: se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor shutting down context by removing known leaks (CL: 0xcb6c98f)
ClassLoaderLeakPreventor: Looping 5 RMI Targets to find leaks
ClassLoaderLeakPreventor: Looping 5 RMI Targets to find leaks
ClassLoaderLeakPreventor: Internal registry of java.beans.PropertyEditorManager not found
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Prototype beans currently in creation with value null will be made stale for later expunging from Thread[http-nio-8080-exec-1,5,main]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Transactional resources with value null will be made stale for later expunging from Thread[http-nio-8080-exec-1,5,main]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Transaction synchronizations with value null will be made stale for later expunging from Thread[http-nio-8080-exec-1,5,main]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Prototype beans currently in creation with value null will be made stale for later expunging from Thread[http-nio-8080-exec-1,5,main]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Transactional resources with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Transaction synchronizations with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Custom ThreadLocal of type org.springframework.core.NamedThreadLocal: Prototype beans currently in creation with value null will be made stale for later expunging from Thread[RMI TCP Connection(idle),5,RMI Runtime]
ClassLoaderLeakPreventor: Since Java 1.6+ is used, we can call public static final void java.util.ResourceBundle.clearCache(java.lang.ClassLoader)
ClassLoaderLeakPreventor: Releasing web app classloader from Apache Commons Logging

Notice that it is using System.out to log because Logging libraries are common causes of leaked ClassLoaders.

JVM settings

I picked this up from Michael Barnes. You will need to specify the following to make the garbage collector work correctly.

-XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
  • -XX:+UseG1GC – tells the JVM to use the Garbage First Garbage Collector. This is a new GC for Java 8.
  • -XX:+CMSClassUnloadingEnabled – Tells the GC that class unloading is enabled.
  • -XX:+CMSPermGenSweepingEnabled – Tells the GC that it should enable permanent generation sweeping. Note, this the JVM will give you a message stating that this is unnecessary. Ignore the message, because it will not work without it.

Note that two of the settings are for the Concurrent Mark and Sweep garbage collector. It isn’t documented, but these flags do indeed work.

December 3rd, 2015

Posted In: Java, java ninja, Javaninja

Tags: , , , , , , , , ,

Leave a Comment

The Servlet 3.1 API is retrieved from maven with the following dependency:

        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.1.0</version>
            <scope>provided</scope>
        </dependency>

That is the for API, meaning the specification. If you are doing something and you need an implementation, then you need to grab the implementation for the specific container you are going to be running. The location of the Tomcat 8 implementation is below:

<dependency>
	<groupId>org.apache.tomcat</groupId>
	<artifactId>tomcat-servlet-api</artifactId>
	<version>8.0.26</version>
</dependency>

September 11th, 2015

Posted In: Java, java ninja, Javaninja

Tags: , , , , ,

Leave a Comment

Tomcat is an implementation of the Servlet API. Tomcat 8 is an implementation of the Servlet 3.1 API. The following is an example of the Servlet 3.1 deployment descriptor (web.xml).

<web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd"
         version="3.1">
</web-app>

September 11th, 2015

Posted In: Java, java ninja, Javaninja

Tags: , , , , , , ,

Leave a Comment

WP to LinkedIn Auto Publish Powered By : XYZScripts.com