Out of Memory

Busting PermGen Myths

In my latest post I explained the reasons that can cause the java.lang.OutOfMemoryError: PermGen space crashes. Now it is time to talk about possible solutions to the problem. Or, more precisely, about what the Internet suggests for possible solutions. Unfortunately, I can only say that I felt my inner Jamie Hyneman from MythBusters awakening when going through the different "expert opinions" on the subject.

I googled for current common knowledge about ways to solve java.lang.OutOfMemoryError: PermGen space crashes and went through a couple dozen pages that seemed more appropriate in Google results. Fortunately, most of the suggestions have already been distilled into this topic of the very respected StackOverflow. As you can see, the topic is truly popular and has some quite highly voted answers. But the irony is that the whole topic contains exactly zero solutions I could recommend myself. Well, aside from “Find the cause of memory leak”, which is absolutely correct, of course, but not very helpful way to respond to the question “How to solve memory leak”. Let us review the suggestions put forward on the SO page.

Use -XX:MaxPermSize=XXXM

There can be two reasons that cause the java.lang.OutOfMemoryError: PermGen space error.

One is that application server and/or application really does use so many classes that they do not fit into default sized Permanent Generation. It is definitely possible and not that rare in fact. In this case increasing the size of Permanent Generation can really save the day. If your only problem is how to fit too many furniture into too small house, then buy the bigger house!

But what if your over-caring mother sends you new furniture every week? You cannot possibly continue to move to the bigger houses over and over again. That is exactly the situation with memory leaks - and also with the classloader leaks, as described in my previous post that I mentioned above. Let me be clear here: no increase in Permanent Generation size will save you from the classloader leak. It can only postpone it. And make it harder to predict how many re-deployments your server will outlive.

-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled

The most popular answer on StackOverflow was to add these options to the server’s command line. And, they say, "maybe add -XX:+UseConcMarkSweepGC too. Just to be sure". My first problem with these JVM flags is that there is no explanation available of what they really do. Neither in the SO answer (and I don't like answers that tell you to do something without the reasoning why you should do it), nor actually in the whole Internet.

Really, I was unable to find any documentation about these options, except for this page. But, in fact, that does not even matter. In no way any tinkering with the Garbage Collector options will help you in case of a classloder leak. Because, by definition, a memory leak is a situation where GC falls short. If there is a valid live hard reference from somewhere within your server’s classloader to an object or class of your application, then the GC will never think of it as garbage and will never reclaim it. Sure, all these JVM flags look very smart and magical. And they really may be required in some situations. But they are certainly notsufficient and don’t solve your Permanent Generation leak.

Use JRockit

The next proposition was to switch to the JRockit JVM. The rationale was that as JRockit has no Permanent Generation, one cannot run out of it. Surely, an interesting proposition. Unfortunately, it will not solve our problem either.

The only result of this “solution” will be getting a java.lang.OutOfMemoryError: Java heap space instead of the java.lang.OutOfMemoryError: PermGen space. In the absence of separate generation for class definitions, JRockit uses the usual Java heap space for them. And as long the root cause of the leak is not fixed, those class definitions will fill up even the largest heap, given enough time.

Restart the server

Yet another way to pretend that the problem is solved, is to restart the application server from time to time. E.g. instead of redeploying the application, just restart the whole server. But the first time you see an application server with more than one application deployed, you will know that this is rarely possible in production environment. And this is not really a solution. It is a way to hide your head in the sand.

Use Tomcat

This one is actually not that hopeless as the previous ones - recent Tomcat versions really do try to solve classloader leaks. See for yourself in their documentation. IF you can use Tomcat as your target server, and IF your leak is one of those Tomcat can successfully fight against, then maybe, just maybe, you are lucky and the problem is solved for you.

Use <Your favorite profiler tool here>

May be a viable solution too. But again, with a couple of IFs. Firstly, you should be able to use that profiler in the affected environment. And as I have previously mentioned in my other post, profilers impose overhead of the level that might not be acceptable in the (production) environment. And secondly, you must know how to use the profiler to extract the required information and conclude the location of the leak. And my 10+ years of experience show that is very rarely the case.

Conclusion

So far we haven’t seen any definite solution to the java.lang.OutOfMemoryError: PermGen space error. There were a few that can be viable in some cases. But I was astounded by the fact that the majority of proposals were just plain invalid! You could waste days or weeks trying them and not even start to solve the real problem: find that rogue reference that is the root cause of the leak!

Fortunately, as of the 1.1 release, Plumbr also discovers PermGen leaks. And it tells you the very reason that keeps the classloader from being freed, sparing you the time of hunting down the leak. So next time, when facing thejava.lang.OutOfMemoryError: PermGen space message, download Plumbr and get rid of the problem for good.

Published at DZone with permission of Nikita Salnikov-tarnovski, author and DZone MVB. (source)

Link: http://java.dzone.com/articles/busting-permgen-myths

java.lang.OutOfMemoryError: Permgen space

Java applications are allowed to use only a limited amount of memory. The exact amount of memory your particular application can use is specified during application startup. To make things more complex, Java memory is separated into different regions as seen on the following diagram:

The size of all those regions, including permgen area, is set during the JVM launch. If you do not set the sizes yourself, platform-specific defaults will be used.

So – the “java.lang.OutOfMemoryError: PermGen space” message indicates that the Permanent Size area in memory is exhausted.

Cause of java.lang.OutOfMemoryError

To understand the cause for the “java.lang.OutOfMemoryError: PermGen space“, we need to start by reminding that everything in Java is represented as an Object. Also, all objects are instances from a specific Class. Even the Class declarations themselves are ultimately just very specific Objects. But what makes the class declarations interesting is the fact that on most JVMs Class declarations are loaded into a specific memory region, named Permanent Generation (PermGen for short).So – to recap: all Java classes are loaded and stored in the Java PermGen. This consists of the following:

• Names of the classes
• Fields of the class
• Methods of a class with the bytecode of the methods
• Constant pool information
• Object arrays and type arrays associated with a class
• Just In Time compiler optimizations

That’s pretty much it. Some more bits and pieces but it does not impact the actual memory consumption by more than few percent. All of these are allocated to PermGen and stay in PermGen.

As you can see, the PermGen size requirements depend both upon the number of classes loaded as well as the size of such class declarations. So it is easy to see the main cause for the “'''java'.'lang'.'OutOfMemoryError': 'PermGen' 'space'''“: either too many classes or too big classes are being loaded to the permanent generation.

Most often such errors are triggered during the redeploy operations. When you are redeploying an application you are trying to get rid of the previous classloader referencing all the previously loaded classes and replace it with a classloader loading new versions of the classes. Unfortunately many libraries used within applications make it impossible to throw away the old classloader which means that during each redeploy all the previous versions of your classes will still reside in PermGen and consume the memory.

==== Trivial example ====

As seen in the causation section, PermGen space usage is strongly correlated with the number of classes loaded into the JVM. The following code serves as the most straightforward example:

import javassist.ClassPool;

public class MicroGenerator {

public static void main(String[] args) throws Exception {
for (int i = 0; i < 100_000_000; i++) {
generate("eu.plumbr.demo.Generated" + i);
}
}

public static Class generate(String name) throws Exception {
ClassPool pool = ClassPool.getDefault();
return pool.makeClass(name).toClass();
}
}

In this example the source code is iterating over a loop and generating classes at the runtime. Class generation complexity is taken care by the javassist library.

Launching the code above will keep generating new classes and loading their definitions in Permgen space until the space is fully utilized and the “java.lang.OutOfMemoryError: Permgen space” is thrown.

==== Realistic example ====

As bit more complex and more realistic example, I demonstrate to you a“java.lang.OutOfMemoryError: Permgen space” error occurring during a redeploy. To reproduce this error yourself, download[1]a[2]demo[3]application bundled with Plumbr, a memory leak detector tool. When running the demo application and attempting a redeploy with Plumbr attached, Plumbr will find a memory leak, similar to the following:

The report highlights the org.hsqldb.jdbcDriver class, held in java.sql.DriverManager. When the application is started, the initializing code loads the HSQL JDBC driver org.hsqldb.jdbcDriver to connect to the database. Corresponding to specification, this JDBC driver registers itself withjava.sql.DriverManager. This registration includes storing a reference to an instance of org.hsqldb.jdbcDriver inside a static field of DriverManager.

Now, when the application is undeployed from the application server, java.sql.DriverManager will still hold that reference. Apparently there is no code in neither the HSQLDB library, the Spring framework or in the application to remove it!

As a result, a jdbcDriver object holds a reference to org.hsqldb.jdbcDriver class which in turn holds reference to the instance of java.lang.Classloader used to load the application. And thatClassLoader still references all classes of the application. In case of this particular demo application, almost 2000 classes are loaded during application startup. These occupy roughly 12MB in PermGen. Which means that it would take about seven redeploys to fill PermGen and get the “java.lang.OutOfMemoryError: PermGen space” error message in your logs.

==== Solution for java.lang.OutOfMemoryError ====

The first solution to the OutOfMemoryError due to PermGen should be obvious. If we have exhausted the PermGen area in the memory we need to increase its size. This solution is indeed helpful if you just have not given your JVM enough elbow room. So alter your application launch configuration and add (or increase if present) the following:

-XX:MaxPermSize=512m

This configuration parameter is indicating to the JVM that PermGen is allowed to grow up to 512 MB before complaining in the form of OutOfMemoryError.

Second possibility is to allow GC to unload classes from PermGen. The standard JVM is rather conservative in this regard – Classes are born to live forever. So once loaded, classes stay in memory even if no one is really using them anymore.

This can become a problem when the application is creating lots of classes dynamically and the generated classes are not needed for longer periods. In such a case, allowing JVM to unload class definitions can be helpful. This is achieved by adding again just one configuration parameter to your startup scripts:

-XX:+CMSClassUnloadingEnabled

By default this is set to false and so to enable this you need to explicitly set the following option in Java options. If you enable CMSClassUnloadingEnabled, GC will sweep PermGen too and remove classes which are no longer used. Keep in mind that this option will work only whenUseConcMarkSweepGC is also enabled using the below option. So when running parallel or, God forbid, serial GCs, make sure you have set your GC to CMS by specifying:

-XX:+UseConcMarkSweepGC

But before calling it a night, be warned – more often than not usage of the recommended “quick fixes” means you are just masking the symptoms by hiding “java.lang.OutOfMemoryError: Permgen space” and are not tackling the underlying problem.

For example, if you really want to fix the leakage in PermGen we introduced in the examples section, the following servlet context listener can be used:

public class JdbcDriverLeakPreventer implements ServletContextListener {

@Override

public void contextInitialized(ServletContextEvent sce) {

  //Nothing to do

}

@Override

public void contextDestroyed(ServletContextEvent sce) {

  ClassLoader applicationClassLoader = this.getClass().getClassLoader();

  Enumeration driverEnumeration = DriverManager.getDrivers();

  while (driverEnumeration.hasMoreElements()) {

    Driver driver = driverEnumeration.nextElement();

    ClassLoader driverClassLoader = driver.getClass().getClassLoader();

    if (driverClassLoader != null

        && driverClassLoader.equals(applicationClassLoader)){

      try {

        DriverManager.deregisterDriver(driver);

      } catch (SQLException e) {

        e.printStackTrace(); //TODO Replace with your exception handling

      }

    }

  }

}

}

This Servlet listener should be registered, for example, in the web.xml file of your application:

<listener>

  <listener-class>user.package.JdbcDriverLeakPreventer</listener-class>

</listener>

There are several ways to find out what has occupied the PermGen space in your JVM and whether the objects had a valid reason to fill it up. Memory dump analyzers, profilers, debuggers – the choice is yours. But if you need to be sure your PermGen is not filled up by garbage from a memory leak we can only recommend using Plumbr and finding it out for free.