Out of Memory

Busting PermGen Myths

In my latest post I explained the reasons that can cause the java.lang.OutOfMemoryError: PermGen space crashes. Now it is time to talk about possible solutions to the problem. Or, more precisely, about what the Internet suggests for possible solutions. Unfortunately, I can only say that I felt my inner Jamie Hyneman from MythBusters awakening when going through the different "expert opinions" on the subject.

I googled for current common knowledge about ways to solve java.lang.OutOfMemoryError: PermGen space crashes and went through a couple dozen pages that seemed more appropriate in Google results. Fortunately, most of the suggestions have already been distilled into this topic of the very respected StackOverflow. As you can see, the topic is truly popular and has some quite highly voted answers. But the irony is that the whole topic contains exactly zero solutions I could recommend myself. Well, aside from “Find the cause of memory leak”, which is absolutely correct, of course, but not very helpful way to respond to the question “How to solve memory leak”. Let us review the suggestions put forward on the SO page.

Use -XX:MaxPermSize=XXXM

There can be two reasons that cause the java.lang.OutOfMemoryError: PermGen space error.

One is that application server and/or application really does use so many classes that they do not fit into default sized Permanent Generation. It is definitely possible and not that rare in fact. In this case increasing the size of Permanent Generation can really save the day. If your only problem is how to fit too many furniture into too small house, then buy the bigger house!

But what if your over-caring mother sends you new furniture every week? You cannot possibly continue to move to the bigger houses over and over again. That is exactly the situation with memory leaks - and also with the classloader leaks, as described in my previous post that I mentioned above. Let me be clear here: no increase in Permanent Generation size will save you from the classloader leak. It can only postpone it. And make it harder to predict how many re-deployments your server will outlive.

-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled

The most popular answer on StackOverflow was to add these options to the server’s command line. And, they say, "maybe add -XX:+UseConcMarkSweepGC too. Just to be sure". My first problem with these JVM flags is that there is no explanation available of what they really do. Neither in the SO answer (and I don't like answers that tell you to do something without the reasoning why you should do it), nor actually in the whole Internet.

Really, I was unable to find any documentation about these options, except for this page. But, in fact, that does not even matter. In no way any tinkering with the Garbage Collector options will help you in case of a classloder leak. Because, by definition, a memory leak is a situation where GC falls short. If there is a valid live hard reference from somewhere within your server’s classloader to an object or class of your application, then the GC will never think of it as garbage and will never reclaim it. Sure, all these JVM flags look very smart and magical. And they really may be required in some situations. But they are certainly notsufficient and don’t solve your Permanent Generation leak.

Use JRockit

The next proposition was to switch to the JRockit JVM. The rationale was that as JRockit has no Permanent Generation, one cannot run out of it. Surely, an interesting proposition. Unfortunately, it will not solve our problem either.

The only result of this “solution” will be getting a java.lang.OutOfMemoryError: Java heap space instead of the java.lang.OutOfMemoryError: PermGen space. In the absence of separate generation for class definitions, JRockit uses the usual Java heap space for them. And as long the root cause of the leak is not fixed, those class definitions will fill up even the largest heap, given enough time.

Restart the server

Yet another way to pretend that the problem is solved, is to restart the application server from time to time. E.g. instead of redeploying the application, just restart the whole server. But the first time you see an application server with more than one application deployed, you will know that this is rarely possible in production environment. And this is not really a solution. It is a way to hide your head in the sand.

Use Tomcat

This one is actually not that hopeless as the previous ones - recent Tomcat versions really do try to solve classloader leaks. See for yourself in their documentation. IF you can use Tomcat as your target server, and IF your leak is one of those Tomcat can successfully fight against, then maybe, just maybe, you are lucky and the problem is solved for you.

Use <Your favorite profiler tool here>

May be a viable solution too. But again, with a couple of IFs. Firstly, you should be able to use that profiler in the affected environment. And as I have previously mentioned in my other post, profilers impose overhead of the level that might not be acceptable in the (production) environment. And secondly, you must know how to use the profiler to extract the required information and conclude the location of the leak. And my 10+ years of experience show that is very rarely the case.

Conclusion

So far we haven’t seen any definite solution to the java.lang.OutOfMemoryError: PermGen space error. There were a few that can be viable in some cases. But I was astounded by the fact that the majority of proposals were just plain invalid! You could waste days or weeks trying them and not even start to solve the real problem: find that rogue reference that is the root cause of the leak!

Fortunately, as of the 1.1 release, Plumbr also discovers PermGen leaks. And it tells you the very reason that keeps the classloader from being freed, sparing you the time of hunting down the leak. So next time, when facing thejava.lang.OutOfMemoryError: PermGen space message, download Plumbr and get rid of the problem for good.

Published at DZone with permission of Nikita Salnikov-tarnovski, author and DZone MVB. (source)

Link: http://java.dzone.com/articles/busting-permgen-myths

java.lang.OutOfMemoryError: Permgen space

Java applications are only allowed to use a limited amount of memory. The exact amount of memory your particular application can use is specified during application startup. To make things more complex, Java memory is separated into different regions which can be seen in the following figure:

The size of all those regions, including the permgen area, is set during the JVM launch. If you do not set the sizes yourself, platform-specific defaults will be used.

The java.lang.OutOfMemoryError: PermGen space message indicates that the Permanent Generation’s area in memory is exhausted.

What is causing it?

To understand the cause for the java.lang.OutOfMemoryError: PermGen space, we would need to understand what this specific memory area is used for.

For practical purposes, the permanent generation consists mostly of class declarations loaded and stored into PermGen. This includes the name and fields of the class, methods with the method bytecode, constant pool information, object arrays and type arrays associated with a class and Just In Time compiler optimizations.

From the above definition you can deduce that the PermGen size requirements depend both on the number of classes loaded as well as the size of such class declarations. Therefore we can say that the main cause for the java.lang.OutOfMemoryError: PermGen space is that either too many classes or too big classes are loaded to the permanent generation.

Give me an example

Minimalistic example

As we described above, PermGen space usage is strongly correlated with the number of classes loaded into the JVM. The following code serves as the most straightforward example:

import javassist.ClassPool;

public class MicroGenerator {
  public static void main(String[] args) throws Exception {
    for (int i = 0; i < 100_000_000; i++) {
      generate("eu.plumbr.demo.Generated" + i);
    }
  }

  public static Class generate(String name) throws Exception {
    ClassPool pool = ClassPool.getDefault();
    return pool.makeClass(name).toClass();
  }
}

In this example the source code iterates over a loop and generates classes at runtime. Class generation complexity is being taken care of by the javassist library.

Launching the code above will keep generating new classes and loading their definitions into Permgen space until the space is fully utilized and the java.lang.OutOfMemoryError: Permgen space is thrown.

Redeploy-time example

For a bit more complex and more realistic example, lets walk you through a java.lang.OutOfMemoryError: Permgen space error occurring during the application redeploy. When you redeploy an application, you would expect that Garbage Collection will get rid of the previous classloader referencing all the previously loaded classes and it gets replaced with a classloader loading new versions of the classes.

Unfortunately many 3rd party libraries and poor handling of resources such as threads, JDBC drivers or filesystem handles makes unloading the previously used classloader impossible. This in turn means that during each redeploy all the previous versions of your classes will still reside in PermGen generating tens of megabytes of garbage during each redeploy.

Let’s imagine an example application that connects to a relational database using JDBC drivers. When the application is started, the initializing code loads the JDBC driver to connect to the database. Corresponding to the specification, the JDBC driver registers itself with java.sql.DriverManager. This registration includes storing a reference to an instance of the driver inside a static field of DriverManager.

Now, when the application is undeployed from the application server, java.sql.DriverManager will still hold that reference. We end up having a live reference to the driver class which in turn holds reference to the instance of java.lang.Classloader used to load the application. This in turn means that the Garbage Collection Algorithms are not able to reclaim the space.

And that instance of java.lang.ClassLoader still references all classes of the application, usually occupying tens of megabytes in PermGen. Which means that it would take just a handful of redeploys to fill a typically sized PermGen and get the java.lang.OutOfMemoryError: PermGen space error message in your logs.

What is the solution?

1. Solving initialization-time OutOfMemoryError

When the OutOfMemoryError due to PermGen exhaustion is triggered during the application launch, the solution is simple. The application just needs more room to load all the classes to the PermGen area so we just need to increase its size. To do so, alter your application launch configuration and add (or increase if present) the -XX:MaxPermSize parameter similar to the following example:

java -XX:MaxPermSize=512m com.yourcompany.YourClass

The above configuration will tell the JVM that PermGen is allowed to grow up to 512MB before it can start complaining in the form of OutOfMemoryError.

2. Solving redeploy-time OutOfMemoryError

When the OutOfMemoryError occurs right after you redeploy the application, your application suffers from classloader leakage. In such a case, the easiest and most straightforward way to solve the problem is to grab a 14-day free trial of Plumbr, find the offending code and solve it in minutes.

For those who cannot use Plumbr or decide not to, alternatives are also available. For this, you should proceed with heap dump analysis – take the heap dump after a redeploy with a command similar to this one:

jmap -dump:format=b,file=dump.hprof <process-id>

Then open the dump with your favourite heap dump analyzer (Eclipse MAT is a good tool for that). In the analyzer, you can look for duplicate classes, especially those loading your application classes. From there, you need to progress to all classloaders to find the currently active classloader.

For the inactive classloaders, you need to determine the reference blocking them from being Garbage Collected via harvesting the shortest path to GC root from the inactive classloaders. Equipped with this information you will have found the root cause. In case the root cause was in a 3rd party library, you can proceed to Google/StackOverflow to see if this is a known issue to get a patch/workaround. If this was your own code, you need to get rid of the offending reference.

3. Solving run-time OutOfMemoryError

When the application runs out of PermGen memory during runtime, the Plumbr dynamic leak detection capability is the best way to find the source for the leakage. Grab the free 14-day trial and get rid of the issue.

An alternative way for those once again who cannot use Plumbr is also available. First step in such case is to check whether the GC is allowed to unload classes from PermGen. The standard JVM is rather conservative in this regard – classes are born to live forever. So once loaded, classes stay in memory even if no code is using them anymore. This can become a problem when the application creates lots of classes dynamically and the generated classes are not needed for longer periods. In such a case, allowing the JVM to unload class definitions can be helpful. This can be achieved by adding just one configuration parameter to your startup scripts:

-XX:+CMSClassUnloadingEnabled

By default this is set to false and so to enable this you need to explicitly set the following option in Java options. If you enable CMSClassUnloadingEnabled, GC will sweep PermGen too and remove classes which are no longer used. Keep in mind that this option will work only when UseConcMarkSweepGC is also enabled using the below option. So when running ParallelGC or, God forbid, Serial GC, make sure you have set your GC to CMS by specifying:

-XX:+UseConcMarkSweepGC

After making sure classes can be unloaded and the issue still persists, you should proceed with heap dump analysis – taking the heap dump with a command similar to following:

jmap -dump:file=dump.hprof,format=b <process-id>

Then opening the dump with your favorite heap dump analyzer (e.g. Eclipse MAT) and progressing to find the most expensive classloaders by the number of classes loaded. From such classloaders, you can proceed to extract the loaded classes and sort such classes by the instances to have the top list of suspects.

For each suspect, you then need to manually trace the root cause back to your application code that generates such classes.

What is a PermGen leak?

July 11, 2012 by Nikita Salnikov-Tarnovski

What follows is a practical introduction to a specific type of memory problems in Java applications. Namely – we will analyze the errors that cause the java.lang.OutOfMemoryError: PermGen spacesymptom in the stack trace.

First of all we will go through the core concepts required to understand the subject – and explain what objects, classes, classloaders and the JVM memory model are. If you are familiar with the basic concepts, you can jump directly to the next section where I will describe two typical cases for the error in question alongside with hints and suggestions for solving it.

Objects, Classes and ClassLoaders

Well, I will not start with the very basics. I guess if you have found us, you should already be familiar with the concept that everything in Java is an Object. And that all Objects are specified by their Class. So every object has a reference to an instance of java.lang.Class describing the structure of that object’s class.

But what actually happens under the hood when you create a new object in your code? For example if you write something truly complicated like

Person boss = new Person()

the Java Virtual Machine (JVM) needs to understand the structure of the object to create. To achieve this, the JVM looks for the class called Person. And if the Person class is accessed for the first time during this particular execution of the program, it has to be loaded by the JVM, normally from the corresponding Person.class file. The process of seeking for the Person.class file on the drive, loading it into memory and parsing it’s structure is called class loading. Ensuring proper class loading process is the responsibility of a ClassLoader. ClassLoaders are instances of java.lang.ClassLoader class and each and every class in a Java program has to be loaded by some ClassLoader. As a result we now have the following relationships:

As you can see from the next diagram every classloader holds references to all the classes it has loaded. For the purpose of our article these relationships are very interesting.

Remember this image, we will need it later.

Permanent Generation

Almost every JVM nowadays uses a separate region of memory, called the Permanent Generation (or PermGen for short), to hold internal representations of java classes. PermGen is also used to store more information – find out the details from this post if you are interested – but for our article it is safe to assume that only the class definitions are being stored in PermGen. The default size of this region on my two machines running java 1.6 is not a very impressive 82MB.

As I have explained in one of my earlier posts, a memory leak in Java is a situation where some objects are no longer used by an application, but the Garbage Collector fails to recognize them as unused. This leads to the OutOfMemoryError if those unused objects contribute to the heap usage significantly enough that the next memory allocation request by the application cannot be fulfilled.

The root cause of java.lang.OutOfMemoryError: PermGen space is exactly the same: the JVM needs to load the definition of a new class but there is not enough space in PermGen to do it – there are already too many classes stored there. A possible reason for this could be your application or server using too many classes for the current size of PermGen to be able to accommodate them. Another common reason could be a memory leak.

Permanent Generation Leak

But still, how on earth it is possible to leak something in PermGen? It holds definitions of java classes and they cannot become unused, can they? Actually, they can. In case of a Java web application deployed into an application server all those classes in your EAR/WAR become worthless when the application is undeployed. The JVM continues to run as the application server is still alive, but a whole bunch of class definitions are not in use anymore. And they should be removed from PermGen. If not, we will have memory leak in the PermGen area.

As a nice sample on the reasons – the Tomcat developers have set up a Wiki page describing different leaks found and fixed in the Apache Tomcat versions 6.0.24 and above.

Leaking Threads

One possible scenario for a classloader leak is through long running threads. This happens when your application or – as often was the case in my experience – a 3rd party library used by your application starts some long running thread. An example of this could be a timer thread whose job is to execute some code periodically.

If the intended lifespan of this thread is not fixed, we are heading directly into a trouble. When any part of your application ever starts a thread, you must make sure that it is not going to outlive the application. In typical cases the developer is either not aware of this responsibility or simply forgets to write the clean-up code.

Otherwise, if some thread continues to run after the application is undeployed it will usually hold a reference to a classloader of the web application it was started by, called context classloader. Which in turn means that all classes of the undeployed application continue to be held in memory. Remedy? If it is your application that starts new threads, you should shut them down during undeployment using a servlet context listener. If it is a 3rd party library, you should search for its own specific shutdown hook. Or file a bug report if there is none.

Leaking Drivers

Another typical case of a leak can be caused by database drivers. We have encountered this leak in our own demo application that we ship with Plumbr. It is a slightly modified Pet Clinic application shipped along with Spring MVC. Let us highlight some things that happen when this application is being deployed to the server.

• The server creates a new instance of java.lang.Classloader and starts to load the application’s classes using it.
• Since the PetClinic uses a HSQL database, it loads the corresponding JDBC driver, org.hsqldb.jdbcDriver.
• This class, being a good-mannered JDBC driver, registers itself with java.sql.DriverManager during initialization, as required per JDBC specification. This registration includes storing inside a static field of DriverManager a reference to an instance of org.hsqldb.jdbcDriver.

Now, when the application is undeployed from the application server, the java.sql.DriverManager will still hold that reference, as there is no code in the HSQLDB library nor in the Spring framework nor in the application to remove that! As was explained above, a jdbcDriver object still holds a reference to an org.hsqldb.jdbcDriver class, which in turn holds a reference to the instance of java.lang.Classloader used to load the application. This classloader now still references all the classes of the application. In case of our particular demo application, during application startup almost 2000 classes are loaded, occupying roughly 10MB in PermGen. Which means that it takes about 5-10 redeploys to fill the PermGen with default size to reach the java.lang.OutOfMemoryError: PermGen space crash.

How to fix that? One possibility is to write a servlet context listener, which de-registers the HSQLDB driver from DriverManager during application shutdown. This is pretty straightforward. But remember – you will have to write the corresponding code in every application using the driver.

Download our latest version of Plumbr with our demo application and play with it to find out how the leak occurs, how Plumbr finds it and how we explain the cause.

Conclusion

There are many reasons why your application might encounter a java.lang.OutOfMemoryError: PermGen space. The root cause for the majority of them is some reference to an object or a class loaded by the application’s class loader that has died after that. Or a direct link to the class loader itself. The actions you need to take for remedy are quite similar for most of these causes. Firstly, find out where that reference is being held. Secondly, add a shutdown hook to your web application to remove the reference during application’s undeployment. You can do that by either using a servlet context listener or by using the API provided by your 3rd party library.

Finding those leaking references has never been easy. We ourselves have spent countless hours trying to track down why some applications require 20MB of PermGen on each redeploy. But as of version 1.1,, Plumbr will show you the cause of the leak and give you a hint on how to fix it. If you want to try it out, register and download the tool.