Julien's tech blog

talking about tech stuff: stuff I'm interested in, intriguing stuff, random stuff.

Detecting low memory in Java Part 2

This is a follow up on my previous post the rationale is explained there.

I ended up spending a little more time on the low memory detection issue and played with the MemoryPoolMXBean. I had found some posts about it but none were satisfying. This post (thanks @techmilind for pointing it out) lead me in the right direction even though it is partially incorrect. Experimenting with a monotonically increasing memory usage is a very special (and invalid) case.
In particular the setUsageThreshold() is not very useful in my case as it is triggered the very first time we use that much memory, regardless of imminent garbage collection. As discussed in my previous post it is useful only if the available memory is measured right after garbage collection.
However setCollectionUsageThreshold() is exactly what I need. This is setting a threshold for notification when the memory is low right after a GC.

We need to do this in two steps, first set a collectionUsageThreshold on the tenured partition.

// heuristic to find the tenured pool (largest heap) as seen on http://www.javaspecialists.eu/archive/Issue092.html
MemoryPoolMXBean tenuredGenPool = null;
for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
  if (pool.getType() == MemoryType.HEAP && pool.isUsageThresholdSupported()) {
    tenuredGenPool = pool;
  }
}
// we do something when we reached 80% of memory usage
tenuredGenPool.setCollectionUsageThreshold((int)Math.floor(tenuredGenPool.getUsage().getMax()*0.8));

Here tenuredGenPool.getCollectionUsage() is the memory usage as measured right after the last garbage collection. This is the value that we can rely on to detect low memory.

Then we setup a listener to get notified. Only MEMORY_COLLECTION_THRESHOLD_EXCEEDED is interesting as explained before.

//set a listener
MemoryMXBean mbean = ManagementFactory.getMemoryMXBean();
NotificationEmitter emitter = (NotificationEmitter) mbean;
emitter.addNotificationListener(new NotificationListener() {
public void handleNotification(Notification n, Object hb) {
if (n.getType().equals(
  MemoryNotificationInfo.MEMORY_COLLECTION_THRESHOLD_EXCEEDED)) {
   // this is the signal => end the application early to avoid OOME
}
}}, null, null);

My experiment slowly increases the memory usage while also freeing up a bunch of existing object. This shows that the MEMORY_THRESHOLD_EXCEEDED is reached pretty quickly when most of the memory gets cleaned afterwards. MEMORY_COLLECTION_THRESHOLD_EXCEEDED is triggered when we actually fill up the memory.

edit: actual code explains it better than words.

Here is some test code to show how that works:

https://github.com/julienledem/blog/blob/master/2011/07/21/detecting-low-memory-in-java-part-2/MemoryTest.java

sample output: (only the last line is the warning we want)

it=   0 - Par Eden Space: u= 49% cu=  0% th=0% - Par Survivor Space: u=  0% cu=  0% th=0% - CMS Old Gen: u=  0% cu=  0% th=79%
it= 100 - Par Eden Space: u= 16% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 35% cu=  0% th=79%
memory threshold exceeded !!! : 
        - Par Eden Space: u= 92% cu=  0% th=0% - Par Survivor Space: u= 96% cu= 96% th=0% - CMS Old Gen: u= 86% cu=  0% th=79%
it= 200 - Par Eden Space: u= 84% cu= 45% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 19% cu= 19% th=79%
memory threshold exceeded !!! : 
        - Par Eden Space: u=  1% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 83% cu= 19% th=79%
it= 300 - Par Eden Space: u= 49% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 79% cu= 74% th=79%
memory threshold exceeded !!! : 
        - Par Eden Space: u=  4% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 84% cu= 74% th=79%
memory threshold exceeded !!! : 
        - Par Eden Space: u=  0% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 83% cu= 33% th=79%
it= 400 - Par Eden Space: u= 15% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 51% cu= 38% th=79%
memory threshold exceeded !!! : 
        - Par Eden Space: u=  6% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 83% cu= 38% th=79%
it= 500 - Par Eden Space: u= 79% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 59% cu= 46% th=79%
memory threshold exceeded !!! : 
        - Par Eden Space: u=  0% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 84% cu= 51% th=79%
it= 600 - Par Eden Space: u= 43% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 64% cu= 56% th=79%
memory threshold exceeded !!! : 
        - Par Eden Space: u=  0% cu=  0% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 84% cu= 56% th=79%
memory collection threshold exceeded !!! : 
        - Par Eden Space: u= 19% cu=  9% th=0% - Par Survivor Space: u= 99% cu= 99% th=0% - CMS Old Gen: u= 85% cu= 85% th=79%
Advertisements

3 responses to “Detecting low memory in Java Part 2

  1. thejasnair July 22, 2011 at 9:40 am

    This is one of the approaches used in pig for Mem Management- http://t.co/IABim9e in SpillableMemoryManager.

    But sometimes sometimes that low mem notification was too late and pig OOMs, so Internal cached bag (and subclasses) were introduced – https://issues.apache.org/jira/browse/PIG-975. They spill to disk when the (estimated) memory used by them reaches a configurable threshold.
    This approach has proved to be much more reliable.

    There is also a library that twitter has open sourced (on github?) that lets you estimate the memory used by java objects.

  2. Diego Rojas Castro October 31, 2013 at 12:21 pm

    Is there a way to receive a second notification when the memory usage has gone below the threshold so the application can resume receiving tasks or continue with its normal processing?

    • julienledem November 7, 2013 at 10:26 am

      Usually the memory usage will not go down unless you release those objects you hold onto.
      So if you take action on the notification then you can check your new memory usage.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: