Tuesday, November 18, 2008

Axis and Xerces can cause Perm problems

Away from the Groovy for a bit and back to my regular Java day job.

We have a Web Service that can occasionally return a large data structure. The server end runs inside a Tomcat and is served by Axis 1.4. Our client runs inside Tomcat in JBoss and also uses Axis 1.4 for its client code.

Recently, we experienced a situation that after the client called the web service to receive a large object a few times, we had a Out Of Memory (OOM) in the Perm of our JBoss (which as a reminder was the client side).

Since, we of course had the  crucial JVM flag (-XX:+HeapDumpOnOutOfMemoryError) on, so we got a memory dump to analyze.

Generally, analyzing perm issues is very difficult, but using Memory Analyzer Tool (did I mention how great that tool is?) we did find a suspiciously large object:

symboltable_full

You can see here that there was an instance of

org.apache.xerces.util.SymbolTable

that took up close to 225 MB.

A little drill down into the object showed that most of the entries in the SymbolTable (well since there were more then a million entries I can’t say that I really checked most of the entries but most of the entries I did check) were related to Soap. You can see one entry here:

symboltable_entry_zoom

You can see that the value in the entry is ns921375:severity which is a Soap wrapper type tag.

So, that object definitely seemed related to our OOM. The only problem was tying it to the Perm. Strings are suspects for perm problems, since in Sun’s JVMs interned strings get stored in the Perm space. The jmap command can give info on perm space used by a running jvm but unfortunately that functionality is only working on the Solaris version of the jvm. So I needed another way to try to tie the object to the Perm space and as the say: Use the Source, Luke

So, from looking at the Xerces code, what I learned is that Xerces uses the SymbolTable as a mechanism to reuse Strings. When the parser gets a new String it can pass it into the SymbolTable to get the canonical instance.

This makes sense in an XML Parser. A lot of tags will probably appear multiple times in an XML file and so a lot of memory can be saved if the parser can reuse the String objects. Also, (and possibly more importantly)  if the same String is continuously reused, then the parser can rely on the much faster == for comparing objects rather then calling equals.

The use of the term canonical instance was on purpose. The javadoc for String describes the intern function as:

public String intern()

Returns a canonical representation for the string object.

A pool of strings, initially empty, is maintained privately by the class String.



When the intern method is invoked, if the pool already contains a string equal to this String object as determined by the equals(Object) method, then the string from the pool is returned. Otherwise, this String object is added to the pool and a reference to this String object is returned.




So, calling intern on a String basically does what I described the SymbolTable does.



The Xerces code in fact refers to this and says that:


The symbol table performs the same task as String.intern() with the following differences:


  • A new string object does not need to be created in order to retrieve a unique reference. Symbols can be added by using a series of characters in a character array.


  • Users of the symbol table can provide their own symbol hashing implementation. For example, a simple string hashing algorithm may fail to produce a balanced set of hashcodes for symbols that are mostly unique. Strings with similar leading characters are especially prone to this poor hashing behavior.



All this sounds good. The exact details of the reasons are less important to me but the point is someone there thought about intern and decided that it wasn’t good enough.



But a little digging into the Xerces source code turned up the following tidbit in the code for SymbolTable. SymbolTable has an inner class of type Entry which is where the Strings are put into to be stored in the SymbolTable. Entry’s constructor looks like this:



public Entry(String symbol, Entry next) {
this.symbol = symbol.intern();
characters = new char[symbol.length()];
symbol.getChars(0, characters.length, characters, 0);
this.next = next;
}


What we see here is that every Entry that gets created calls intern on the String as well.  After building a uniqueness mechanism in SymbolTable, they still relied on intern and as I mentioned before, intern Strings are stored in the perm space.



So, the bottom line here is that as large XML files with lots of unique Strings are parsed, the SymbolTable will slowly fill up your perm space as well. 



Soap envelopes are full of tags like ns921375:severity which are going to be unique and so will fill up your perm.



Final Note: We changed our soap function to return the object as an attachment and it seemed to solve the problem. 

Wednesday, November 5, 2008

Groovy Sort Follow-up

Here is another Groovy post, but after last weeks post about sorting in Groovy, I just wanted to make a couple of points on that.

First I had left with a question:

how Groovy distinguishes between the Closure version of the Comparator and the straight Closure which returns the Comparable object

My guess was that there is special code to deal with it, and a little digging into the Groovy source code showed this to be correct.

The magic happens in the file src\main\org\codehaus\groovy\runtime\DefaultGroovyMethods.java:
public static List sort(Collection self, Closure closure) { 
List list = asList(self); 
// use a comparator of one item or two 
int params = closure.getMaximumNumberOfParameters(); 
if (params == 1) { 
   Collections.sort(list, new OrderBy(closure)); 
} else { 
   Collections.sort(list, new ClosureComparator(closure)); 

return list; 

As you can see, the Groovy Java support code does in fact check what type of Closure it gets, and then builds an appropriate Comparator to pass into the regular Java sort function.

At work, a standard question that always comes up as we consider doing a production project in Groovy is: How does Groovy perform versus Java. It is true that the code is much nicer in Groovy, but how much slower will it run?

As we can see above, both Groovy and Java end up in the exact same function. The only difference is the work the Comparator will do. In Java, it is a straightforward compare. In Groovy it is either doing closure work or some other kind of comparator.

I timed a number of different possibilities in Groovy and Java. (This of course comes with the standard caveat of Beware of Microbenchmarks).

In order to avoid problems I did the following:

  • I randomly built lists with 20,000 Things to sort. (see thel last Groovy post as a reminder on what Things are). I created the same lists for all the sorting code at the same time.
  • After sorting the lists, I printed out the first element of each list to ensure that the optimizer wasn’t cutting it out or something.
  • I created and sorted lists 3 times to “warm up” the code
  • I then ran 100 times and averaged the runs.

So keeping those risks in mind, I timed the following codes (the time in parens is the average time for sorting the list using the given code):
  1. Basic Java sort (15 ms):
    Collections.sort(inList,new Comparator<Thing>() {
    public int compare(Thing o1, Thing o2) {
    return (o1.val.compareTo(o2.val));
    }
    });



  2. Basic Groovy Closure (487 ms):

    inList.sort {
    it.val
    }



  3. Groovy using the two param closure (596 ms):

    inList.sort {o1, o2 ->
    o1.val.compareTo(o2.val)
    }

  4. Groovy using a dynamically built Comparator (612 ms):

    inList.sort([compare:{a,b->
    a.val.compareTo(b.val) } ] as Comparator)



  5. Groovy using a prebuilt Comparator (444 ms):

    The Groovy Comparator:

    class StringComparator implements Comparator {
    int compare(o1, o2) {
    return o1.val.compareTo(o2.val)
    }
    }
    StringComparator sc = new StringComparator();

    And here is the sort code:

    inList.sort(sc)



Some interesting things can be seen here:


  • Java is shockingly faster – so much so that I am worried that I am falling for one of the micro-benchmark problems


  • The dynamic comparator versus the pre-built comparator is 50% slower – probably caused by the extra redirection and lookup the dynamic one requires.


  • I am also surprised that the two param closure was slower then the one param version. I would have thought that they would do similar work.