There are various ways to combat this issue and the first instinct would be suggest that there is a memory leak. After eliminating this as a possibility, the next challenge was to identify where the memory was being taken up. This took some time and effort and the hibernate second level cache was identified. We were storing far too much in the second level cache.

This is another double edged sword. The hibernate second level cache is absolutely imperative to a high performance system. It does however, come with a price. The cache needs to be managed carefully to ensure that balance between performance and memory requirements.

To this end, it was important to be able to identify what was taking up all the memory in the cache. Each object might only take a couple of hundred bytes, but with our second level cache set to store hundreds of thousands of items, this quickly takes up hundreds of megabytes. With the metadata of the cache, this could easily hike it up near a gigabyte of memory usage. This gets substantially worse with cache evictions and the adding of new items into the cache.

The correct way to resolve this is to identify specific object types that “overload” the cache. i.e. items that have an large number of instances stored in the cache. Identifying classes that store a large number of items is easy enough - we just traverse the cache and count up the number of items. However, there might be a class that stores a smaller number of items but take a sizeable amount of memory. For this reason, it is important to understand the object sizes in memory as well.

If you have ever tried to find a way to identify object sizes, you will know that this is no easy task. You can calculate to some degree of accuracy the size of an object based on the data it stores but this is a manual process.

The only real way to get this information is to use a java agent and use that to calculate a more accurate memory usage. For this purpose, we used the classmexer agent which requires a simple installation step of adding the following parameter to java -javaagent:classmexer.jar. You can then figure out the memory utilisation of an object by calling

You can also pass in a collection of objects:

This was the simple part.

Traversing the node structure of jboss cache and collating a collection statistics with regards to the number of each type of object and its memory utilisation was a little more interesting.

I will cover this separately

Simple? NO! The main problem was the management of the data set. I had set up, in the past fairly interesting classes to test the functionality using reflection, and injecting the data from the classes themselves through the data provider mechanism of TestNG. However, this was error prone and clunky at best. It also made dependency management of data quite cumbersome.

With a view to resolving this, I also looked at DbUnit, unitils and Ejb3Unit. They all did some things that I liked but lacked some functionality that was important.

This led me to write a simple testing infrastructure. The goal was straightforward.

• I need to be able to define data in a CSV (actually it was seperated by the pipe character |, so PSV) based on entities.
• The framework should automatically persist the data (and fail on errors)
• It should test that it can load all that data back
• It should run as many automated tests on the DOM as possible.

The framework uses the CSV files to read the data for each of the classes (using the excellent SuperCsv library). It needs an Id field for internal reference. As long as the id’s match within the CSV files for the relationships, it will be persisted correctly into the database even when the persisted id’s are different.

For example, I could have a Contact.csv with 5 records (ids 1 through 5) and a Company.csv with 3 records (ids 1 through 3).

The Contact.csv records can map to the id specified in the Company.csv file and when the records get persisted, they will be associated correctly, even if the id’s in the database end up being different.

The framework also looks for the CSV file which has the same name as the class within the location defined within the configuration file. This means that as long as the filename matches the class name, the data loading is automatic.

For simple classes, the Test case is as simple as: