GC and server tuning

Server tuning is still a black art to a degree. The best hints are to look at what we do in tests like ECPerf of jSpecApp. There, you'll see our experts do what ever they can to go faster.

If you're writing a pretty critical response time critical application like a trading system then GC can be a problem. You really need GC to happen rarely and when it does to only hold up processing for the minimum time possible.

I'm finding that the following works well for me in terms of JVM tuning on IBM JDKs:

"-Xgcpolicy:optavgpause" "-Xgcthreads2"

This slightly slows response time because the JDK is doing more work on each alloc command but the GC pauses when they come can be half the size. The gcthreads parameter should be set to the number of CPUs assigned to your application. This speeds the GC operation by parallelizing it.

Heap Size
The JVM heap should also be only as big as you really need it. You may be tempted to set your heap to 3GB but while this results in less frequent GC operations, the GC pauses will be longer, there is simply more memory to GC, it takes longer. If you are sensitive to GCs then this is the wrong thing to do in my opinion anyway. A small heap may result in more frequent GCs but the pauses should be lower meaning that when they occur the response time has a minimum hit. The former case means, yep, less transactions are impacted but when the GC comes, it increases the response time at that moment significantly. So, if what you want is more consistent response times then a smaller heap is better.

Thick versus Thin JDBC Drivers
Check if the thick JDBC driver generates less GC than the thin driver. Remember, the thick driver is written in C with a JNI wrapper. There are simply less Java objects getting created here. It may be slightly slower than a thin one (because thin doesn't cross JNI) but it may generate significantly less garbage and hence the GC pauses may be shorter when they come. Again, it's consistency versus flat out performance over a short interval.

Thread pool size
Only use as many threads as you need to saturate the box. If in testing, you notice that 6 threads pushes your 4 way to 100% CPU running your tran mix then there is no point in having more than 6 threads in the pools. More threads will SLOW IT DOWN because of context switching. So, if your applications can be tested and you see this how many threads at 100% then just set the pool maximums to this and it'll perform better.

Comments

[Cameron]

Thread pool should be set to the value that provides the highest throughput. That may mean adding a couple threads even when the CPU shows 100%, or it may mean not getting the CPU to 100%.

To determine this value, you need to load test, and "narrow in" on the optimal number. Try 10 .. try 20 .. if you can add threads and throughput increases, then add threads. Once that trend levels or reverses, go backwards in small increments until you find the peak.

Peace.

[Billy]

No argument with that.

[Bruce Ritchie]

One of the places I recommend people start with when looking at gc tuning with the latest Sun JVM is http://java.sun.com/docs/hotspot/gc1.4.2/

As well, Sun has released an application that analyzes GC logs which can prove useful - http://192.168.1.206:8080/jive3/admin/index.jsp

Regards,

Bruce Ritchie

[Bruce Ritchie]

Oops, that last url should be http://developer.java.sun.com/developer/technicalArticles/Programming/GCPortal/

Regards,

Bruce Ritchie

[Mo Marikar]

in regards to GC policy
Testing of a servlet serving JSP pages interfacing to Oracle 9i with a Novell redirector (yet another Struts withpersonalization capabilities) -
"Xgcpolicy:optavgpause" "Xgcthreads2"
what I'm finding on a 2 CPU Linux Blade running WAS 5.1 is as follows:
1. The above after about 40 minutes into a 60 minute test crashed the IBM 1.4.1 JVM. Repeated the tests again, and the TPS of the app dies and WAS hangs. Verbose GC clearly indicates concurrent GC collection
2. What I found to be best was
Xinitsh2M -Xgcpolicy:optthruput -Xgcthreads6
I also replaced the standard JDBC Oracle driver
with Oracle's "Heavy" driver and this combination gave the best performance.