Slow starup of Java application - best practices for fine tuning JVM?

Question

We are having problems with a java application, which takes a long time to startup.
In order to understand our question we better start with some background info. You will find the question(s) after that.
Background:
The setup is as follows:
In a client-server solution we have a win xp, fat client running java 1.6.0.18.
(Sun JRE). The fat client containt a lot of GUI, and connects to a server for DB access. Client machines are typical 1 to 3 years old (there are problems even on brand new machines). They have the client version of JRE - standard edition installed (Java SE 6 update 10 or better) Pretty much usual stuff so far.
We have done a lot of profiling on the client code, and yes we have found parts of our own Java code that needs improving. we are all over this. Server side seems ok with good response times. So far, we havent found anything about shaky net connections or endless loops in the java client code or similiar.
Still, things are not good. Starting the application takes a long time. too long.
There are many complicating factors, but here is what we think we have observed:
There is a problem with cold vs. varm starts of the application. Apparently, after a reboot of the client PC - things are really, really bad - and it takes (sometimes) up to 30-40 secs to start the application (until we arrive at the start GUI in our app).
If we run our application, close it down, and then restart
without rebooting, things are a lot better. It then usually takes
something like 15 - 20 sec. which is "acceptable". Not good, but acceptable,
Any ideas why?
I have googled it, and some links seems to suggest that the reason could be disk cache. Where vital jar are already in disk cache on th warm start? Does that make any sense? Virus scanners presumable runs in both cases.
People still think that 15 - 20 sec in start up on the warm start is an awful long time, even though there is a lot, a lot, of functionality in the application.
We got a suggestion to use IBMs JRE - as it can do some tricks (not sure what) our SUN JRE cant do concerning the warm and cold start problem. But thats is not an option for us. And noone has come up with any really good suggestions with the SUN JRE so far?
On the Java Quick Starter (JQS) -
improves initial startup time for most java applets and applications.
Which might be helpful? People on the internet seem more interested
in uninstalling the thing than actually installing it though?
And it seems very proprietary, where we cant give our Jar files to it?
We could obviously try to "hide" the problem in some way and make it "seem" quicker. Where perceived performance can be just as good as actual performance. But it does seem a bad solution. So for the cold start we will probably try reading the jar files and thereby have them in disk cache before startup of our application. And see if that helps us.
Still, ok the cold start is the real killer, but warm start isn't exactly wonderfull either.
People have suggested that we read more on the JVM and performance.
java.sun.com.javase/technologies/performance.jsp
java.sun.com.docs/hotspot/gc5.0/gc_tuning_5.html
the use of JVM flags "-Xms" "-Xmx" etc etc.
And here comes the question .. da da ...
Concerning various suggested reading material.
it is very much appreciated - but we will like to ask people here - if it is possibe to get more specific pointers. to where the gold might be buried.
I.e. in a an ideal world we would have time to read and understand all of these documents in depth. However, in this less than ideal world we are also doing a lot of very timeconsuming profiling in our own java code.
E.g. java garbage collection is is a huge subject - and JVm settings also. Sure, in the end we will probably have to do this all very thoroughly. But for now we are hoping for some heuristics on what other people are doing when facing a problem like ours..?
Young generation, large memory pages, garbage collection threads ect. all sounds interesting - but what would you start with?
If you don't have info to decide - what kind of profiling would you be running and then adjust what JVM setting in your trials?
In this pressed for time scenario. Ignorance is not bliss. But makes it hard to pinpoint the or those JVM parameters to adjust. So some good pointers from experienced JVM "configurators" will be much appreciated!
Actually, If we can establish that finetuning of these parameters is a good idea, it will certainly also be much easier to allocate the time for doing so. - reading, experimenting etc. in our project.
So, All in all , what kinds of performance improvements can we hope for? 5 out of 20 secs on the warm start? Or is it 10 % nitpicking? Whats the ball park figure for what we can hope to achieve here given our setup? What do you think based on above?
Maybe someone out there have done some finetuning of JVM parameters in a similiar PC environments like, with similiar fat clients...? Finetuning so and so - gave 5 secs. So start your work with these one-two parameters?
Something like that - some best practices? Thats what we are hoping for.
best wishes
-Simon

Answer 1

Thanks for helpful answer from both you and kajbj.
The app doesn't use shared network drives.
What are you doing between main starts to get executed and the UI is
displayed?
Basicly, Calculating what to show in the UI. Accessing server - not so much, there are some reads from a cache, but the profiling doesnt indicate that it should be a problem. Sure, I could shift the startup time to some other slot, but sofar I havent found a place where the end-user wouldnt be annoyed.> Caching of something would seem most obvious. Normal VM stuff >seems unlikely. With profiling i basicly find that ''everything'' takes a lot longer in the cold start scenario. Some of our local java methods are going to be rewritten following our review. But what else can be tuned?You guys dont think the Java Quick Start approach, with more jars in disk cache will give something? And how should that be done/ what does people do?I.e. For the class loader I read something about
1.Bootstrap class loader
2.Extensions class loader
3.System class loader
and is wondering if this has something to do with the cold start problem?
The extensions class loader loads the code in the extensions directories (<JAVA_HOME>/lib/ext
So, we should move app classes to ext? Put them in one jar file? (We have many). Best practice about that?
Otherwise it seems to me that it must be about finetuning the JVM?
I imagine that it is a question about:
1. the right heap size
2. the right garbage collection scheme
Googling heap size for XP
CHE22 writes:
You are right; -Xms1600M works well, but -Xms1700M bombs
Thats one best practice or what?
On garbage collection, there are numerous posts, and much "masters of Java black art" IMHO, And according to profiling GC is not really that much of a problem anyway? Still,
Based on my description I was hoping for a short reply like "try setting these two parameters on your xp box, it worked for me" ...or something like that. With no takers on that one, I fear people are saying that there is nothing to be gained there?
we read:
[ -Xmx3800m -Xms3800m
Configures a large Java heap to take advantage of the large memory system.
-Xmn2g
Configures a large heap for the young generation (which can be collected in parallel), again taking advantage of the large memory system. It helps prevent short lived objects from being prematurely promoted to the old generation, where garbage collection is more expensive.
Unless you have problems with pauses, try granting as much memory as possible to the virtual machine. The default size (64MB) is often too small.
Setting -Xms and -Xmx to the same value increases predictability by removing the most important sizing decision from the virtual machine. On the other hand, the virtual machine can't compensate if you make a poor choice.
The -XX:+AggressiveHeap+ option inspects the machine resources (size of memory and number of processors) and attempts to set various parameters to be optimal for long-running, memory allocation-intensive jobs]
So is Setting -Xms and -Xmx and -XX:AggressiveHeap
best practice? What kind of performance improvement should we expect?
Concerning JIT:
I read this one
[the impact of the JIT compiler is obvious on the graph: at startup the time taken is around 500us for the first few values, then quickly drops to 130us, before falling again to 70us, where it stays for 30 minutes,
for this specific issue, I greatly improved my performances by configuring another VM argument: I set -XX:CompileThreshold=50]
The size of the cache can be changed with
-Xmaxjitcodesize
This sounds like you should do something with JIT args, but reading
// We disable the JIT during toolkit initialization. This
// tends to touch lots of classes that aren't needed again
// later and therefore JITing is counter-productiive.
java.lang.Compiler.disable();
However, finding
the sweet spots for compilation thresholds has been tricky, so we're
still experimenting with the recompilation policy. Work on it
continues.
sounds like there is no such straigth forward path, it all depends...
Ok, its good, when
[Small methods that can be more easily analyzed, optimized, and inlined where necessary (and not inlined where not necessary). Clearly delineated uses of data so that usage patterns and lifetimes are apparent. ]
but when I read this:
[The virtual machine is responsible for byte code execution, storage allocation, thread synchronization, etc. Running with the virtual machine are native code libraries that handle input and output through the operating system, especially graphics operations through the window system. Programs that spend significant portions of their time in those native code libraries will not see their performance on HotSpot improved as much as programs that spend most of their time executing byte codes.]
I have the feeling that we might not able to improve performance that way?
Any comments?
otherwise i was wondering about
-XX:CompileThreshold=50 -Xmaxjitcodesize (large, how large?)
Somehow, we still feel that someone out there should have experienced similiar problems? But obviously there is no guarantee that the someone should surf by here!
In c++ we used to just write everything ourselves. Here it does seem to be a question about the right use of other peoples stuff?
Where you are kind of hoping for a shortcut, so you dont have to read endless number of documents, but can find a short document that actually addresses your problem ... well.
-Simon
Edited by: simoncpm on Mar 15, 2010 3:43 PM
Edited by: simoncpm on Mar 15, 2010 3:53 PM