100% CPU load but no clue in thread dump, EP. Win a bottle of champagne...

Question

Hi,
We are suddenly facing 100% CPU load in our EP cluster (5 x 8-way Xeon multiprocessor machines). We have a serious performance problem that is burning a lot of our time and causes a lot of stress for over 3 weeks now.
We have taken tens of thread dumps from Application Nodes.
- In none of the thread dumps we see the Finalizer thread
running.
- From the garbage collector log we see that full garbage
collection runs occur very rarily (once an hour).
- We use the compacting garbage collector.
- In a lot of thread dumps we do not even see our own portal code in the stack traces of the threads!
- In Windows Task Manager we see the jlaunch.exe processes
consume all available CPU time.
- Portal users see a blank portal page when the CPU load hits 100%. When the load goes down again, things return to normal.
- There are no errors logged in any of the log files of the portal. We checked all of them. We expected a log full of errors somewhere but nothing even remotely interesting was found. Windows Event Viewer shows nothing either.
- The amount of sockets in CLOSE_WAIT status is < 10 on every machine in the cluster.
This forces me to conclude that something in the jlaunch.exe executable consumes the CPU time. This raises the following 3 questions:
- What does the mystifying jlaunch.exe do besides executing java.exe ?
- Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ? Can it be GZip compression ?
- If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?
Our development- and support teams are desperate. All suggestions are welcome. The person that comes up with
the solution to our problem gets a nice bottle of champagne.
Regards,
Chris Twigt

Answer 1

Hi,
- What does the mystifying jlaunch.exe do besides executing java.exe ?
This is so that the startupframework can connect more easily and take control of the JVM in some situations
- Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ?
I assume when SAP release their JVM (in the next major release), that will be called directly.
- If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?
Loads of thing, but there should be clues in the thread dumps.  (the reason why your code is not in those thread dumps is that your code is only active during the processing of a particular request, unless you have a service, afterwards there are no trace of it as the thread which does the processing goes back to sleep)
I've experienced a similar situation with a an 6.40 portal, and it was then caused by the following:
1. User A comes logs in and sees that some cache timeout has occured , therefore it issues a SQL query which does a full table scan on a table of approx. 1 GB (on of the UME tables)
2. User B comes in just afterwards and also sees that the cache timeout has occured, and issues the same SQL query as User A
3. User C .... and so on untill the query from User A eventually finishes
So
1. Check database. Any big queries running ?
2. Check file activity, are there a lot of writing ?
3. Check network activity, especially to the state controller which is the weakest link
4. What about portal logs ? Any activity during the hang ?
Also, please provide one of the thread dumps for further analysis..
cheers
Dagfinn