Frequent heap dumps and system load average

Question

Dear Experts,
Performance of our development has gone worst. There has been frequent Heap Dumps and Paging, also System Load Average '/Kernel/SAPJVM/SystemProblemReporting/System Load Average' is over 133%.
We are using NW731 with EP core.
JVM parameters arememory are according to SAP recommendations
-Xms4096m
-Xmx4096m
-XX:NewSize=1365m
-XX:MaxNewSize=1365m
-XX:PermSize=1024m
-XX:MaxPermSize=1024m
-XX:NewRatio=6
SCS, Application and Database are on single host.
Number of nodes=2.
Physical memory size is 15GB
Appreciate your assistance in analyzing and fixing the performance issue.
Thanks,
Kumar

Answer 1

Hi Divyanshu,
I was away on to another project for configuring change control tool.
As mentioned earlier I don't have OS access.
But could able to download Thread Dumps via NWA as mentioned by David...., but couldn't able to open them.
Opened OOM.hprof file through MAT to check leak suspects etc. I could hardly understand as it is my first Java AS project ....here is bits and pieces from the dump analyzer tool.
Leap Suspects:
Problem Suspect 1:
The thread com.sap.engine.core.thread.execution.CentralExecutor$SingleThread @ 0x58dcd248 HTTP Worker [@662359980] keeps local variables with total size 3,66,84,54,240 (89.70%) bytes.
The memory is accumulated in one instance of "java.lang.Object[]" loaded by "<system class loader>".
The stacktrace of this Thread is available. See stacktrace.
And here is the stacktrace...
HTTP Worker [@662359980]
  at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:25)
  at java.lang.reflect.Array.newInstance(Ljava/lang/Class;I)Ljava/lang/Object; (Array.java:52)
  at oracle.jdbc.driver.BufferCache.get(Ljava/lang/Class;I)Ljava/lang/Object; (BufferCache.java:226)
  at oracle.jdbc.driver.PhysicalConnection.getByteBuffer(I)[B (PhysicalConnection.java:7665)
  at oracle.jdbc.driver.T4C8TTIClob.read([BJJZ[CI)J (T4C8TTIClob.java:226)
  at oracle.jdbc.driver.T4CConnection.getChars(Loracle/sql/CLOB;JI[C)I (T4CConnection.java:3211)
  at oracle.sql.CLOB.getChars(JI[C)I (CLOB.java:459)
  at oracle.jdbc.driver.OracleClobReader.needChars(I)Z (OracleClobReader.java:187)
  at oracle.jdbc.driver.OracleClobReader.read([CII)I (OracleClobReader.java:142)
  at java.io.Reader.read([C)I (Reader.java:123)
  at com.sap.sql.jdbc.common.CommonWrappedReader.read([C)I (CommonWrappedReader.java:108)
  at com.technidata.core.persistency.sql.QueryResult.getString(Ljava/lang/String;)Ljava/lang/String; (QueryResult.java:190)
  at com.technidata.em.facility.controller.PC_EmissionRes.getEmissionResBo(Lde/technidata/ecs/persist/sql/persistency/iface/IQueryResult;)Lcom/technidata/em/facility/detail/EmissionResBo; (PC_EmissionRes.java:2235)
  at com.technidata.em.facility.controller.PC_EmissionRes.readData(JJJJLjava/sql/Date;Ljava/sql/Date;Ljava/lang/String;ZJI)Ljava/util/List; (PC_EmissionRes.java:682)
  at com.technidata.em.facility.controller.PC_EmissionRes.readData(JJJLjava/sql/Date;Ljava/sql/Date;ZI)Ljava/util/List; (PC_EmissionRes.java:599)
  at com.technidata.em.facility.controller.PC_EmissionRes.readData(Lcom/technidata/em/facility/TypeCapArgument;)Ljava/util/List; (PC_EmissionRes.java:573)
  at com.technidata.em.facility.ESourceBo.getTypeCapData(Ljava/lang/String;Lcom/technidata/em/facility/TypeCapArgument;Ljava/lang/String;Ljava/lang/String;)Ljava/util/List; (ESourceBo.java:707)
  at com.technidata.em.facility.mgmt.EmissionResTable.initialize(Z)V (EmissionResTable.java:1793)
  at com.technidata.em.facility.mgmt.EmissionResTable.init(Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;Ljava/lang/String;)V (EmissionResTable.java:1454)
  at com.technidata.em.facility.mgmt.wdp.InternalEmissionResTable.wdInvokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (InternalEmissionResTable.java:723)
  at com.sap.tc.webdynpro.progmodel.generation.DelegatingView.invokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (DelegatingView.java:142)
  at com.sap.tc.webdynpro.progmodel.components.Component.fireEvent(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;)V (Component.java:492)
  at com.technidata.em.facility.mgmt.wdp.InternalTabsInterface.wdFireEventInitTab(Ljava/lang/String;)V (InternalTabsInterface.java:398)
  at com.technidata.em.facility.mgmt.EmissionResWinInterfaceView.onPlugIn(Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)V (EmissionResWinInterfaceView.java:97)
  at com.technidata.em.facility.mgmt.wdp.InternalEmissionResWinInterfaceView.wdInvokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (InternalEmissionResWinInterfaceView.java:95)
  at com.sap.tc.webdynpro.progmodel.generation.DelegatingWindow.invokeEventHandler(Ljava/lang/String;Lcom/sap/tc/webdynpro/progmodel/api/IWDCustomEvent;)Ljava/lang/Object; (DelegatingWindow.java:121)
please help me fix the performance issue. And let me know if I can provide some details to help you analyse further.
Thanks,
Kumar