Oracle BPM 11.1.1.5 Performance issues

Question

Hi,
I have 2 Node Cluster of Oracle SOA 11.1.1.5 installed. Have a separate cluster for SOA/BPM, BAM, OSB, WSM. Here are the OS and Java versions
Java Vendor: HP
Java Version: 1.6.0.14
OS: HP-UX
OS Version: B.11.31
Running into an issue where Oracle BPM is performing very slow the task forms takes forever to come up and performing any actions takes too long to proceed som time it keeps timing out. I have over 100 SOA Process running plus some BPM Process but BMP Workspace application is running too slow as have customized tasks forms. But the same works a bit better in other instance which is non-clustered. Any idea what can be done on the server level to get around the performance issues. So far have modified Audit Levels and reduced soa-infra bpm log have the following settings in setDomainEnv.sh for USER_MEM_ARGS=-Xms2000m -Xmx6000m -XX:PermSize=1000m
But not help. Any idea what else to look into to get around these bpm performance issues. Here is what I have in setSOADomainEnv.sh
# 8395254: add -da:org.apache.xmlbeans... in EXTRA_JAVA_PROPERTIES
EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -da:org.apache.xmlbeans..."
XENGINE_DIR="${SOA_ORACLE_HOME}/soa/thirdparty/edifecs/XEngine"
DEFAULT_MEM_ARGS="-Xms512m -Xmx1024m"
PORT_MEM_ARGS="-Xms768m -Xmx1536m"
if [ "${JAVA_VENDOR}" != "Oracle" ] ; then
  DEFAULT_MEM_ARGS="${DEFAULT_MEM_ARGS} -XX:PermSize=128m -XX:MaxPermSize=512m"
  PORT_MEM_ARGS="${PORT_MEM_ARGS} -XX:PermSize=256m -XX:MaxPermSize=512m"
fi
#========================================================
# setup LD_LIBRARY_PATH if directory is present...
#========================================================
if [ -d ${XENGINE_DIR}/bin ]; then
   LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${XENGINE_DIR}/bin"
   export LD_LIBRARY_PATH
fi
#========================================================
# setup platform specific environment variables
#========================================================
case ${PLATFORM_TYPE} in
  # AIX
  AIX)
    if [ -d ${XENGINE_DIR}/bin ]; then
       LIBPATH="${LIBPATH}:${XENGINE_DIR}/bin"
       export LIBPATH
    fi
    USER_MEM_ARGS=${PORT_MEM_ARGS}
    export USER_MEM_ARGS
    # Fix for 7828060
    POST_CLASSPATH=${POST_CLASSPATH}:${SOA_ORACLE_HOME}/soa/modules/soa-ibm-addon.jar
    # Fix for 7520915 and 8264518 and 8305217
    EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Djavax.xml.datatype.DatatypeFactory=org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl -Djava.endorsed.dirs=${SOA_ORACLE_HOME}/bam/modules/org.apache.xalan_2.7.1"
    export EXTRA_JAVA_PROPERTIES
  # HPUX
  HP-UX)
    if [ -d ${XENGINE_DIR}/bin ]; then
       SHLIB_PATH="${SHLIB_PATH}:${XENGINE_DIR}/bin"
       export SHLIB_PATH
    fi
    LD_LIBRARY_PATH="${XENGINE_DIR}/bin:${LD_LIBRARY_PATH}"
    export LD_LIBRARY_PATH
    USER_MEM_ARGS="-d64 ${PORT_MEM_ARGS}"
    export USER_MEM_ARGS
    ;;And here is what I have in setDomainEnv.sh
XMS_SUN_64BIT="256"
export XMS_SUN_64BIT
XMS_SUN_32BIT="256"
export XMS_SUN_32BIT
XMX_SUN_64BIT="512"
export XMX_SUN_64BIT
XMX_SUN_32BIT="512"
export XMX_SUN_32BIT
XMS_JROCKIT_64BIT="256"
export XMS_JROCKIT_64BIT
XMS_JROCKIT_32BIT="256"
export XMS_JROCKIT_32BIT
XMX_JROCKIT_64BIT="512"
export XMX_JROCKIT_64BIT
XMX_JROCKIT_32BIT="512"
export XMX_JROCKIT_32BIT
if [ "${JAVA_VENDOR}" = "Sun" ] ; then
     WLS_MEM_ARGS_64BIT="-Xms256m -Xmx512m"
     export WLS_MEM_ARGS_64BIT
     WLS_MEM_ARGS_32BIT="-Xms256m -Xmx512m"
     export WLS_MEM_ARGS_32BIT
else
     WLS_MEM_ARGS_64BIT="-Xms512m -Xmx512m"
     export WLS_MEM_ARGS_64BIT
     WLS_MEM_ARGS_32BIT="-Xms512m -Xmx512m"
     export WLS_MEM_ARGS_32BIT
fi
if [ "${JAVA_VENDOR}" = "Oracle" ] ; then
     CUSTOM_MEM_ARGS_64BIT="-Xms${XMS_JROCKIT_64BIT}m -Xmx${XMX_JROCKIT_64BIT}m"
     export CUSTOM_MEM_ARGS_64BIT
     CUSTOM_MEM_ARGS_32BIT="-Xms${XMS_JROCKIT_32BIT}m -Xmx${XMX_JROCKIT_32BIT}m"
     export CUSTOM_MEM_ARGS_32BIT
else
     CUSTOM_MEM_ARGS_64BIT="-Xms${XMS_SUN_64BIT}m -Xmx${XMX_SUN_64BIT}m"
     export CUSTOM_MEM_ARGS_64BIT
     CUSTOM_MEM_ARGS_32BIT="-Xms${XMS_SUN_32BIT}m -Xmx${XMX_SUN_32BIT}m"
     export CUSTOM_MEM_ARGS_32BIT
fi
MEM_ARGS_64BIT="${CUSTOM_MEM_ARGS_64BIT}"
export MEM_ARGS_64BIT
MEM_ARGS_32BIT="${CUSTOM_MEM_ARGS_32BIT}"
export MEM_ARGS_32BIT
if [ "${JAVA_USE_64BIT}" = "true" ] ; then
     MEM_ARGS="${MEM_ARGS_64BIT}"
     export MEM_ARGS
else
     MEM_ARGS="${MEM_ARGS_32BIT}"
     export MEM_ARGS
fi
MEM_PERM_SIZE_64BIT="-XX:PermSize=128m"
export MEM_PERM_SIZE_64BIT
MEM_PERM_SIZE_32BIT="-XX:PermSize=128m"
export MEM_PERM_SIZE_32BIT
if [ "${JAVA_USE_64BIT}" = "true" ] ; then
     MEM_PERM_SIZE="${MEM_PERM_SIZE_64BIT}"
     export MEM_PERM_SIZE
else
     MEM_PERM_SIZE="${MEM_PERM_SIZE_32BIT}"
     export MEM_PERM_SIZE
fi
MEM_MAX_PERM_SIZE_64BIT="-XX:MaxPermSize=512m"
export MEM_MAX_PERM_SIZE_64BIT
MEM_MAX_PERM_SIZE_32BIT="-XX:MaxPermSize=512m"
export MEM_MAX_PERM_SIZE_32BIT
if [ "${JAVA_USE_64BIT}" = "true" ] ; then
     MEM_MAX_PERM_SIZE="${MEM_MAX_PERM_SIZE_64BIT}"
     export MEM_MAX_PERM_SIZE
else
     MEM_MAX_PERM_SIZE="${MEM_MAX_PERM_SIZE_32BIT}"
     export MEM_MAX_PERM_SIZE
fi
if [ "${JAVA_VENDOR}" = "Sun" ] ; then
     if [ "${PRODUCTION_MODE}" = "" ] ; then
          MEM_DEV_ARGS="-XX:CompileThreshold=8000 ${MEM_PERM_SIZE} "
          export MEM_DEV_ARGS
     fi
fi
# Had to have a separate test here BECAUSE of immediate variable expansion on windows
if [ "${JAVA_VENDOR}" = "Sun" ] ; then
     MEM_ARGS="${MEM_ARGS} ${MEM_DEV_ARGS} ${MEM_MAX_PERM_SIZE}"
     export MEM_ARGS
fi
if [ "${JAVA_VENDOR}" = "HP" ] ; then
     MEM_ARGS="${MEM_ARGS} ${MEM_MAX_PERM_SIZE}"
     export MEM_ARGS
fi
if [ "${JAVA_VENDOR}" = "Apple" ] ; then
     MEM_ARGS="${MEM_ARGS} ${MEM_MAX_PERM_SIZE}"
     export MEM_ARGS
fi
if [ "${debugFlag}" = "true" ] ; then
     JAVA_OPTIONS="${JAVA_OPTIONS} -da:org.apache.xmlbeans... "
     export JAVA_OPTIONS
fi
export USER_MEM_ARGS="-Xms4g -Xmx6g -XX:PermSize=2g -XX:+UseParallelGC -XX:+UseParallelOldGC"Here is the output of the top command
Load averages: 0.08, 0.06, 0.06
315 processes: 205 sleeping, 110 running
Cpu states:
CPU   LOAD   USER   NICE    SYS   IDLE  BLOCK  SWAIT   INTR   SSYS
0    0.06   4.0%   0.2%   1.0%  94.8%   0.0%   0.0%   0.0%   0.0%
2    0.07   3.6%   5.0%   0.2%  91.2%   0.0%   0.0%   0.0%   0.0%
4    0.07   2.0%   0.2%   0.0%  97.8%   0.0%   0.0%   0.0%   0.0%
6    0.07   2.4%   0.2%   0.8%  96.6%   0.0%   0.0%   0.0%   0.0%
8    0.09   1.2%   0.2%  10.9%  87.7%   0.0%   0.0%   0.0%   0.0%
10    0.12   3.0%   0.0%  11.1%  85.9%   0.0%   0.0%   0.0%   0.0%
12    0.08   3.0%   0.2%   6.6%  90.3%   0.0%   0.0%   0.0%   0.0%
14    0.09   4.2%   1.2%   0.8%  93.8%   0.0%   0.0%   0.0%   0.0%
avg   0.08   3.0%   1.0%   3.8%  92.2%   0.0%   0.0%   0.0%   0.0%
System Page Size: 4Kbytes
Memory: 38663044K (38379156K) real, 149420048K (148978096K) virtual, 26349848K f
ree  Page# 1/63
CPU TTY  PID USERNAME PRI NI   SIZE    RES STATE    TIME %WCPU  %CPU COMMAND
4   ?  3926 root     152 20   213M 70040K run    694:47 10.05 10.04 cimprovagt
4   ?  6855 user1  152 20  7704M  1125M run      4:36  9.31  9.29 java
0   ?  6126 user2  152 20  2790M  1863M run     22:57  4.16  4.15 javaHere is the Memory on the box
Memory: 98132 MB (95.83 GB)
Thanks

Answer 1

After changing JVM settigns for soa cluster it's a bit better but still slow so wondering what other tweaks can be done on the JVM side. Here is what is for the SOA Cluster
USER_MEM_ARGS="-server -Xms12928m -Xmx12928m -XX:PermSize=3072m -Xmn3232m -XX:+SXTElimination -XX:+UseParallelGC -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:-TraceClassLoading -XX:-TraceClassUnloading"
We are running multiple instances on the same boxes and the total RAM on the machine is 95 GB on each box which is being shared across 4 cluster environments. For now other 3 cluster environments have the SOA Cluster JVM setting as
USER_MEM_ARGS="-server -Xms4096m -Xmx4096m -XX:PermSize=1024m -Xmn1152m -XX:+SXTElimination -XX:+UseParallelGC -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:-TraceClassLoading -XX:-TraceClassUnloading"
Any help on what else I can tweak or set in JVM to get a better performance.
Thanks