How best to debug Weblogic 10.3 request slowness

Question

We recently migrated our app from another container to Weblogic 10.3 and are having significant performance problems. We are struggling to put our finger on the actual issue. Does anyone have any suggestions for debugging the below problem that might push us in a direction to resolve this?
Client is a standalone process and the server process is a stateless session bean in a 3 managed server cluster w/ 'random-affinity' load balancing policy (we have to use this policy due to other app requirements). When the client calls the ejb we see either the request to the server get delayed or the response back from the server get delayed. Here is example of getting delayed coming back.
Client - 14.29.20.875 findByFilter() RQID=1289593760875 - START
Server - 14:29:20.878 findByFilter() RQID=1289593760875_null - START
Server - 14:29:20.892 findByFilter() - FINISHED - 13 ms
Client - 14.29.22.879 findByFilter() - FINISHED - returned in 2004 ms.This call does a simple select against our database and we are not setting a container managed transaction strategy so no transaction is being explicitly created. The object returned from this call is only 3 KB. We don't understand where the ~ 1.9 seconds are going. They seem to be lost to WL post method processing, network latency, or something else. Has anyone run into this or has any suggestions for what to look at from a WL perspective to debug further?
Lance Johnson

Answer 1

You can use a diagnostic module to debug the timing. The diagnostic module must be placed
in the META-INF directory (also where application.xml is located) and must be called weblogic-diagnostics.xml.
An example of a diagnostic module is the following:
<?xml version="1.0" encoding="UTF-8"?>
<wldf-resource xmlns="http://xmlns.oracle.com/weblogic/weblogic-diagnostics"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-diagnostics
               http://xmlns.oracle.com/weblogic/weblogic-diagnostics/1.0/weblogic-diagnostics.xsd">
    <instrumentation>
        <enabled>true</enabled>
        <wldf-instrumentation-monitor>
            <name>AddKlantInstrumentationEJB</name>
            <action>MethodInvocationStatisticsAction</action>
            <location-type>around</location-type>
            <pointcut>execution(* datamodel.logic.BedrijfBean addKlant(...))</pointcut>
        </wldf-instrumentation-monitor>
        <wldf-instrumentation-monitor>
            <name>OnMessageInstrumentationMDB</name>
            <action>MethodInvocationStatisticsAction</action>
            <location-type>around</location-type>
            <pointcut>execution(* datamodel.logic.BedrijfMDB onMessage(...))</pointcut>
        </wldf-instrumentation-monitor>
    </instrumentation>
</wldf-resource>You also need to a create diagnostic system modules using the WebLogic Console and target the module
to the appropriate server and enable instrumentation using the Enabled checkbox on the Instrumentation Configuration tab.
The information recorded by the MethodInvocationStatisticsAction is available through the application's
WLDFInstrumentationRuntime MBean. This MBean has a MethodInvocationStatistics attribute that returns
a nested set of maps containing of all the recorded statistics for the application. It also has a getMethodInvocationStatisticsData()
operation that allows you to query for a subset of the data. The listing below is an example WLST script that
queries the WLDFInstrumentationRuntimeMBean and displays all of the results. For example
connect('weblogic','transfer11g');
cd('serverRuntime:/WLDFRuntime/WLDFRuntime/WLDFInstrumentationRuntimes/ApplicationName');
def formatTime(t):
     return "%.2f" % (t/1e6)
print('OBTAINING STATISTICS');
for classStats in cmo.getMethodInvocationStatistics().entrySet():
     print('OBTAINED THE CLASS STATISTICS');
     for methodStats in classStats.value.entrySet():
          print('OBTAINED THE METHOD STATISTICS');
          for signatureStats in methodStats.value.entrySet():
               print('PRINTING SOME OUTPUT');
               print "%s.%s(): %d %s %s %s %s" % (
                    classStats.key,
                    methodStats.key,
                    signatureStats.value['count'],
                    formatTime(signatureStats.value['min']),
                    formatTime(signatureStats.value['avg']),
                    formatTime(signatureStats.value['max']),
                    formatTime(signatureStats.value['std_deviation']),)If you run the above WLST script, you see something like the following output:
model.logic.VideotheekWizardBean.confirm(): 50 1.52 8.46 66.94 11.09
model.logic.VideotheekMDB.onMessage():      50 0.13 0.75  2.46  0.70All values are in milliseconds.
I do not know what your findByFilter method is doing, but to best pin point the problem
is divide it in individual actions (methods) it is performing which you can profile using the
above.