Scalability Issues - Too Many Active Sessions?

Question

Hello,
I'm having an issue with an application I built for one of the campuses at the college I work at. The application is a queuing system where there are stations for students to check in, admin stations where staff can see these students and "call" them, and displays outside each employees office that shows the student that was called. There are about 20 of these last type of display panels. I have the following code in my page footer to poll the DB for the most recent called student for a specific room:
<script type="text/javascript">
<!--
var refresh_region = function( workstation_in, div_in ) {
    $.get(
        'wwv_flow.show',
        {"p_request"      : 'APPLICATION_PROCESS=F_NEXT_STUDENT',
         "p_flow_id"      : $v('pFlowId'),      //app id
         "p_flow_step_id" : $v('pFlowStepId'),  //page id
         "p_instance"     : $v('pInstance'),    //session id
         "x01"            : workstation_in
        function(data) {
            $(div_in).html(data);
    setTimeout(function() { refresh_region( workstation_in, div_in ) }, 5000);
refresh_region( '&P7_WORKSTATION_IN.', '#next_student_div' );
//-->
</script>The OnDemand process, F_NEXT_STUDENT runs the following query and returns the result:
select a.FIRST_NAME || ' ' || a.LAST_NAME
into   full_name
from   ONESTOP_QUEUE a
where  a.WORKSTATION_ID_CALLED = in_workstation_id
and    a.STATUS = 'CALLED'
and    a.QUEUE_ID = (
   select min( c.QUEUE_ID )
   from   ONESTOP_QUEUE c
   where  c.WORKSTATION_ID_CALLED = in_workstation_id
and    c.STATUS = 'CALLED');However, when all of these display panels are turned on (and I use code like this in other pages for similar purposes) the application becomes sluggish and eventually unresponsive. At first we had the application running off a box with Oracle XE. We eventually migrated to a full blown 11g install with APEX Listener and GlassFish. My DBA says everything looks ok on the DB side so I've been trying to dig in other areas to see where the bottleneck may be. After inspecting the Active Sessions report in APEX, I saw that there's a ton of connections being generated (> 30,000). This doesn't seem like a good thing to me and I'm trying to figure out what I'm doing wrong.
At first I was using $.post() instead of $.()get. I was also using setInterval() instead of a setTimeout() loop. However, none of these changes seemed to really help the situation much. I'm at a loss for how else to improve the performance of this application. Any suggestions on what I can try?
Most of the app's functionality is on apex.oracle.com
WORKSPACE: SCCC_TEST
USER/PASS: TEST/test
Direct URL to the page (I pass in the worksation ID): http://apex.oracle.com/pls/apex/f?p=65890:7:0::::P7_WORKSTATION_IN:ADMISSIONS_1
Thanks in advance for any help.

Answer 1

Hi Patrick,
UPDATE as of 3PM Eastern:
This afternoon all users lost the ability to connect to the application. My DBA is still reviewing logs but it seems that the error isn't on the DB side. The application came back up after he restarted the Apex listener. We found a bunch of the following error in the Glassfish server.log file:
[#|2013-02-25T14:34:39.021-0500|WARNING|oracle-glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=11;_ThreadName=Thread-2;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-80(73).|#]The max threads is currently set to 100.
After we came back up I went to page 4350:45 and cleared out all sessions. After a couple minutes I rechecked the number of sessions on this page:
Total Sessions: 27,674
Distinct Users over all sessions = 2
Sessions older than 15 minute(s) = 4Seems like way too many sessions to have after just a couple minutes.
End UPDATE
Again, thank you for taking the time to reply. Everything seems to be working fine for the past couple days, but I figured I'd provide some current data, especially since I'm still curious about all these "sessions".
Are we talking about page 4350:45 which shows the following information
Total Sessions: 9
Distinct Users over all sessions = 4
Sessions older than 1 day(s) = 0
Where does it show 17,400 sessions for you? It almost appears that your daily APEX jobs are not running which do normally purge old APEX sessions automatically. See http://docs.oracle.com/cd/E37097_01/doc/doc.42/e35129/dbms_jobs001.htm
Yes, this was the page I was referring to. I just checked it now and it showed me the following:
Total Sessions: 10,236
Distinct Users over all sessions = 2
Sessions older than 1 day(s) = 0And it does appear that the APEX jobs are running since there are no sessions older than 1 day... unless I'm interpreting this information incorrectly.
Also, I was able to get some more data regarding page loading using the Debug info:
14763     7751818952614     nobody     101     7     show     46     4 seconds ago     0.0000
14760     7751818952614     nobody     101     7     show     46     9 seconds ago     0.5300
14757     7751818952614     nobody     101     7     show     46     14 seconds ago     0.0150
14754     7751818952614     nobody     101     7     show     46     19 seconds ago     0.0160
14751     7751818952614     nobody     101     7     show     46     24 seconds ago     0.0160
14748     7751818952614     nobody     101     7     show     46     29 seconds ago     0.0160
14745     7751818952614     nobody     101     7     show     46     34 seconds ago     0.0160
14742     7751818952614     nobody     101     7     show     46     39 seconds ago     0.0160
14739     7751818952614     nobody     101     7     show     46     44 seconds ago     0.0160
14736     7751818952614     nobody     101     7     show     46     49 seconds ago     0.0160
14733     7751818952614     nobody     101     7     show     46     54 seconds ago     0.0160
14730     7751818952614     nobody     101     7     show     46     59 seconds ago     0.0000
14727     7751818952614     nobody     101     7     show     46     64 seconds ago     0.0160
14724     7751818952614     nobody     101     7     show     46     69 seconds ago     0.0160
14721     7751818952614     nobody     101     7     show     46     74 seconds ago     0.0160
14718     7751818952614     nobody     101     7     show     46     79 seconds ago     0.0160
14715     7751818952614     nobody     101     7     show     46     84 seconds ago     0.0150
14712     7751818952614     nobody     101     7     show     46     89 seconds ago     0.5300
14709     7751818952614     nobody     101     7     show     46     94 seconds ago     0.0000
14706     7751818952614     nobody     101     7     show     46     99 seconds ago     0.0150
14703     7751818952614     nobody     101     7     show     46     104 seconds ago     0.0150
14700     7751818952614     nobody     101     7     show     46     109 seconds ago     0.0150
14697     7751818952614     nobody     101     7     show     46     114 seconds ago     0.0150
14694     7751818952614     nobody     101     7     show     46     119 seconds ago     0.0160
14691     7751818952614     nobody     101     7     show     46     2 minutes ago     0.5310
14688     7751818952614     nobody     101     7     show     46     2 minutes ago     0.5300
14685     7751818952614     nobody     101     7     show     46     2 minutes ago     0.5150
14682     7751818952614     nobody     101     7     show     46     2 minutes ago     0.5300
14679     7751818952614     nobody     101     7     show     46     2 minutes ago     0.5300
14676     7751818952614     nobody     101     7     show     46     2 minutes ago     0.5300
14673     7751818952614     nobody     101     7     show     46     3 minutes ago     0.0000
14670     7751818952614     nobody     101     7     show     46     3 minutes ago     0.5930
14667     7751818952614     nobody     101     7     show     46     3 minutes ago     0.5300
14664     7751818952614     nobody     101     7     show     46     3 minutes ago     0.5460So I'm seeing a page load time of ~0.016 or ~0.53. When I click on the details for one of the longer page view, I get the following:
0.00000     0.00000     S H O W: application="101" page="7" workspace="" request="APPLICATION_PROCESS=F_NEXT_STUDENT" session="7751818952614"     4
0.00000     0.04700     Reset NLS settings     4
0.04700     0.03100     alter session set NLS_LANGUAGE="AMERICAN"     4
0.07800     0.03100     alter session set NLS_TERRITORY="AMERICA"     4
0.10900     0.01600     alter session set NLS_CALENDAR="GREGORIAN"     4
0.12500     0.03100     alter session set NLS_SORT="BINARY"     4
0.15600     0.00000     alter session set NLS_COMP="BINARY"     4
0.15600     0.00000     ...NLS: Set Decimal separator="."     4
0.15600     0.00000     ...NLS: Set NLS Group separator=","     4
0.15600     0.00000     ...NLS: Set g_nls_date_format="DD-MON-RR"     4
0.15600     0.00000     ...NLS: Set g_nls_timestamp_format="DD-MON-RR HH.MI.SSXFF AM"     4
0.15600     0.03100     ...NLS: Set g_nls_timestamp_tz_format="DD-MON-RR HH.MI.SSXFF AM TZR"     4
0.18700     0.00000     NLS of database and client differs, characterset conversion needed     4
0.18700     0.01600     ...Setting session time_zone to -05:00     4
0.20300     0.03100     Reset NLS settings     4
0.23400     0.03100     alter session set NLS_LANGUAGE="AMERICAN"     4
0.26500     0.01600     alter session set NLS_TERRITORY="AMERICA"     4
0.28100     0.03100     alter session set NLS_CALENDAR="GREGORIAN"     4
0.31200     0.03100     alter session set NLS_SORT="BINARY"     4
0.34300     0.00000     alter session set NLS_COMP="BINARY"     4
0.34300     0.00000     ...NLS: Set Decimal separator="."     4
0.34300     0.00000     ...NLS: Set NLS Group separator=","     4
0.34300     0.00000     ...NLS: Set g_nls_date_format="DD-MON-RR"     4
0.34300     0.00000     ...NLS: Set g_nls_timestamp_format="DD-MON-RR HH.MI.SSXFF AM"     4
0.34300     0.01600     ...NLS: Set g_nls_timestamp_tz_format="DD-MON-RR HH.MI.SSXFF AM TZR"     4
0.35900     0.03100     ...Setting session time_zone to -05:00     4
0.39000     0.03100     Setting NLS_DATE_FORMAT to application date format: DD-MON-YYYY HH:MIPM     4
0.42100     0.01600     Setting NLS_TIMESTAMP_FORMAT to application timestamp format: DD-MON-YYYY HH:MIPM     4
0.43700     0.03100     Setting NLS_TIMESTAMP_TZ_FORMAT to application timestamp time zone format: DD-MON-YYYY HH:MIPM     4
0.46800     0.00000     ...NLS: Set g_nls_date_format="DD-MON-YYYY HH:MIPM"     4
0.46800     0.00000     ...NLS: Set g_nls_timestamp_format="DD-MON-YYYY HH:MIPM"     4
0.46800     0.00000     ...NLS: Set g_nls_timestamp_tz_format="DD-MON-YYYY HH:MIPM"     4
0.46800     0.00000     NLS: wwv_flow.g_flow_language_derived_from=0: wwv_flow.g_browser_language=en     4
0.46800     0.00000     Application 101, Authentication: PLUGIN, Page Template: 61331314513900454147     4
0.46800     0.00000     Authentication check: No Authentication (NATIVE_DAD)     4
0.46800     0.00000     ...fetch session state from database     4
0.46800     0.01600     fetch items (exact)     4
0.48400     0.00000     ... sentry+verification success     4
0.48400     0.00000     ...Session ID 7751818952614 can be used     4
0.48400     0.01500     ...Application session: 7751818952614, user=nobody     4
0.49900     0.03100     ...Setting session time_zone to -05:00     4
0.53000     0.00000     Session: Fetch session header information     4
0.53000     0.00000     Run APPLICATION_PROCESS= request     4
0.53000     0.00000     ...Execute Statement: begin sys.htp.p( F_NEXT_STUDENT( in_workstation_id => apex_application.g_x01 ) ); end;     4
0.53000     0.00000     Stop APEX Engine detected     4
0.53000     -     Final commit     4Again, not sure if I'm reading this correctly but it seems that the steps that are taking the most time seem to be related to NLS settings... and I have translating turned off. This is consistent with all of the longer page views. As a side note, my DBA did turn archive log mode back on this weekend.
Again, everything seems to be running smoothly at the moment so the above data is more to help satisfy my curiosity about the inner workings of Apex.
Regards,
Tadeusz
Edited by: tdsacilowski on Feb 25, 2013 3:04 PM