Ntlm authenticated apps fails after 3.1.1 upgrade

Question

I upgraded my apex instance to 3.1.1 on Friday without any issues. I can log into application builder without any problems and the version 3.1.1.00.09.
Everything in app builder works as expected. However, when I try to run my NTLM authenticated application, I get errors and the page fails to load.
Furthermore, this only happens on my 11g database.
The exact same app, using the same NTLM authentication works just fine on 10g.
The Apache errors log states:
mod_plsql: /pls/apex/f HTTP-404 ORA-03113: end-of-file on communication channel\n
mod_plsql: Unable to reset state for mode 0: Err 3114 url=>/pls/apex/f           I have PlsqlErrorStyle          DebugStyle set, so the page returns a fair amount of data.
Wed, 28 May 2008 14:07:17 GMT
ORA-03113: end-of-file on communication channel
  DAD name: apex
  PROCEDURE  : f
  URL        : http://ecydblcyorwqt03.ecy.wa.lcl:80/pls/apex/f?p=127:51:339228564056494:::::
  PARAMETERS :
  ===========
  p:
   127:51:339228564056494:::::
  ENVIRONMENT:
  ============
    PLSQL_GATEWAY=WebDb
    GATEWAY_IVERSION=2
    SERVER_SOFTWARE=Oracle-Application-Server-10g/10.1.3.1.0 Oracle-HTTP-Server
    GATEWAY_INTERFACE=CGI/1.1
    SERVER_PORT=80
    SERVER_NAME=ecydblcyorwqt03.ecy.wa.lcl
    REQUEST_METHOD=GET
    QUERY_STRING=p=127:51:339228564056494:::::
    PATH_INFO=/f
    SCRIPT_NAME=/pls/apex
    REMOTE_HOST=
    REMOTE_ADDR=165.151.57.100
    SERVER_PROTOCOL=HTTP/1.1
    REQUEST_PROTOCOL=HTTP
    REMOTE_USER=ECY\taus461
    ORACLE_SSO_USER=
    OSSO_IDLE_TIMEOUT_EXCEEDED=
    OSSO_USER_GUID=
    HTTP_CONTENT_LENGTH=
    HTTP_CONTENT_TYPE=
    HTTP_USER_AGENT=Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14
    HTTP_HOST=ecydblcyorwqt03
    HTTP_ACCEPT=text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    HTTP_ACCEPT_ENCODING=gzip,deflate
    HTTP_ACCEPT_LANGUAGE=en-us,en;q=0.5
    HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.7
    HTTP_COOKIE=WEBWPLCS_USER=TAUS461; WEBWPLCS_LAST=04.29.2008 11:41:38; ORA_WWV_R1=%23ALL; ORA_WWV_R2=%23ALL; ORA_WWV_R3=%23ALL; ORA_WWV_REMEMBER_UN=ADMIN:webwplcs; ORACLE_PLATFORM_REMEMBER_UN=ADMIN:webwplcs; ORA_WWV_USER=3B1A5D9EA835D646; WWV_CUSTOM-F_1021906798187125_122=9F806B35C3D9AF51
    HTTP_IF_MODIFIED_SINCE=
    HTTP_REFERER=http://ecydblcyorwqt03/pls/apex/f?p=4000:4150:339228564056494::NO:::
    HTTP_SOAPACTION=
    HTTP_ORACLE_ECID=1211983633:165.151.5.125:6156:6252:488,0
    HTTP_ORACLE_CACHE_VERSION=
    HTTP_AUTHORIZATION=NTLM  xyz
    WEB_AUTHENT_PREFIX=
    DAD_NAME=apex
    DOC_ACCESS_PATH=docs
    DOCUMENT_TABLE=wwv_flow_file_objects$
    PATH_ALIAS=
    REQUEST_CHARSET=AL32UTF8
    REQUEST_IANA_CHARSET=UTF-8
    SCRIPT_PREFIX=/pls
    HTTP_IF_MATCH=
    HTTP_CACHE_CONTROL=
    SOAP_BODY=
    HTTP_X_ORACLE_DEVICE_CLASS=
    HTTP_X_ORACLE_DEVICE_ORIENTATION=
    HTTP_X_ORACLE_DEVICE_MAXDOCSIZE=
    HTTP_X_ORACLE_DEVICE=
    HTTP_X_ORACLE_ORIG_ACCEPT=
    HTTP_X_ORACLE_ORIG_USER_AGENT=
    HTTP_X_ORACLE_USER_LOCALE=
    HTTP_X_ORACLE_USER_NAME=
    HTTP_X_ORACLE_USER_DISPLAYNAME=
    HTTP_X_ORACLE_USER_USERKIND=
    HTTP_X_ORACLE_USER_AUTHKIND=
    HTTP_X_ORACLE_USER_DEVICEID=
    HTTP_X_ORACLE_USER_LOCATION_ADDRESSLINE1=
    HTTP_X_ORACLE_USER_LOCATION_ADDRESSLINE2=
    HTTP_X_ORACLE_USER_LOCATION_ADDRESSLASTLINE=
    HTTP_X_ORACLE_USER_LOCATION_BLOCK=
    HTTP_X_ORACLE_USER_LOCATION_CITY=
    HTTP_X_ORACLE_USER_LOCATION_COMPANYNAME=
    HTTP_X_ORACLE_USER_LOCATION_COUNTY=
    HTTP_X_ORACLE_USER_LOCATION_STATE=
    HTTP_X_ORACLE_USER_LOCATION_POSTALCODE=
    HTTP_X_ORACLE_USER_LOCATION_POSTALCODEEXT=
    HTTP_X_ORACLE_USER_LOCATION_COUNTRY=
    HTTP_X_ORACLE_USER_LOCATION_TYPE=
    HTTP_X_ORACLE_USER_LOCATION_X=
    HTTP_X_ORACLE_USER_LOCATION_Y=
    HTTP_X_ORACLE_SERVICE_HOME_URL=
    HTTP_X_ORACLE_SERVICE_PARENT_URL=
    HTTP_X_ORACLE_HOME_URL=
    HTTP_X_ORACLE_MODULE_CALLBACK_URL=
    HTTP_X_ORACLE_MODULE_CALLBACK_LABEL=
    HTTP_X_ORACLE_CACHE_USER=
    HTTP_X_ORACLE_CACHE_SUBID=
    HTTP_X_ORACLE_CACHE_AUTH=
    HTTP_X_ORACLE_CACHE_DEVICE=
    HTTP_X_ORACLE_CACHE_LANG=
    HTTP_X_ORACLE_CACHE_ENCRYPT=
    HTTP_X_ORACLE_ASSERT_USER=There are no invalid objects in the FLOWS schema and the page sentry function I use for NTLM is also valid.
There isn't a database connection issue since both builder and SQL Plus works.
Here is my NTLM Page Sentry which is a slightly modified version of the GreenIT version
CREATE OR REPLACE FUNCTION modNtlmPageSentry(pApexUser IN VARCHAR2 DEFAULT 'APEX_PUBLIC_USER')
RETURN BOOLEAN
IS
  vAuthenticatedUsername  VARCHAR2(512);
  vCurrentSessionId       NUMBER;
  l_cnt binary_integer :=0;
BEGIN
  -- Get Authenticated User.
  vAuthenticatedUsername := UPPER(owa_util.get_cgi_env('REMOTE_USER'));
  vAuthenticatedUsername := substr(vAuthenticatedUsername,instr(vAuthenticatedUsername,'\')+1);
  if to_char(v('APP_ID')) = '127' -- WebWPLCS
  then
       apex_util.set_session_state('P18_USERNAME',vAuthenticatedUsername);
  elsif to_char(v('APP_ID')) = '124' --TMS
  then
  -- check to see if they are a listed TMS manager or overall admin
      select sum(cnt) into l_cnt
      from (
           select count(0) cnt
           from tms_managers
           where username=vAuthenticatedUsername
           union
           select count(0) cnt
           from tms_admin
           where username=vAuthenticatedUsername
           union
           select count(0) cnt
           from web_admin
           where username=vAuthenticatedUsername
      if l_cnt < 1
      then
     return FALSE;
      end if;
  end if;
  -- Check to ensure that we are running as the correct database user.
  IF USER ^= UPPER(pApexUser) THEN
    RETURN FALSE;
  END IF;
  IF vAuthenticatedUsername IS NULL THEN
    RETURN FALSE;
  END IF;
  -- Get SessionId.
  vCurrentSessionId := wwv_flow_custom_auth_std.get_session_id_from_cookie;
  -- Check Application Session Cookie.
  IF wwv_flow_custom_auth_std.is_session_valid THEN
    apex_application.g_instance := vCurrentSessionId;
    -- Check Authenticated User --> Username from wwv_flow_session$ for
    --   current Session.
    IF vAuthenticatedUsername = wwv_flow_custom_auth_std.get_username THEN
      wwv_flow_custom_auth.define_user_session(p_user => vAuthenticatedUsername,
        p_session_id => vCurrentSessionId);
      RETURN TRUE;
    ELSE
      -- Unset the Session Cookie and redirect back here to take other branch.
      wwv_flow_custom_auth_std.logout(p_this_flow => v('FLOW_ID'),
        p_next_flow_page_sess => v('FLOW_ID') || ':' || NVL(v('FLOW_PAGE_ID'), 0)
        || ':' || vCurrentSessionId);
      -- Tell Apex Engine to quit.
      apex_application.g_unrecoverable_error := TRUE;
      RETURN FALSE;
    END IF;
  ELSE
    -- Application Session Cookie not valid --> Define a new Apex Session.
    wwv_flow_custom_auth.define_user_session(p_user => vAuthenticatedUsername,
      p_session_id => wwv_flow_custom_auth.get_next_session_id);
    -- Tell Apex Engine to quit.
    apex_application.g_unrecoverable_error := TRUE;
    IF owa_util.get_cgi_env('REQUEST_METHOD') = 'GET'  THEN
      wwv_flow_custom_auth.remember_deep_link(p_url => 'f?' ||
        wwv_flow_utilities.url_decode2(owa_util.get_cgi_env('QUERY_STRING')));
    ELSE
      wwv_flow_custom_auth.remember_deep_link(p_url => 'f?p=' ||
        TO_CHAR(apex_application.g_flow_id) || ':' ||
        TO_CHAR(NVL(apex_application.g_flow_step_id, 0)) || ':' ||
        TO_CHAR(apex_application.g_instance));
    END IF;
    -- Register the Session in Apex Sessions Table, set Cookie, redirect back.
    wwv_flow_custom_auth_std.post_login(p_uname => vAuthenticatedUsername,
      p_session_id => nv('APP_SESSION'), p_flow_page => apex_application.g_flow_id
      || ':' || NVL(apex_application.g_flow_step_id, 0));
    RETURN FALSE;       
  END IF;   
END modNtlmPageSentry;Does anyone have any ideas on where to look next?
Regards, Tony
<b>Update</b>
For kicks, I added the page sentry function to the list in the <b>wwv_flow_epg_include_mod_local</b> function.
I bounced both the HTTP Server and the database.
None of these actions solved the problem.

Answer 1

Joel -
The alert log states that there is a 7445 error now from Apache
host_id='ECYDBLCYORWQT01' host_addr='165.151.5.123' module='Apache.exe'
pid='416'>
<txt>Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0x0] [PC:0x69A2AB3, _pfrinstr_BRNCCOND()+39]
msg_id='1422874948' type='INCIDENT_ERROR' group='Access Violation'
level='1' host_id='ECYDBLCYORWQT01' host_addr='165.151.5.123'
prob_key='ORA 7445 [pfrinstr_BRNCCOND()+39]' upstream_comp='' downstream_comp=''
ecid='' errid='12252' ORA-07445: exception encountered: core dump [pfrinstr_BRNCCOND()+39] [ACCESS_VIOLATION] [ADDR:0x0] [PC:0x69A2AB3] [UNABLE_TO_READ] []The trace file just states the same 7445 error:
ORA-07445: exception encountered: core dump [pfrinstr_BRNCCOND()+39] [ACCESS_VIOLATION] [ADDR:0x0] [PC:0x69A2AB3] [UNABLE_TO_READ] []The incident trace file states that the current SQL was:
----- Current SQL Statement for this session (sql_id=bng4udk9mvtsh) -----
declare function x return boolean is begin
return mergedwplcs.modNtlmPageSentry; return false; end;
begin
wwv_flow.g_boolean := x; end;
----- PL/SQL Stack -----
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
2B6ACD34      1020  package body FLOWS_030100.WWV_FLOW_CUSTOM_AUTH_STD
2B6ACD34       662  package body FLOWS_030100.WWV_FLOW_CUSTOM_AUTH_STD
2B6BB44C        59  function MERGEDWPLCS.MODNTLMPAGESENTRY
2B6BBD1C         2  anonymous block
2B6BBD1C         4  anonymous block
2B6BC674      1815  package body SYS.DBMS_SYS_SQL
2B6BD29C       296  package body SYS.WWV_DBMS_SQL
2B70B5D0      1352  package body FLOWS_030100.WWV_FLOW_SECURITY
2B70B5D0      1158  package body FLOWS_030100.WWV_FLOW_SECURITY
2B71BA2C      8847  package body FLOWS_030100.WWV_FLOW
2B72FB04       255  procedure FLOWS_030100.F
2B7E4F1C        31  anonymous blockWhich makes sense given that I was trying to log into the application. All of these functions and packages are valid.