Unable to reach Workspace Login Page

Question

Hi,
First some background - I am an Apex developer / administrator. I do not have a DBA background. DBAs install the Apex software and I administer the webserver (although my experience with webservers is that which I've picked up using Apex) and develop applications.
We are currently upgrading our DEV, QA and PROD APEX installations from 2.2.1 to 3.1.1 (specifically 3.1.1.00.09).
We are using Oracle HTTP Server from the Oracle 10.1 companion disk.
We completed this for our DEV and QA environments without a problem. We have not done this for our PROD environment yet.
We also installed APEXLIB 1.7 in both DEV and QA. This was working successfully in DEV. We had not tried it yet in QA.
Both 3.1.1 upgraded environments have been running successfully for about 10 days since the upgrade.
However our QA environment (Oracle 10.2.0.3 running on HP-UX B.11.11 ) experienced a problem yesterday - we can no longer access any Apex applications including the Workspace login pages or application login pages.
The error we get in the browser is:
Expecting p_company or wwv_flow_company cookie to contain security group id of application owner.
Error ERR-7620 Could not determine workspace for application ().
We experienced a problem with our QA environment before (prior to upgrade) and we needed to flush the shared pool a few months ago as a temporary fix for this as has been noted on this forum. The problem in question was addressed in this forum post: Re: ORA-06502:PL/SQL: numeric or value error: NULL index table key value
However, with the current problem, flushing the shared pool has not solved the issue. It may not be related.
When we access the workspace login page: http://our_url/apex/crmqa/f?p=4550:1, I see nothing in the Apache logs to indicate an error.
The access_log.xxx file in the Apache logs lists this ...
ip_address_here - apex_public_user [16/Jul/2008:15:15:15 -0500] "GET /apex/crmqa/f?p=4550:1 HTTP/1.1" 200 455
The error_log.xxx file is empty.
I temporarily turned on mod_plsql logging
When I tried to access http://our_url/apex/crmqa/f?p=4550:1
The cid1.log file contained...
<337703562 ms>[ReqStartTime: 16/Jul/2008:15:23:43]
<337703562 ms>Request ID ReqID:6136_1216239823
<337703562 ms>Connecting to database with connect string : "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ip_address_here)(PORT=1526)))(CONNECT_DATA=(SID=crmqa1)))"
<337704171 ms>Doing alter session set nls_language= "AMERICAN" nls_territory= "AMERICA"
<337704171 ms>OpenCursor
<337704234 ms>Altered session to nls_language=AMERICAN nls_territory=AMERICA
<337704234 ms>DeinitCursor
<337704234 ms>OpenCursor
<337704453 ms>DBCharSet is AMERICAN_AMERICA.WE8ISO8859P1, OWAVersion 10.1.2.0.6, 1001020006 (rc=0)
<337704453 ms>DeinitCursor
<337704453 ms>OpenCursor
<337704453 ms>(wpd.c,1757) Logged in as (unknown)
<337704453 ms>(wpx.c,559) Going to select...
<337704453 ms>(wpx.c,613) Have been asked to execute a request
<337704453 ms>(wppa.c,326) Building Arglist based on Parsed Content from WRB
<337704453 ms>(wppa.c,1007) Enter ParseUrlData
<337704453 ms>GET
<337704453 ms>(wppa.c,1056) Getting Values from QUERY_STRING
<337704453 ms>[headers begin]
<337704453 ms>[headers end]
<337704453 ms>p=4550:1
<337704453 ms>(wppa.c,1499) indx = 1, entryCnt = 1
<337704453 ms>(wppa.c,1849) Listing distinct actual names:
<337704453 ms>(wppa.c,1851) p
<337704453 ms>(wppa.c,1853) Listing actuals of array with large entries:
<337704453 ms>(wppa.c,1858) Listing distinct actual names and values:
<337704453 ms>(wppa.c,1890) p, type = 0, value (7) = 4550:1
<337704453 ms>(wppa.c,421) Arglist built, 1 unique entries
<337704453 ms>(wpx.c,620) Going to wpprodb_OciDoBlock...
<337704453 ms>(wpd.c,2734) Cache enabled. Gathering cache information.
<337704453 ms>(wpd.c,2752) Language for this request is en-us
<337704453 ms>(wpd.c,2803) Using user apex_public_user for caching.
<337704453 ms>cache: Checking for user level hit
<337704453 ms>cache: Cache MISS user - D:\oracle\product\10.1.0\db_1/Apache/modplsql/cache\plsql\712\2063
<337704453 ms>cache: Checking for system level hit
<337704453 ms>cache: Cache MISS system - D:\oracle\product\10.1.0\db_1/Apache/modplsql/cache\plsql\sys\878\4773
<337704453 ms>(wppr.c,393) start working with f
<337704453 ms>(wppr.c,1005) The CALL block: len=503, bind_count=9
declare
rc__ number;
start_time__ binary_integer;
begin
start_time__ := dbms_utility.get_time;
owa.init_cgi_env(:n__,:nm__,:v__);
htp.HTBUF_LEN := 84;
null;
null;
null;
null;
f(p=>:p);
if (wpg_docload.is_file_download) then
rc__ := 1;
wpg_docload.get_download_file(:doc_info);
null;
null;
null;
commit;
else
rc__ := 0;
null;
null;
null;
commit;
owa.get_page(:data__,:ndata__);
end if;
:rc__ := rc__;
:db_proc_time__ := dbms_utility.get_time - start_time__;
end;
<337704453 ms>(wppr.c,462) Pl/sql block parsed...
<337704453 ms>(wpdenv.c,1495) CGI Environment has 29 vars. Max name len 128, Max Value Len 128
<337704453 ms> PLSQL_GATEWAY(14)=(6)WebDb
<337704453 ms> GATEWAY_IVERSION(17)=(2)2
<337704453 ms> SERVER_SOFTWARE(16)=(59)Oracle-Application-Server-10g/9.0.4.0.0 Oracle-HTTP-Server
<337704453 ms> GATEWAY_INTERFACE(18)=(8)CGI/1.1
<337704453 ms> SERVER_PORT(12)=(3)80
<337704453 ms> SERVER_NAME(12)=(12)server_name_here
<337704453 ms> REQUEST_METHOD(15)=(4)GET
<337704453 ms> QUERY_STRING(13)=(9)p=4550:1
<337704453 ms> PATH_INFO(10)=(3)/f
<337704453 ms> SCRIPT_NAME(12)=(12)/apex/crmqa
<337704453 ms> REMOTE_ADDR(12)=(14)remote_ip_address_here
<337704453 ms> SERVER_PROTOCOL(16)=(9)HTTP/1.1
<337704453 ms> REQUEST_PROTOCOL(17)=(5)HTTP
<337704453 ms> REMOTE_USER(12)=(17)apex_public_user
<337704453 ms> HTTP_USER_AGENT(16)=(95)Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
<337704453 ms> HTTP_HOST(10)=(12)hostname_here
<337704453 ms> HTTP_ACCEPT(12)=(4)*/*
<337704453 ms> HTTP_ACCEPT_ENCODING(21)=(14)gzip, deflate
<337704453 ms> HTTP_ACCEPT_LANGUAGE(21)=(6)en-us
<337704453 ms> HTTP_ORACLE_ECID(17)=(37)1216239823:ecid_ip_address_here??:7200:6136:1,0
<337704453 ms> WEB_AUTHENT_PREFIX(19)=(1)
<337704453 ms> DAD_NAME(9)=(6)crmqa
<337704453 ms> DOC_ACCESS_PATH(16)=(5)docs
<337704453 ms> DOCUMENT_TABLE(15)=(23)wwv_flow_file_objects$
<337704453 ms> PATH_ALIAS(11)=(1)
<337704453 ms> REQUEST_CHARSET(16)=(5)UTF8
<337704453 ms> REQUEST_IANA_CHARSET(21)=(6)UTF-8
<337704453 ms> SCRIPT_PREFIX(14)=(6)/apex
<337704453 ms> HTTP_IV_USER(13)=(1)
<337704453 ms>StrArrPosBind pos 2 Charset Id : 871
<337704453 ms>StrArrPosBind pos 3 Charset Id : 871
<337704453 ms>StrArrPosBind pos 6 Charset Id : 871
<337704781 ms>(wpd.c,1954) Begin header parsing...
<337704781 ms>(wpd.c,2003) Got a line (47 bytes): X-ORACLE-IGNORE: IGNORE
<337704781 ms>(wpd.c,2021) X-ORACLE-IGNORE parsed
<337704781 ms>(wpd.c,2003) Got a line (47 bytes): X-ORACLE-IGNORE: IGNORE
<337704781 ms>(wpd.c,2021) X-ORACLE-IGNORE parsed
<337704781 ms>(wpd.c,2003) Got a line (47 bytes): X-ORACLE-IGNORE: IGNORE
<337704781 ms>(wpd.c,2021) X-ORACLE-IGNORE parsed
<337704781 ms>(wpd.c,2003) Got a line (47 bytes): X-ORACLE-IGNORE: IGNORE
<337704781 ms>(wpd.c,2021) X-ORACLE-IGNORE parsed
<337704781 ms>(wpd.c,2003) Got a line (77 bytes): Content-type: text/html; charset=UTF-8
<337704781 ms>(wpd.c,2102) Parsed header - Content-Type:text/html; charset=UTF-8
<337704781 ms>(wpd.c,2003) Got a line (49 bytes): X-DB-Content-length: 443
<337704781 ms>(wpd.c,2162) Parsed header - X-DB-Content-length:443
<337704781 ms>(wpd.c,2003) Got a line (1 bytes):
<337704781 ms>(wpd.c,2010) End of headers detected
<337704843 ms>(wpcs.c, 76) Executed 'begin dbms_session.reset_package; end;' (rc=0)
<337704843 ms>(wpd.c,1812) Going to close cursor
<337704843 ms>DeinitCursor
<337704843 ms>(wpx.c,626) Normal completion
<337704843 ms>(wpx.c,654) Shutdown has been called
<337704843 ms>(wpx.c,666) Going to logoff
<337704843 ms>Logoff: Pooling this connection
<337704843 ms>[ReqEndtime: 16/Jul/2008:15:23:44]
<337704843 ms>[ReqExecTime: 1281 ms]
and the 6136.log file contained...
<337703562 ms>[ReqStartTime: 16/Jul/2008:15:23:43]
<337703562 ms>Request ID ReqID:6136_1216239823
<337703562 ms>(wpdenv.c,663) script_name='/apex/crmqa' path_info='/f'script_prefix='/apex' dad_name='crmqa'
<337703562 ms>(wpdenv.c,776) User-Agent is Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
<337703562 ms>(wpdenv.c,1388) dadname = 'crmqa', path_info = 'f'
<337703562 ms>(wpdenv.c,1427) Service will NOT use dynamic auth
<337703562 ms>(wpx.c,387) Initialized successfully 0
<337703562 ms>(wpx.c,309) SetRemoteUser : Remote User set to apex_public_user for this request.
<337703562 ms>(wpx.c,476) Auth info from .APP file is being used
<337703562 ms>(wpd.c,1724) Attempting to logon with '(unknown)'
<337704843 ms>Maximum memory allocated by the request is 75158 bytes
<337704843 ms>[ReqEndtime: 16/Jul/2008:15:23:44]
<337704843 ms>[ReqExecTime: 1281 ms]
The DAD config file contains the following entries...
# ============================================================================
# mod_plsql DAD Configuration File
# ============================================================================
# 1. Please refer to dads.README for a description of this file
# ============================================================================
# /i contains version 221 images
Alias /i "D:\oracle\product\10.1.0\db_1\Apache\Apache\images"
Alias /i31 "D:\oracle\product\10.1.0\db_1\Apache\Apache\images311"
<Location /apex/crmqa>
SetHandler pls_handler
Order deny,allow
Allow from all
AllowOverride None
PlsqlCGIEnvironmentList HTTP_IV_USER
PlsqlDatabaseUsername apex_public_user
PlsqlDatabasePassword xxx
PlsqlDatabaseConnectString ip_address:port_number:crmqa1
PlsqlDefaultPage apex
PlsqlDocumentTablename wwv_flow_file_objects$
PlsqlDocumentPath docs
PlsqlDocumentProcedure wwv_flow_file_mgr.process_download
PlsqlAuthenticationMode Basic
PlsqlNLSLanguage AMERICAN_AMERICA.UTF8
</Location>
<Location /apex/crmdev>
SetHandler pls_handler
Order deny,allow
Allow from all
AllowOverride None
PlsqlCGIEnvironmentList HTTP_IV_USER
PlsqlDatabaseUsername apex_public_user
PlsqlDatabasePassword xxx
PlsqlDatabaseConnectString ip_address:port_number:crmdev1
PlsqlDefaultPage apex
PlsqlDocumentTablename wwv_flow_file_objects$
PlsqlDocumentPath docs
PlsqlDocumentProcedure wwv_flow_file_mgr.process_download
PlsqlAuthenticationMode Basic
PlsqlNLSLanguage AMERICAN_AMERICA.UTF8
</Location>
The DAD entries above point to databases on which Apex is version 3.1.1 and we have modified the image prefix on these database to be a value of /i31, using the script reset_image_prefix.sql.
This DAD file also contains DAD entries (not included above) for other database running Apex 2.2.1 which are using the default image prefix value of /i
I can ping the database ip address from the machine on which the OHS webserver is running.
I have installed OHS on my local laptop and have the same problems.
I can however execute a procedure on the database using the same OHS / DAD configuration running something like http://our_url/apex/crmqa/schema_name.test_connection?v_wait=1where the proc test_connection proc sleeps for v_wait seconds before returning a simple html page confirming that it worked.
I can login to the apex_public_user account via my SQL Developer client
At this point I do not know what else to try to resolve this issue. I am about to start trawling through metalink for related issues if I can find anything.
Any assistance is very much appreciated...
Thank you.
Alan

Answer 1

I also see this post on Metalink - could it be related?
https://metalink.oracle.com/metalink/plsql/f?p=130:14:12042222629340980284::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,429261.1,1,1,1,helvetica
Subject: After upgrading database to 10.2.0.X, getting ORA-06502: PL/SQL: numeric or value error:
Doc ID: Note:429261.1 Type: PROBLEM
Last Revision Date: 08-MAY-2008 Status: MODERATED
In this Document
Symptoms
Changes
Cause
Solution
References
This document is being delivered to you via Oracle Support's Rapid Visibility (RaV) process, and therefore has not been subject to an independent technical review.
Applies to:
Oracle Application Express (formerly HTML DB) - Version: 2.0.0.0.49
This problem can occur on any platform.
Symptoms
On the Application builder page, when attempting to click details to view the applications, the following error occurs:
ORA-06502: PL/SQL: numeric or value error: NULL index table key value
However, the problem seems to disappear after a while..
Changes
Upgrade of database (from any version to 10.2.0.1 or 10.2.0.2)
Cause
The reason is due to database Bug 4752541 Abstract: APPSST 10G: ORA-6550 AFTER UPGRADE TO 10.2.0.1. However Patch 47552541 is obsoleted and not available via metalink, it was superceded by Patch 5705795.
This bug is fixed in patchset 10.2.0.4 and above. For current and older versions, Patch 5705795 must be applied. If the patch cannot be found for a particular version or O/S , log a service request with Oracle Support to obtain the patch.
This is also documented in the Apex forum page: Re: wwv_flow.accept error lists the following bug as
root cause.
Solution
Interim solution:
Flush the database shared pool when the problem occurs
Permanent solution:
Upgrade database to 10.2.0.4 or above
OR
Download and apply database Patch 5705795. If the version for your database is not available , create a new Service request from metalink under product RDBMS to request for a patch
References
Bug 4752541 - APPSST 10G: ORA-6550 AFTER UPGRADE TO 10.2.0.1
Keywords
UPGRADE~DATABASE; UPGRADE~TO~10.2.0.4; UPGRADE~TO~10.2.0.1; APPLICATION~DETAILS; UPGRADE~FROM~10.2.0.2; UPGRADE~TO~10.2.0.3; VIEW~DETAILS;