During switchover operation in IE shows error "503 Service Unavailable".

Question

Hi,
I have installed Oracle Enterprise Manager 10g Release 2 (10.2.0.2) on Windows 2003 server service pack 2.Everything works fine, but when I go for switchover/failover operation firstly error comes as “500 Internal Server Error” and then after refreshing the page “503 Service Unavailable”. Following are the logs.
I applied the patch Grid Control Release 3 (10.2.0.3.0), but still same error occurs.
Both database status is up and EM console home shows 100% up.
Some Collected errors from emoms.trc
2007-11-03 12:24:35,453 [Thread-25] ERROR em.dataguard updateDGStatus.453 - ConfigBean:updateDGStatus(): Exception java.lang.NullPointerException
2007-11-03 12:25:01,703 [Thread-26] ERROR em.dataguard updateDGStatus.453 - ConfigBean:updateDGStatus(): Exception java.lang.NullPointerException
2007-11-03 12:25:11,156 [Thread-27] ERROR em.dataguard updateDGStatus.453 - ConfigBean:updateDGStatus(): Exception java.lang.NullPointerException
2007-11-03 12:25:17,906 [EMUI_12_25_17_/console/targets] ERROR jdk.NLSUtil getFormattedDateWithTZ.407 - Invalid region name format: GMT+05:30
2007-11-03 12:25:17,906 [EMUI_12_25_17_/console/targets] ERROR jdk.NLSUtil getFormattedDateWithTZ.407 - Invalid region name format: GMT+05:30
2007-11-03 12:25:19,343 [EMUI_12_25_19_/console/home] ERROR jdk.NLSUtil getFormattedDateWithTZ.407 - Invalid region name format: GMT+05:30
2007-11-03 12:30:16,562 [Thread-34] ERROR db.rac getEMDFromRep.806 - Closed Connection
java.sql.SQLException: Closed Connection
     at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:137)
     at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:174)
     at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:239)
     at oracle.jdbc.driver.PhysicalConnection.prepareStatement(PhysicalConnection.java:928)
     at oracle.jdbc.driver.PhysicalConnection.prepareStatement(PhysicalConnection.java:824)
     at oracle.jdbc.OracleConnectionWrapper.prepareStatement(OracleConnectionWrapper.java:72)
     at oracle.sysman.util.jdbc.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:467)
     at oracle.sysman.db.rac.util.ClusterUtil.getEMDFromRep(ClusterUtil.java:795)
     at oracle.sysman.db.rac.util.ClusterUtil.getEMD(ClusterUtil.java:888)
     at oracle.sysman.emo.util.DBVerify.getDBProperties(DBVerify.java:2454)
     at oracle.sysman.emo.util.DBStateUtil.doDatabaseShutdown(DBStateUtil.java:703)
     at oracle.sysman.db.dg.util.RemoteCtrl.shutdownDB(RemoteCtrl.java:596)
     at oracle.sysman.db.dg.util.ClientOperation.doClientOp(ClientOperation.java:368)
     at oracle.sysman.db.dg.util.ClientOperation$BounceThread.run(ClientOperation.java:215)
     at java.lang.Thread.run(Thread.java:534)
2007-11-03 12:30:16,562 [Thread-34] ERROR emo.util verifyDBProperties.2073 - DBStateUtil.verifyDBProperties: EMD URL was null or empty.
2007-11-03 12:30:23,265 [EMUI_12_30_23_/console/database/dataguard] ERROR em.dataguard onEvent.1233 - Data Guard: found errors during role change: Unable to retrieve the URL from repository. Check the specified target name and its corresponding target type. Other possible causes: remote EMD is not pointing to the web server, or remote target information has not been loaded.
2007-11-03 12:30:41,328 [ApplicationServerThread-5] ERROR app.SessionObjectManager sessionDestroyed.128 - java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1
ORA-01219: database not open: queries allowed on fixed tables/views only
2007-11-03 12:30:42,968 [PingHeartBeatRecorder] WARN emdrep.pingHBRecorder resetConnection.305 - SQLException caught while resetting connection ORA-00604: error occurred at recursive SQL level 1
ORA-01219: database not open: queries allowed on fixed tables/views only
2007-11-03 12:30:43,093 [Job Dispatcher Thread] ERROR em.jobs mainTask.2346 - encountered error:
java.sql.SQLException: ORA-01092: ORACLE instance terminated. Disconnection forced
2007-11-03 12:30:44,046 [SeverityLoad https://ashok:3872/emd/main/] ERROR eml.XMLLoader markFileAsError.854 - Error repeated 2 times: ERROR-400|ORA-00604: error occurred at recursive SQL level 1
ORA-01219: database not open: queries allowed on fixed tables/views only
2007-11-03 12:30:46,796 [AJPRequestHandler-ApplicationServerThread-6] ERROR eml.OMSHandshake processFailure.806 - OMSHandshake failed.(AGENT URL = https://ashok:3872/emd/main/)(ERROR = INTERNAL_ERROR)(CAUSE =java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1
ORA-01219: database not open: queries allowed on fixed tables/views only
2007-11-03 12:33:29,843 [AJPRequestHandler-ApplicationServerThread-6] ERROR conn.ConnectionService verifyRepositoryEx.817 - Invalid Connection Pool. ERROR = ORA-00604: error occurred at recursive SQL level 1
ORA-01219: database not open: queries allowed on fixed tables/views only
Regards,
S Reddy

Answer 1

Hi,
Thanks for your reply.
Steps Performed while configuring data guard.
- Install Oracle Enterprise Manager 10 g with new database option on windows server 2003 enterprise edition service pack 2.
- Applied patch of Oracle database 10g Release 2 (10.2.0.3)
- Applied patch of Oracle Enterprise Manager 10g Grid Control Release 3 (10.2.0.3.0)
Primary Database=GOOGLE
Secondary Database=YAHOO
Run the following commands in CMD.
Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.
C:\Documents and Settings\oracle>set oracle_home=C:\Oracle_NET\db10g
C:\Documents and Settings\oracle>set oracle_sid=GOOGLE
C:\Documents and Settings\oracle>sqlplus /nolog
SQL*Plus: Release 10.1.0.4.2 - Production on Thu Nov 8 09:44:58 2007
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> CONNECT SYS/infodb1 as sysdba
Connected to an idle instance.
SQL> startup;
ORACLE instance started.
Total System Global Area 536870912 bytes
Fixed Size 790068 bytes
Variable Size 162525644 bytes
Database Buffers 373293056 bytes
Redo Buffers 262144 bytes
Database mounted.
Database opened.
SQL> show parameters unique;
NAME TYPE VALUE
db_unique_name string GOOGLE
SQL> select log_mode from v$database;
LOG_MODE
NOARCHIVELOG
SQL> create pfile from spfile;
File created.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.4.0 - Pr
oduction
With the Partitioning, OLAP and Data Mining options
C:\Documents and Settings\oracle>sqlplus /nolog
SQL*Plus: Release 10.1.0.4.2 - Production on Thu Nov 8 10:38:31 2007
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> CONNECT SYS/infodb1 as sysdba
Connected to an idle instance.
SQL> [b]STARTUP NOMOUNT PFILE=C:\Oracle_NET\db10g\database\INITgoogle.ORA;
ORACLE instance started.
Total System Global Area 536870912 bytes
Fixed Size 790068 bytes
Variable Size 162525644 bytes
Database Buffers 373293056 bytes
Redo Buffers 262144 bytes
SQL> create spfile from pfile;
File created.
SQL> shutdown immediate;
ORA-01507: database not mounted
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.4.0 - Pr
oduction
With the Partitioning, OLAP and Data Mining options
C:\Documents and Settings\oracle>sqlplus /nolog
SQL*Plus: Release 10.1.0.4.2 - Production on Thu Nov 8 10:41:32 2007
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> CONNECT SYS/infodb1 as sysdba
Connected to an idle instance.
SQL> STARTUP MOUNT;
ORACLE instance started.
Total System Global Area 536870912 bytes
Fixed Size 790068 bytes
Variable Size 162525644 bytes
Database Buffers 373293056 bytes
Redo Buffers 262144 bytes
Database mounted.
SQL> alter database archivelog;
Database altered.
SQL> alter database force logging;
Database altered.
SQL> select log_mode from v$database;
LOG_MODE
ARCHIVELOG
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.4.0 - Pr
oduction
With the Partitioning, OLAP and Data Mining options
Then I login to site https://ashok:1159/em by using SYSMAN account which is the console for em.
-     Then I created standby database through.<TARGETS-DATABASES-GOOGLE- MAINTENANCE -DATA GUARD- SET UP AND MANAGE-ADD STANDBY DATABASE- CREATE A NEW PHYSICAL STANDBY DATABASE>
-     Then I verify the configuration it shows the following log.
Initializing
Connected to instance ASHOK:yahoo
Starting alert log monitor...
Updating Data Guard link on database homepage...
Data Protection Settings:
Protection mode : Maximum Performance
Log Transport Mode settings:
GOOGLE: ARCH
YAHOO: ARCH
Checking standby redo log files.....OK
Checking Data Guard status
GOOGLE : Normal
YAHOO : Normal
Request ID: 2: 1: 638446177, Operation: CTL_GET_STATUS
Checking Inconsistent Properties
Checking agent status
GOOGLE ... WARNING: No credentials available for target ASHOK
Attempting agent ping ... OK
YAHOO ... WARNING: No credentials available for target ASHOK
Attempting agent ping ... OK
Switching log file 90.Done
Checking applied log on GOOGLE...OK
Processing completed.
Following are the files:
Additional parameters in[b] INITgoogle.ora
*.db_unique_name='GOOGLE
*.LOG_ARCHIVE_CONFIG='DG_CONFIG=(GOOGLE,YAHOO)'
*.log_archive_dest_1='location=C:\Oracle_NET\ALD_DEST_1','VALID_FOR=(ALL_LOGFILES,ALL_ROLES)','mandatory','arch','DB_UNIQUE_NAME=GOOGLE','REOPEN=30'
*.log_archive_dest_2='SERVICE=YAHOO','LGWR','ASYNC','VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)','DB_UNIQUE_NAME=YAHOO'
*.LOG_ARCHIVE_DEST_STATE_1=ENABLE
*.LOG_ARCHIVE_DEST_STATE_2=ENABLE
*.REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE
*.LOG_ARCHIVE_FORMAT=%t_%s_%r.arc
*.LOG_ARCHIVE_MAX_PROCESSES=10
*.fal_server=YAHOO
*.fal_client=GOOGLE
*.standby_file_management=auto
*.db_file_name_convert='YAHOO/YAHOO','GOOGLE/GOOGLE'
*.log_file_name_convert='C:\Oracle_NET\OD\YAHOO\YAHOO','C:\Oracle_NET\OD\GOOGLE\GOOGLE'
listener.ora
# listener50312.ora Network Configuration File: C:\DOCUME~1\oracle\LOCALS~1\Temp\listener50312.ora
# Generated by Oracle configuration tools.
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = GOOGLE)
(ORACLE_HOME = C:\Oracle_NET\db10g)
(PROGRAM = extproc)
(SID_DESC =
(GLOBAL_DBNAME = GOOGLE_DGMGRL)
(ORACLE_HOME = C:\Oracle_NET\db10g)
(SID_NAME = GOOGLE)
(SID_DESC =
(GLOBAL_DBNAME = YAHOO_DGMGRL)
(ORACLE_HOME = C:\Oracle_NET\db10g)
(SID_NAME = YAHOO)
(SID_DESC =
(GLOBAL_DBNAME = YAHOO_DGMGRL)
(ORACLE_HOME = C:\Oracle_NET\db10g)
(SID_NAME = YAHOO)
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = ashok)(PORT = 1521))
Tnsnames.ora
# tnsnames.ora Network Configuration File: C:\Oracle_NET\db10g\network\admin\tnsnames.ora
# Generated by Oracle configuration tools.
GOOGLE =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ashok)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = GOOGLE)
YAHOO =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ashok)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = YAHOO)
EXTPROC_CONNECTION_DATA =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
(CONNECT_DATA =
(SID = PLSExtProc)
(PRESENTATION = RO)
-     Then Both databases are running
-     It showing the data guard home page.
-     Then I go for switchover operation the error occurs.