Dbconsole failed to start on one RAC node
Hi
I have 2 RAC nodes (RHEL 4) and 10.2.0.1. On one dbconsole is running and on other I get the following. Earlier dbconsole
on both the nodes used to run perfectly fine. I will appreacite any suggestions to rectify this problem.
Regards
oracle@rac01<18>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> emctl start dbconsole
TZ set to Canada/Newfoundland
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
http://rac01:1158/em/console/aboutApplication
Agent Version : 10.1.0.4.1
OMS Version : Unknown
Protocol Version : 10.1.0.2.0
Agent Home : /u01/app/oracle/product/10.2/db_1/rac01_RACDB1
Agent binaries : /u01/app/oracle/product/10.2/db_1
Agent Process ID : 23329
Parent Process ID : 21132
Agent URL : http://rac01:3938/emd/main
Started at : 2007-07-25 11:37:32
Started by user : oracle
Last Reload : 2007-07-25 11:37:32
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 371
Size of XML files pending upload(MB) : 7.66
Available disk space on upload filesystem : 44.78%
Agent is already started. Will restart the agent
Stopping agent ... stopped.
Starting Oracle Enterprise Manager 10g Database Control ............................................................................................. failed.
Logs are generated in directory /u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log
oracle@rac01<19>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log>
ON OTHER NODE:
oracle@rac02<2>:/u01/app/oracle> emctl start dbconsole
TZ set to Canada/Newfoundland
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
http://rac01:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 10g Database Control .................................... started.
Logs are generated in directory /u01/app/oracle/product/10.2/db_1/rac02_RACDB2/sysman/log
oracle@rac02<3>:/u01/app/oracle>
Thanks for your time and reply .
Well, here is what I got, couldn't make out from here.
Regards
oracle@rac01<19>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> ls -lart
total 13500
drwxr----- 7 oracle dba 4096 Jul 14 10:48 ..
-rw-r----- 1 oracle dba 0 Jul 14 10:48 emdctl.log
drwxrwx--- 2 oracle dba 4096 Jul 14 10:54 nmcRACDB11521
-rw-r----- 1 oracle dba 4655792 Jul 24 23:01 emoms.trc
-rw-r----- 1 oracle dba 4655792 Jul 24 23:01 emoms.log
drwxr----- 3 oracle dba 4096 Jul 25 11:35 .
-rw-r----- 1 oracle dba 4096 Jul 25 12:05 emdb.nohup.lr
-rw-r----- 1 oracle dba 1074 Jul 25 12:05 emagent_perl.trc
-rw-r----- 1 oracle dba 1731 Jul 25 12:06 emagent.log
-rw-r----- 1 oracle dba 1080 Jul 25 12:07 emagentfetchlet.trc
-rw-r----- 1 oracle dba 1080 Jul 25 12:07 emagentfetchlet.log
-rw-r----- 1 oracle dba 81089 Jul 25 13:28 emdctl.trc
-rw-r----- 1 oracle dba 3309143 Jul 25 13:28 emdb.nohup
-rw-r----- 1 oracle dba 1044518 Jul 25 13:28 emagent.trc
oracle@rac01<20>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> cat emagent.log
2007-07-14 10:50:44 Thread-3086936288 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-14 10:51:16 Thread-3086936288 EMAgent started successfully (00702)
2007-07-14 14:38:21 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-14 14:39:00 Thread-3086935744 EMAgent started successfully (00702)
2007-07-24 07:05:06 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-24 07:07:11 Thread-3086935744 target {+ASM1_rac01, osm_instance} is broken: cannot compute dynamic properties in time. (00155)
2007-07-24 07:07:14 Thread-3086935744 EMAgent started successfully (00702)
2007-07-24 12:06:27 Thread-3086935744 EMAgent normal shutdown (00703)
2007-07-24 12:08:26 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-24 12:08:51 Thread-3086935744 EMAgent started successfully (00702)
2007-07-25 11:35:35 Thread-3086935744 EMAgent normal shutdown (00703)
2007-07-25 11:37:32 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-25 11:39:29 Thread-3086935744 target {+ASM1_rac01, osm_instance} is broken: cannot compute dynamic properties in time. (00155)
2007-07-25 11:39:30 Thread-3086935744 EMAgent started successfully (00702)
2007-07-25 12:03:36 Thread-3086935744 EMAgent normal shutdown (00703)
2007-07-25 12:05:15 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-25 12:06:23 Thread-3086935744 target {+ASM1_rac01, osm_instance} is broken: cannot compute dynamic properties in time. (00155)
2007-07-25 12:06:24 Thread-3086935744 EMAgent started successfully (00702)
oracle@rac01<21>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> cat emagentfetchlet.log
2007-07-14 11:01:44,208 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-14 14:40:29,096 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-24 07:10:44,123 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-24 12:12:48,187 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-25 11:41:25,628 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-25 12:07:30,335 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
oracle@rac01<22>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log>
oracle@rac01<22>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> tail -40 emagentfetchlet.trc
2007-07-14 11:01:44,208 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-14 14:40:29,096 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-24 07:10:44,123 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-24 12:12:48,187 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-25 11:41:25,628 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-25 12:07:30,335 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
oracle@rac01<25>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> tail -10 emdctl.trc
2007-07-25 13:01:02 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:04:41 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:07:12 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:10:50 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:14:32 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:18:09 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:20:40 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:24:27 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:28:06 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:31:43 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
oracle@rac01<28>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> tail -10 emagent.trc
2007-07-25 13:31:44 Thread-43162528 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:31:44 Thread-43162528 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
2007-07-25 13:32:14 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:32:14 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
2007-07-25 13:32:14 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:32:14 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
2007-07-25 13:32:44 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:32:44 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
2007-07-25 13:32:44 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:32:44 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
Message was edited by:
Singh
Similar Messages
-
Ocmstart.sh is not starting in one RAC node
I am having a trouble starting "oracm" on one node. I can see it is working on second node but not on first node. Is there any one haveing the same issue.
Hi Chandra Sorry for the delay, I got the output from cm.out file during restart of ocmstart.sh script
=========================================================
TRACE: HandleJoin(): JOIN from node(1)->(1), tid = ClusterListener:3076 file = nmlisten.c, line = 362 {Thu Apr 10 21:39:10 2003 }
TRACE: HandleStatus(): node(0) UNKNOWN, tid = ClusterListener:3076 file = nmlisten.c, line = 404 {Thu Apr 10 21:39:11 2003 }
TRACE: HandleStatus(): src[0] dest[1] dom[0] seq[33] sync[10], tid = ClusterListener:3076 file = nmlisten.c, line = 415 {Thu Apr 10 21:39:11 2003 }
TRACE: HandleSync(): src[0] dest[1] dom[0] seq[34] sync[10], tid = ClusterListener:3076 file = nmlisten.c, line = 506 {Thu Apr 10 21:39:11 2003 }
TRACE: SendAck(): node(0) domain(0) syncSeqNo(10) type(11), tid = ClusterListener:3076 file = nmmember.c, line = 1913 {Thu Apr 10 21:39:11 2003 }
TRACE: HandleVote(): src[0] dest[1] dom[0] seq[35] sync[10], tid = ClusterListener:3076 file = nmlisten.c, line = 643 {Thu Apr 10 21:39:12 2003 }
TRACE: SendVoteInfo(): node(0) domain(0) syncSeqNo(10), tid = ClusterListener:3076 file = nmmember.c, line = 1727 {Thu Apr 10 21:39:12 2003 }
TRACE: HandleShutdown(): src[0] dest[1] dom[0] seq[0] sync[10] type[4], tid = ClusterListener:3076 file = nmlisten.c, line = 1087 {Thu Apr 10 21:39:13 2003 }
TRACE: IncrementEventValue: *(80f2920) = (1, 1), tid = ClusterListener:3076 file = unixinc.c, line = 253 {Thu Apr 10 21:39:13 2003 }--- End Dump ---
oracm, version[ 9.2.0.2.0.47 ] started {Thu Apr 10 21:40:07 2003 }
KernelModuleName is hangcheck-timer {Thu Apr 10 21:40:07 2003 }
OemNodeConfig(): Network Address of node0: 192.168.0.50 (port 9998)
{Thu Apr 10 21:40:07 2003 }
OemNodeConfig(): Network Address of node1: 192.168.0.51 (port 9998)
{Thu Apr 10 21:40:07 2003 }
WARNING: OemInit2: Opened file(/var/opt/oracle/oradata/orcl/CMQuorumFile 8), tid = main:1024 file = oem.c, line = 491 {Thu Apr 10 21:40:07 2003 }Debug Hang : ClusterListener (PID=2541) Registered withwatchdog daemon. {Thu Apr 10 21:40:09 2003 }
InitializeCM: ModuleName = hangcheck-timer {Thu Apr 10 21:40:09 2003 }
InitializeCM: Kernel module hangcheck-timer is already loaded {Thu Apr 10 21:40:09 2003 }
Debug Hang : CmConnectListener (PID=2542):Registered with watchdog daemon. {Thu Apr 10 21:40:09 2003 }
Debug Hang :StartNMMon (PID=2536) Registered with watchdog daemon. {Thu Apr 10 21:40:09 2003 }
CreateLocalEndpoint(): Network Address: 192.168.0.51
{Thu Apr 10 21:40:09 2003 }
Debug Hang :PollingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
Debug Hang : DiskPingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
Debug Hang :SendingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
--- DUMP GROUP STATE DB ---
--- END OF GROUP STATE DUMP ---
--- Begin Dump ---
TRACE: LogListener: Spawned with tid 0x803., tid = 2051 file = logging.c, line = 116 {Thu Apr 10 21:40:07 2003 }oracm, version[ 9.2.0.2.0.47 ] started {Thu Apr 10 21:40:07 2003 }
TRACE: Can't read registry value for WatchdogTimerMargin, tid = main:1024 file = unixinc.c, line = 1011 {Thu Apr 10 21:40:07 2003 }
TRACE: Can't read registry value for WatchdogSafetyMargin, tid = main:1024 file = unixinc.c, line = 1011 {Thu Apr 10 21:40:07 2003 }KernelModuleName is hangcheck-timer {Thu Apr 10 21:40:07 2003 }
TRACE: Can't read registry value for ClientTimeout, tid = main:1024 file = unixinc.c, line = 1011 {Thu Apr 10 21:40:07 2003 }
TRACE: InitNMInfo: setting clientTimeout to 215s based on MissCount 215 and PollInterval 1000ms, tid = main:1024 file = nmconfig.c, line = 137 {Thu Apr 10 21:40:07 2003 }
TRACE: InitClusterDb(): getservbyname on CMSrvr failed - 22 : assigning 9998, tid = main:1024 file = nmconfig.c, line = 212 {Thu Apr 10 21:40:07 2003 }OemNodeConfig(): Network Address of node0: 192.168.0.50 (port 9998)
{Thu Apr 10 21:40:07 2003 }
OemNodeConfig(): Network Address of node1: 192.168.0.51 (port 9998)
{Thu Apr 10 21:40:07 2003 }
TRACE: OemCreateListenPort: bound at 9998, tid = main:1024 file = oem.c, line = 858 {Thu Apr 10 21:40:07 2003 }
TRACE: InitClusterDb(): found my node info at 1 name rac2, priv rac2priv, port 3623, tid = main:1024 file = nmconfig.c, line = 265 {Thu Apr 10 21:40:07 2003 }
TRACE: InitClusterDb(): Local Node(1) NodeName[rac2], tid = main:1024 file = nmconfig.c, line = 283 {Thu Apr 10 21:40:07 2003 }
TRACE: InitClusterDb(): Cluster(Oracle) with (2) Defined Nodes, tid = main:1024 file = nmconfig.c, line = 286 {Thu Apr 10 21:40:07 2003 }
TRACE: OEMInits(): CM Disk File (/var/opt/oracle/oradata/orcl/CMQuorumFile), tid = main:1024 file = oem.c, line = 243 {Thu Apr 10 21:40:07 2003 }
WARNING: OemInit2: Opened file(/var/opt/oracle/oradata/orcl/CMQuorumFile 8), tid = main:1024 file = oem.c, line = 491 {Thu Apr 10 21:40:07 2003 }
TRACE: ReadOthersDskInfo(): node(0) rcfg(11) wrtcnt(1052) lastcnt(0) alive(1052), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(1) rcfg(10) wrtcnt(2) lastcnt(0) alive(1), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(2) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(3) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(4) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(5) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(6) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(7) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(8) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(9) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(10) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(11) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(12) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(13) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(14) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(15) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(16) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(17) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(18) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(19) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(20) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(21) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(22) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(23) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(24) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(25) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(26) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(27) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(28) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(29) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(30) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(31) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(32) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(33) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(34) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(35) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(36) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(37) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(38) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(39) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(40) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(41) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(42) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(43) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(44) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(45) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(46) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(47) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(48) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(49) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(50) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(51) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(52) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(53) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(54) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(55) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(56) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(57) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(58) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(59) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(60) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(61) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(62) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ReadOthersDskInfo(): node(63) rcfg(0) wrtcnt(0) lastcnt(0) alive(0), tid = main:1024 file = oem.c, line = 1442 {Thu Apr 10 21:40:09 2003 }
TRACE: ClusterListener: Spawned with tid 0xc04., tid = 3076 file = nmlistener.c, line = 53 {Thu Apr 10 21:40:09 2003 }Debug Hang : ClusterListener (PID=2541) Registered withwatchdog daemon. {Thu Apr 10 21:40:09 2003 }
TRACE: ClusterListener (pid=2541, tid=3076): Registered with watchdog daemon., tid = 3076 file = nmlistener.c, line = 75 {Thu Apr 10 21:40:09 2003 }InitializeCM: ModuleName = hangcheck-timer {Thu Apr 10 21:40:09 2003 }
InitializeCM: Kernel module hangcheck-timer is already loaded {Thu Apr 10 21:40:09 2003 }
TRACE: CmConnectListener: Spawned with tid 0x1005., tid = 4101 file = cmclient.c, line = 215 {Thu Apr 10 21:40:09 2003 }Debug Hang : CmConnectListener (PID=2542):Registered with watchdog daemon. {Thu Apr 10 21:40:09 2003 }
TRACE: CmConnectListener (pid=2542, tid=4101): Registered with watchdog daemon., tid = 4101 file = cmclient.c, line = 246 {Thu Apr 10 21:40:09 2003 }Debug Hang :StartNMMon (PID=2536) Registered with watchdog daemon. {Thu Apr 10 21:40:09 2003 }
TRACE: StartNMMon (pid=2536, tid=1024): Registered with watchdog daemon., tid = main:1024 file = cmnodemon.c, line = 251 {Thu Apr 10 21:40:09 2003 }CreateLocalEndpoint(): Network Address: 192.168.0.51
{Thu Apr 10 21:40:09 2003 }
TRACE: StartClusterJoin(): clusterState(0) nodeState(0), tid = main:1024 file = nmmember.c, line = 277 {Thu Apr 10 21:40:09 2003 }
TRACE: PollingThread: Spawned with tid 0x1406., tid = 5126 file = nmmember.c, line = 742 {Thu Apr 10 21:40:09 2003 }Debug Hang :PollingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
TRACE: PollingThread (pid=2543, tid=5126): Registered with watchdog daemon., tid = 5126 file = nmmember.c, line = 760 {Thu Apr 10 21:40:09 2003 }
TRACE: DiskPingThread: Spawned with tid 0x1807., tid = 6151 file = nmmember.c, line = 1050 {Thu Apr 10 21:40:09 2003 }Debug Hang : DiskPingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
TRACE: DiskPingThread (pid=2544, tid=6151): Registered with watchdog daemon., tid = 6151 file = nmmember.c, line = 1074 {Thu Apr 10 21:40:09 2003 }
TRACE: SendingThread: Spawned with tid 0x1c08, 0x0., tid = 7176 file = nmmember.c, line = 511 {Thu Apr 10 21:40:09 2003 }Debug Hang :SendingThread (PID=135159169): Registered with {Thu Apr 10 21:40:09 2003 }
TRACE: SendingThread (pid=2545, tid=7176): Registered with watchdog daemon., tid = 7176 file = nmmember.c, line = 576 {Thu Apr 10 21:40:09 2003 }
TRACE: HandleJoin(): src[1] dest[1] dom[0] seq[1] sync[0], tid = ClusterListener:3076 file = nmlisten.c, line = 346 {Thu Apr 10 21:40:09 2003 }
TRACE: HandleJoin(): JOIN from node(1)->(1), tid = ClusterListener:3076 file = nmlisten.c, line = 362 {Thu Apr 10 21:40:09 2003 }
TRACE: HandleStatus(): node(0) UNKNOWN, tid = ClusterListener:3076 file = nmlisten.c, line = 404 {Thu Apr 10 21:40:10 2003 }
TRACE: HandleStatus(): src[0] dest[1] dom[0] seq[36] sync[11], tid = ClusterListener:3076 file = nmlisten.c, line = 415 {Thu Apr 10 21:40:10 2003 }
TRACE: HandleSync(): src[0] dest[1] dom[0] seq[37] sync[11], tid = ClusterListener:3076 file = nmlisten.c, line = 506 {Thu Apr 10 21:40:10 2003 }
TRACE: SendAck(): node(0) domain(0) syncSeqNo(11) type(11), tid = ClusterListener:3076 file = nmmember.c, line = 1913 {Thu Apr 10 21:40:10 2003 }
TRACE: HandleVote(): src[0] dest[1] dom[0] seq[38] sync[11], tid = ClusterListener:3076 file = nmlisten.c, line = 643 {Thu Apr 10 21:40:11 2003 }
TRACE: SendVoteInfo(): node(0) domain(0) syncSeqNo(11), tid = ClusterListener:3076 file = nmmember.c, line = 1727 {Thu Apr 10 21:40:11 2003 }
TRACE: HandleShutdown(): src[0] dest[1] dom[0] seq[0] sync[11] type[4], tid = ClusterListener:3076 file = nmlisten.c, line = 1087 {Thu Apr 10 21:40:12 2003 }
TRACE: IncrementEventValue: *(80f2920) = (1, 1), tid = ClusterListener:3076 file = unixinc.c, line = 253 {Thu Apr 10 21:40:12 2003 }--- End Dump ---
========================================================= -
What would happened when one RAC node's public NIC down ?
Dear all,
There's a two-node RAC on my office. As my observation, when one RAC node's public NIC is down, the "crs_stat -t" command wouldn't show any resource is OFFLINE. So at this moment if i use sqlplus to connect to my RAC db, it still will choose node1 or node2 instance randomly right? And if i've been assigned to the node's instance whose public NIC is down, my sqlplus would failed to connect to db?
So, can we say that Oracle RAC can't supply HA function if one node's public NIC is down? Or is there just any other solution to solve this issue? Any suggestion would be appreciated.Public node is down means that only one node would be able to take the load i.e. in your case it will be Node 2. It may happen that CRS is unable to record the status - in such case there may be failed connections to one node.
-
PRCR-1079 Failed to start resource ora.rac.db - during installation
Hi
After successful installation of Grid Infrasturcture I proceeded with database installation on clusterware and at the stage when the installer was creating clone database I got the following errors ( this was my 2nd attempt and I got the same errors both the time ) :
Errors:
PRCR-1079 : Failed to start resource ora.rac.db
ORA-01092 : ORACLE instance terminated. Disconnection forced
ORA-00704 : bootstrap process failure
ORA-00604 : error occurred at resursiive SQL level 2
ORA-01578 : ORACLE data block corrupted (file # 1, block # 5505)
ORA-01110 : data file 1:'+DATA/rac/datafile/system.256.799676855'
Process ID : 23498
Session ID : 63 Serial number 3
CRS-2674 Start of 'ora.rac.db' on 'rac2' failed
CRS-2632 There are no more servers to try to place resource 'ora.rac.db' on that would satisfy its placement policy
There are no logs on that node (rac2)
I am running Oracle Linux 5.4 64 bit
As mentioned above this was my 2nd attempt afresh and I got the same errors both the times, please let me know what is the problem as the rac2 is replica of rac2 in VMWare.
Thanks for your help
Rgds
THi
I tried again for the 3rd time and go the same error again, this time I rebuilt the node 2 - Can someone ple ase help me with this issue why it keeps failing on node 2 at the same stage for the 2rd time in a row.
Also please help me clone the database manually from node 1 to node 2 so I don't have to try to reinstall it again, there must be ways to do it
Thanks for your help in advance
Rgds
T -
How to execute DBMS_JOB at exactly one RAC node
Hello,
after unsuccessfully searching for "RAC" and "DBMS_JOB" I open this thread.
Can you tell me how to dedicate one RAC-node for doing my batch-jobs
which are started by using dbms_job (so there is no tnsnames.ora which is used).
Thanks in advancehi,
Let's say the instances are named:I1, I2, I3, I4
Issue:ALTER SYSTEM SET JOB_QUEUE_PROCESSES=0 SCOPE=BOTH SID='I1';
ALTER SYSTEM SET JOB_QUEUE_PROCESSES=0 SCOPE=BOTH SID='I2';
ALTER SYSTEM SET JOB_QUEUE_PROCESSES=0 SCOPE=BOTH SID='I3';
ALTER SYSTEM SET JOB_QUEUE_PROCESSES=10 SCOPE=BOTH SID='I4';So that only instance I4 will run jobs.
Regards,
Yoann. -
Failed to start GSD on local node
I am trying to install Oracle 9.2 Real Application Cluster on Linux 7.3.
Everything is OK until DBCA asks to run gsdctl. After running gsdctl start in
several minutes I can see "Failed to start GSD on local node" without any
explanations.
What should I do?To get more information about the problem, I've seen the following used in the "gsd.sh" script:
exec $JRE -nojit -DPROGRAM=gsd -DTRACING.ENABLED=true -DTRACING.LEVEL=0 -classpath $CLASSPATH oracle.ops.mgmt.daemon.OPSMDaemon $MY_OHOME
Start the gsd with: "gsd.sh" rather than "gsdctl start".
The "TRACING.ENABLED" and "TRACING.LEVEL" will cause the JRE to emit a great deal of debug information. The "nojit" option turns off the "just in time" feature, but I don't know if that's too important (same error, either way). I found this information through some newsgroup. However, I haven't actually gotten the "gsd" daemon to run for more than a few minutes. After a minute or two, the error is as follows:
[main] [20:19:6:880] [OCRTree.<init>:80] Going to load the ocr librarysrvmocr
[main] [20:19:6:883] [OCRTree.<init>:80] loaded ocr libraries
[main] [20:19:6:884] [OCRTree.<init>:80] in OCR constructor
[main] [20:19:6:885] [OCRTree.<init>:80] Initializing OCR: NEW
initializeDaemon: localNodeName = orac01
Native Code Debug is ONinitializeOCR: procr_init retval = 0
SIGSEGV 11* segmentation violation
stackbase=BFFFF308, stackpointer=BFFFC6F4
Full thread dump:
"process reaper" (TID:0x404a4800, sys_thread_t:0x815c8f8, state:CW, thread_t: t@49159, sp:0x0
threadID:0xbfa, stack_base:0x413fba50, stack_size:0x200000) prio=0
java.lang.Thread.run(Thread.java)
"KeepAlive" (TID:0x404a4200, sys_thread_t:0x8148be8, state:CW, thread_t: t@40966, sp:0x0 threadID:0xbf9, stack_base:0x413daa50, stack_size:0x200000) prio=0
sun.rmi.transport.KeepAlive.run(ObjectTable.java:182)
java.lang.Thread.run(Thread.java)
"Reaper" (TID:0x404a3e38, sys_thread_t:0x814ea48, state:CW, thread_t: t@32773, sp:0x0 threadID:0xbf8, stack_base:0x413b9a50, stack_size:0x200000) prio=-1073756771
sun.rmi.transport.Reaper.run(ObjectTable.java:199)
java.lang.Thread.run(Thread.java)
"TCP Accept-1" (TID:0x404a3d88, sys_thread_t:0x8148898, state:R, thread_t: t@24580, sp:0x0 threadID:0xbf7, stack_base:0x41398a50, stack_size:0x200000) prio=-1073756774
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:379)
java.net.ServerSocket.implAccept(ServerSocket.java:199)
java.net.ServerSocket.accept(ServerSocket.java:181)
sun.rmi.transport.proxy.HttpAwareServerSocket.accept(HttpAwareServerSocket.java:70)
sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:376)
java.lang.Thread.run(Thread.java)
"SIGQUIT handler" (TID:0x404972a0, sys_thread_t:0x807d768, state:R, thread_t: t@16387, sp:0x0
threadID:0xbf6, stack_base:0x411a7a50, stack_size:0x200000) prio=-1073756763
"Finalizer thread" (TID:0x40497088, sys_thread_t:0x807d640, state:CW, thread_t: t@8194, sp:0x0 threadID:0xbf5, stack_base:0x41186a50, stack_size:0x200000) prio=-1073756763
"main" (TID:0x404970b0, sys_thread_t:0x8064810, state:R, thread_t: t@8192, sp:0x0 threadID:0xbe9, stack_base:0xbffff308, stack_size:0x200000) prio=-1073756763 current thread
oracle.ops.mgmt.rawdevice.OCR.<init>(OCR.java:101)
oracle.ops.mgmt.rawdevice.OCRTree.<init>(OCRTree.java:80)
oracle.ops.mgmt.rawdevice.OCRTree.init(OCRTree.java:93)
oracle.ops.mgmt.rawdevice.RawDeviceConfig.<init>(RawDeviceConfig.java:98)
oracle.ops.mgmt.rawdevice.RawDeviceConfig.init(RawDeviceConfig.java:113)
oracle.ops.mgmt.daemon.OPSMDaemon.<init>(OPSMDaemon.java:207)
oracle.ops.mgmt.daemon.OPSMDaemon.main(OPSMDaemon.java:726)
Monitor Cache Dump:
java.net.PlainSocketImpl@1078606656/1079100848: owner "TCP Accept-1" (0x8148898, 1 entry)
<unknown key> (0x0x8148be8): <unowned>
Waiting to be notified:
"KeepAlive" (0x8148be8)
java.lang.Class@1078622888/1079239192: owner "main" (0x8064810, 1 entry)
<unknown key> (0x0x814ea48): <unowned>
Waiting to be notified:
"Reaper" (0x814ea48)
java.lang.Class@1078623280/1079245648: owner "main" (0x8064810, 1 entry)
Registered Monitor Dump:
Fork_Wait_monitor: <unowned>
Waiting to be notified:
"process reaper" (0x815c8f8)
Thread queue lock: <unowned>
Name and type hash table lock: <unowned>
String intern lock: <unowned>
JNI pinning lock: <unowned>
JNI global reference lock: <unowned>
BinClass lock: <unowned>
Class loading lock: <unowned>
Java stack lock: <unowned>
Code rewrite lock: <unowned>
Heap lock: <unowned>
Has finalization queue lock: <unowned>
Finalize me queue lock: <unowned>
Waiting to be notified:
"Finalizer thread" (0x807d640)
Monitor registry: owner "main" (0x8064810, 1 entry)
Aborted
This error was generated with RedHat Linux 8, the normal edition, and Oracle 9iR2 ORAC. Is this the error you generated? Perhaps we are missing something special about RedHad AS? I would certainly appreciate it if someone could offer specifics. -
RAC, ASM failed to start up on second node , ORA-03113: end-of-file on comm
i'm installing an RAC with 2 nodes on top of ASM
when creating ASM Diskgroup , it failed and reported error CRS-0215 failed to start asm on node2
Oracle 10.2.0.1
linux CentOs 4.x
u01/app/oracle/product/10.2.0/db_1/bin/dbca -progress_only -configureASM -templateName NO_VALUE -gdbName NO -sid NO -emConf
iguration NONE -diskList /dev/raw/raw2,/dev/raw/raw3 -diskGroupName DATA -datafileJarLocation /u01/app/oracle/product/10.2.0/db_
1/assistants/dbca/templates -responseFile NO_VALUE -nodeinfo node1,node2 -obfuscatedPasswords true -oratabLocation /u01/app/o
racle/product/10.2.0/db_1/install/oratab -asmSysPassword 05dbb0be38ecf8cca822cf3cf99e675448 -redundancy EXTERNA
[oracle@node2 bin]$ ./crs_stat -t -v
Name Type R/RA F/FT Target State Host
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE node1
ora....E1.lsnr application 0/5 0/0 ONLINE ONLINE node1
ora.node1.gsd application 0/5 0/0 ONLINE ONLINE node1
ora.node1.ons application 0/3 0/0 ONLINE ONLINE node1
ora.node1.vip application 0/0 0/0 ONLINE ONLINE node1
ora....SM2.asm application 0/5 0/0 OFFLINE OFFLINE
ora....E2.lsnr application 0/5 0/0 ONLINE ONLINE node2
ora.node2.gsd application 0/5 0/0 ONLINE ONLINE node2
ora.node2.ons application 0/3 0/0 ONLINE ONLINE node2
ora.node2.vip application 0/0 0/0 ONLINE ONLINE node2
i checked the status , asm is able to start on both nodes if not at the same time ,
when trying to start the second node , with srvctl or sqlplus , each give the error 03113
can anyone suggest me of how to bring up both instances ,
thanks~
[oracle@node2 bin]$ srvctl stop asm -n node1
[oracle@node2 bin]$ srvctl start asm -n node1
[oracle@node2 bin]$ srvctl start asm -n node2
PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Wed May 27 16:14:50 2009
node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:Copyright (c) 1982, 2005, Oracle. All rights reserved.
node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:Enter user-name: Connected to an idle instance.
node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:SQL> ORA-03113: end-of-file on communication channel
node2:ora.node2.ASM2.asm:SQL> Disconnected
node2:ora.node2.ASM2.asm:
[code/]
Edited by: zs_hzh on May 27, 2009 1:25 AMIs it possible to start ASM on second node with SQL*Plus in NOMOUNT state?
-
Scan-vip running only on one RAC node
Hi ,
While setting up RAC11.2 on Centos 5.7 , I was getting this error during the grid installation:
PRCR-1079 : Failed to start resource ora.scan1.vip
CRS-5005: IP Address: 192.168.100.208 is already in use in the network
CRS-2674: Start of 'ora.scan1.vip' on 'falcen6b' failed
CRS-2632: There are no more servers to try to place resource 'ora.scan1.vip' on that would satisfy its placement policy
PRCR-1079 : Failed to start resource ora.scan2.vip
CRS-5005: IP Address: 192.168.100.209 is already in use in the network
CRS-2674: Start of 'ora.scan2.vip' on 'falcen6b' failed
CRS-2632: There are no more servers to try to place resource 'ora.scan2.vip' on that would satisfy its placement policy
PRCR-1079 : Failed to start resource ora.scan3.vip
CRS-5005: IP Address: 192.168.100.210 is already in use in the network
CRS-2674: Start of 'ora.scan3.vip' on 'falcen6b' failed
CRS-2632: There are no more servers to try to place resource 'ora.scan3.vip' on that would satisfy its placement policy
I figured that the scan service is able to run only on one node at a time. When I stopped the service on rac1 and started it on rac2 the service is starting.
But I think for the grid installation the scan service has to simultaneously run on both the nodes.
How do I resolve it?
Any suggestions please.
PS - I am planning to try with the patch 11.0.2.3 but it will be a while till i get access to it.
Till then can someone suggest a workaround?Hi Balazs Papp and onedbguru,
I was able to resolve that error by running the following command on rac2, now that part of the installer passed.
crsctl start res ora.scan1.vip
However the cluster verification utility is failing at the end of installer.
When I executed the below command, this is my output:
[oracle@falcen6a grid]$ ./runcluvfy.sh stage -post crsinst -n falcen6a,falcen6b -verbose
Performing post-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "falcen6a"
Destination Node Reachable?
falcen6a yes
falcen6b yes
Result: Node reachability check passed from node "falcen6a"
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
falcen6b passed
falcen6a passed
Result: User equivalence check passed for user "oracle"
Checking time zone consistency...
Time zone consistency check passed.
Checking Cluster manager integrity...
Checking CSS daemon...
Node Name Status
falcen6b running
falcen6a running
Oracle Cluster Synchronization Services appear to be online.
Cluster manager integrity check passed
UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations
UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations
Check default user file creation mask
Node Name Available Required Comment
falcen6b 0022 0022 passed
falcen6a 0022 0022 passed
Result: Default user file creation mask check passed
Checking cluster integrity...
Cluster is divided into 2 partitions
Partition 1 consists of the following members:
Node Name
falcen6b
Partition 2 consists of the following members:
Node Name
falcen6a
Cluster integrity check failed. Cluster is divided into 2 partition(s).
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations
ERROR:
PRVF-4193 : Asm is not running on the following nodes. Proceeding with the remaining nodes.
Checking OCR config file "/etc/oracle/ocr.loc"...
OCR config file "/etc/oracle/ocr.loc" check successful
ERROR:
PRVF-4195 : Disk group for ocr location "+DATA" not available on the following nodes:
Checking size of the OCR location "+DATA" ...
Size check for OCR location "+DATA" successful...
OCR integrity check failed
Checking CRS integrity...
ERROR:
PRVF-5316 : Failed to retrieve version of CRS installed on node "falcen6b"
The Oracle clusterware is healthy on node "falcen6b"
The Oracle clusterware is healthy on node "falcen6a"
CRS integrity check failed
Checking node application existence...
Checking existence of VIP node application
Node Name Required Status Comment
falcen6b yes unknown failed
falcen6a yes unknown failed
Result: Check failed.
Checking existence of ONS node application
Node Name Required Status Comment
falcen6b no unknown ignored
falcen6a no online passed
Result: Check ignored.
Checking existence of GSD node application
Node Name Required Status Comment
falcen6b no unknown ignored
falcen6a no does not exist ignored
Result: Check ignored.
Checking existence of EONS node application
Node Name Required Status Comment
falcen6b no unknown ignored
falcen6a no online passed
Result: Check ignored.
Checking existence of NETWORK node application
Node Name Required Status Comment
falcen6b no unknown ignored
falcen6a no online passed
Result: Check ignored.
Checking Single Client Access Name (SCAN)...
SCAN VIP name Node Running? ListenerName Port Running?
falcen6-scan unknown false LISTENER 1521 false
WARNING:
PRVF-5056 : Scan Listener "LISTENER" not running
Checking name resolution setup for "falcen6-scan"...
SCAN Name IP Address Status Comment
falcen6-scan 192.168.100.210 passed
falcen6-scan 192.168.100.208 passed
falcen6-scan 192.168.100.209 passed
Verification of SCAN VIP and Listener setup failed
OCR detected on ASM. Running ACFS Integrity checks...
Starting check to see if ASM is running on all cluster nodes...
PRVF-5137 : Failure while checking ASM status on node "falcen6b"
Starting Disk Groups check to see if at least one Disk Group configured...
Disk Group Check passed. At least one Disk Group configured
Task ACFS Integrity check failed
Checking Oracle Cluster Voting Disk configuration...
Oracle Cluster Voting Disk configuration check passed
Checking to make sure user "oracle" is not in "root" group
Node Name Status Comment
falcen6b does not exist passed
falcen6a does not exist passed
Result: User "oracle" is not part of "root" group. Check passed
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
falcen6b passed
falcen6a passed
Result: CTSS resource check passed
Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
falcen6b Observer
falcen6a Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?
falcen6b yes
falcen6a yes
Result: Liveness check passed for "ntpd"
Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
Node Name Slewing Option Set?
falcen6b yes
falcen6a yes
Result:
NTP daemon slewing option check passed
Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"
Check: NTP daemon's boot time configuration
Node Name Slewing Option Set?
falcen6b yes
falcen6a yes
Result:
NTP daemon's boot time configuration check for slewing option passed
NTP common Time Server Check started...
NTP Time Server "133.243.236.19" is common to all nodes on which the NTP daemon is running
NTP Time Server "133.243.236.18" is common to all nodes on which the NTP daemon is running
NTP Time Server "210.173.160.86" is common to all nodes on which the NTP daemon is running
NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Checking on nodes "[falcen6b, falcen6a]"...
Check: Clock time offset from NTP Time Server
Time Server: 133.243.236.19
Time Offset Limit: 1000.0 msecs
Node Name Time Offset Status
falcen6b 15.332 passed
falcen6a -1.503 passed
Time Server "133.243.236.19" has time offsets that are within permissible limits for nodes "[falcen6b, falcen6a]".
Time Server: 133.243.236.18
Time Offset Limit: 1000.0 msecs
Node Name Time Offset Status
falcen6b 15.115 passed
falcen6a -1.614 passed
Time Server "133.243.236.18" has time offsets that are within permissible limits for nodes "[falcen6b, falcen6a]".
Time Server: 210.173.160.86
Time Offset Limit: 1000.0 msecs
Node Name Time Offset Status
falcen6b 15.219 passed
falcen6a -1.527 passed
Time Server "210.173.160.86" has time offsets that are within permissible limits for nodes "[falcen6b, falcen6a]".
Time Server: .LOCL.
Time Offset Limit: 1000.0 msecs
Node Name Time Offset Status
falcen6b 0.0 passed
falcen6a 0.0 passed
Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[falcen6b, falcen6a]".
Clock time offset check passed
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Oracle Cluster Time Synchronization Services check passed
Post-check for cluster services setup was unsuccessful on all the nodes.
[oracle@falcen6a grid]$
Any suggestions? -
Failed to start Oracle Weblogic Server node in 11g
Hello friends,
I have a problem starting the Weblogic Server node Oracle 11g (10.3.5).
Any suggestions to solve the problem
Logs----------------------------------------------------------------------------------
<Nov 3, 2011 12:33:03 PM> <INFO> <NodeManager> <Server output log file is '/opt/middleware/user_projects/domains/DomainOWL/servers/Server01/logs/Server01.out'>
<Nov 3, 2011 12:33:07 PM PET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Nov 3, 2011 12:33:07 PM PET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<Nov 3, 2011 12:33:11 PM PET> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) Server VM Version 19.1-b02 from Sun Microsystems Inc.>
<Nov 3, 2011 12:33:18 PM PET> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<Nov 3, 2011 12:33:24 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Nov 3, 2011 12:33:24 PM PET> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Nov 3, 2011 12:33:25 PM PET> <Notice> <Log Management> <BEA-170019> <The server log file /opt/middleware/user_projects/domains/DomainOWL/servers/Server01/logs/Server01.log is opened. All server side log events will be written to this file.>
<Nov 3, 2011 12:33:25 PM PET> <Warning> <NodeManager> <BEA-300043> <Node manager native library not found - server process id not saved.>
<Nov 3, 2011 12:33:26 PM PET> <Error> <Socket> <BEA-000438> <Unable to load performance pack. Using Java I/O instead. Please ensure that a native performance library is in: '"/opt/jdk1.6.0/jre/lib/sparcv9/server:/opt/jdk1.6.0/jre/lib/sparcv9:/opt/jdk1.6.0/jre/../lib/sparcv9:/opt/middleware/patch_wls1035/profiles/default/native:/opt/middleware/patch_ocp360/profiles/default/native:/opt/middleware/wlserver_10.3/server/native/solaris/sparc64:/opt/middleware/wlserver_10.3/server/native/solaris/sparc64:/opt/middleware/wlserver_10.3/server/native/solaris/sparc64/oci920_8:/usr/jdk/packages/lib/sparcv9:/lib/64:/usr/lib/64"'
>
<Nov 3, 2011 12:33:37 PM PET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Nov 3, 2011 12:33:40 PM PET> <Warning> <Store> <BEA-280101> <The persistent file store "_WLS_Server01" is forced to use buffered I/O and so may have significantly degraded performance. Either the OS/hardware environment does not support the chosen write policy or the native wlfileio library is missing. See store open log messages for the requested and final write policies. See the documentation on store synchronous write policy configuration for advice.>
<Nov 3, 2011 12:33:49 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Nov 3, 2011 12:33:49 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Nov 3, 2011 12:33:56 PM PET> <Notice> <Log Management> <BEA-170027> <The Server has established connection with the Domain level Diagnostic Service successfully.>
<Nov 3, 2011 12:33:56 PM PET> <Notice> <Cluster> <BEA-000197> <Listening for announcements from cluster using unicast cluster messaging>
<Nov 3, 2011 12:33:56 PM PET> <Notice> <Cluster> <BEA-000133> <Waiting to synchronize with other running members of oimClusterT.>
<Nov 3, 2011 12:34:26 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN>
<Nov 3, 2011 12:34:27 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING>
<Nov 3, 2011 12:34:27 PM PET> <Notice> <Cluster> <BEA-000162> <Starting "async" replication service with remote cluster address "null">
<Nov 3, 2011 12:34:31 PM PET> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias DemoIdentity from the jks keystore file /opt/middleware/wlserver_10.3/server/lib/DemoIdentity.jks.>
<Nov 3, 2011 12:34:31 PM PET> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/middleware/wlserver_10.3/server/lib/DemoTrust.jks.>
<Nov 3, 2011 12:34:31 PM PET> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/jdk1.6.0/jre/lib/security/cacerts.>
<Nov 3, 2011 12:34:33 PM PET> <Alert> <Security> <BEA-090152> <Demo trusted CA certificate is being used in production mode: [
Version: V3
Subject: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4
Key: SunPKCS11-Solaris RSA public key, 512 bits (id 58638024, session object)
modulus: 9550192877869244258838480703390456015046425375252278279190673063544122510925482179963329236052146047356415957587628011282484772458983977898996276815440753
public exponent: 65537
Validity: [From: Thu Mar 21 15:12:27 PET 2002,
To: Tue Mar 22 15:12:27 PET 2022]
Issuer: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
SerialNumber: [ 33f10648 fcde0deb 4199921f d64537f4]
Certificate Extensions: 1
[1]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Algorithm: [MD5withRSA]
Signature:
0000: 9D 26 4C 29 C8 91 C3 A7 06 C3 24 6F AE B4 F8 82 .&L)......$o....
0010: 80 4D AA CB 7C 79 46 84 81 C4 66 95 F4 1E D8 C4 .M...yF...f.....
0020: E9 B7 D9 7C E2 23 33 A4 B7 21 E0 AA 54 2B 4A FF .....#3..!..T+J.
0030: CB 21 20 88 81 21 DB AC 90 54 D8 7D 79 63 23 3C .! ..!...T..yc#<
] The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.>
<Nov 3, 2011 12:34:33 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=Entrust Root Certification Authority - G2,OU=(c) 2009 Entrust\, Inc. - for authorized use only,OU=See www.entrust.net/legal-terms,O=Entrust\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:33 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=thawte Primary Root CA - G3,OU=(c) 2008 thawte\, Inc. - For authorized use only,OU=Certification Services Division,O=thawte\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 3,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 2,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=GlobalSign,O=GlobalSign,OU=GlobalSign Root CA - R3". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "OU=Security Communication RootCA2,O=SECOM Trust Systems CO.\,LTD.,C=JP". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=VeriSign Universal Root Certification Authority,OU=(c) 2008 VeriSign\, Inc. - For authorized use only,OU=VeriSign Trust Network,O=VeriSign\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=KEYNECTIS ROOT CA,OU=ROOT,O=KEYNECTIS,C=FR". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=GeoTrust Primary Certification Authority - G3,OU=(c) 2008 GeoTrust Inc. - For authorized use only,O=GeoTrust Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Server> <BEA-002613> <Channel "DefaultSecure" is now listening on 192.168.1.10:7006 for protocols iiops, t3s, CLUSTER-BROADCAST-SECURE, ldaps, https.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.1.10:7005 for protocols iiop, t3, CLUSTER-BROADCAST, ldap, snmp, http.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <WebLogicServer> <BEA-000330> <Started WebLogic Managed Server "Server01" for domain "DomainOWL" running in Production Mode>
Exception in thread "[STANDBY] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:217)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Exception in thread "[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:36:27 PM PET> <Error> <HTTP> <BEA-101020> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Servlet failed with Exception
java.lang.OutOfMemoryError: PermGen space
>
Exception in thread "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Timer-1" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:36:53 PM PET> <Error> <HTTP> <BEA-101020> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Servlet failed with Exception
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:36:24 PM PET> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: PermGen space.
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:36:33 PM PET> <Error> <JMX> <BEA-149501> <An exception occurred while registering the MBean com.bea:Name=idmAdminServerT,Type=Server at property WebService.
java.lang.OutOfMemoryError: PermGen space
>
Exception in thread "[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:37:16 PM PET> <Warning> <Socket> <BEA-000402> <There are: 5 active sockets, but the maximum number of socket reader threads allowed by the configuration is: 4. You may want to alter your configuration.>
Exception in thread "[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:39:41 PM PET> <Error> <HTTP> <BEA-101216> <Servlet: "CoordinatorPortTypeServlethttp" failed to preload on startup in Web application: "wls-wsat.war".
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:42:32 PM PET> <Error> <HTTP> <BEA-101017> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Root cause of ServletException.
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:43:15 PM PET> <Error> <HTTP> <BEA-101017> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Root cause of ServletException.
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:43:26 PM PET> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: PermGen space.
java.lang.OutOfMemoryError: PermGen space
>
Exception in thread "[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
Exception in thread "[STANDBY] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:45:53 PM PET> <Error> <HTTP> <BEA-101017> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Root cause of ServletException.
java.lang.OutOfMemoryError: PermGen space
>
Exception in thread "[ACTIVE] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "weblogic.timers.TimerThread" java.lang.OutOfMemoryError: PermGen space
Thanks<Nov 4, 2011 11:42:08 AM PET> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: PermGen space.
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
Truncated. see log file for complete stacktrace
>
<Nov 4, 2011 11:42:34 AM PET> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
at java.lang.Class.getDeclaredMethod(Class.java:1935)
at java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1349)
at java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:52)
Truncated. see log file for complete stacktrace
>
The WebLogic Server encountered a critical failure
Reason: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "main" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 4, 2011 11:44:08 AM PET> <Warning> <Socket> <BEA-000402> <There are: 5 active sockets, but the maximum number of socket reader threads allowed by the configuration is: 4. You may want to alter your configuration.>
Exception in thread "[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 4, 2011 11:44:43 AM PET> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: PermGen space.
java.lang.OutOfMemoryError: PermGen space
I made the change and continue with the same problem, you can tell me which file should make this change, very grateful -
11GR2 Dbconsole failed to start
Hi all,
Here is the content of agabend.log file :
The agentTZRegion value in /home/ora11202/11.2.0/anatole_BDIT/sysman/config/emd.properties is not in agreement with what agent thinks it s
hould be.Please verify your environment to make sure that TZ setting has not changed since the last start of the agent.
If you modified the timezone setting in the environment, please stop the agent and exectute 'emctl resetTZ agent' and also execute the script
mgmt_target.set_agent_tzrgn(<agent_name>, <new_tz_rgn>) to get the value propagated to repository.
I'm on AIX 5.3
I deconfigure and reconfigure from scratch, with :
#export TZ=Europe/Paris
#emca -config dbcontrol db -repos create
But, DB console failed to start.
I don't know what to do.
Thanks a lot.Hi;
Pelase see
Grid Agent Configuration: How to Troubleshoot Timezone Issues in Enterprise Manager 10g Grid Agent Setup [ID 330737.1]
Grid Control Agent Startup Fails on AIX Server due to Invalid Timezone Setting at the OS [ID 825933.1]
Regard
Helios -
One RAC node is down give the following error when starting the database!
wHEN TRYING TO START THE DATABASE ON RAC ENVIORNMENT
SQL> connect sys as sysdba
Enter password:
Connected to an idle instance.
SQL> startup
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
SQL> exit
Disconnected
When we r trying to start the database it said
$ bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 2097152 240944 1841896 12% /
/dev/vg00/lvol1 344064 115376 226944 34% /stand
/dev/vg00/lvol8 10485760 9370960 1106232 89% /var
/dev/vg00/lvol7 4866048 2557680 2290400 53% /usr
/dev/vg00/u02 10485760 3502229 6547116 35% /u02
/dev/vg00/u01 10485760 10476596 9164 100% /u01
/dev/vg00/lvol4 2097152 601872 1483944 29% /tmp
/dev/vg00/lvol6 4194304 3231000 955792 77% /opt
/dev/vg00/lvol5 524288 311520 211136 60% /home
WHERE /U01 WAS 100%. Now i emptied the space in /u01 to
$ bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 2097152 240944 1841896 12% /
/dev/vg00/lvol1 344064 115376 226944 34% /stand
/dev/vg00/lvol8 10485760 9370960 1106232 89% /var
/dev/vg00/lvol7 4866048 2557680 2290400 53% /usr
/dev/vg00/u02 10485760 3502229 6547116 35% /u02
/dev/vg00/u01 10485760 9508934 930943 91% /u01
/dev/vg00/lvol4 2097152 601872 1483944 29% /tmp
/dev/vg00/lvol6 4194304 3231000 955792 77% /opt
/dev/vg00/lvol5 524288 311520 211136 60% /home
When trying to start the db again its giving the following error...
SQL> connect sys as sysdba
Enter password:
Connected to an idle instance.
SQL> startup
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
SQL> exit
Disconnected
here i changed the sga_target and now it says
ORACLE instance started.
Total System Global Area 436207616 bytes
Fixed Size 1297912 bytes
Variable Size 148648456 bytes
Database Buffers 285212672 bytes
Redo Buffers 1048576 bytes
ORA-01105: mount is incompatible with mounts by other instances
ORA-19808: recovery destination parameter mismatch
What could be the issue..
Ur help would be highly appreciated...Hello
SQL> startup
ORA-27102: out of memory
HPUX-ia64 Error: 12: Not enough space
SQL> exiterror is not related to space on your mount point. it is related to memory.
if you are getting this error means chekc at the OS level whether something is consuming more memory due to which it is not allowing oracle to allocate sga.
Check top/sar/glance to see who is consuming more memory
Total System Global Area 436207616 bytes
Fixed Size 1297912 bytes
Variable Size 148648456 bytes
Database Buffers 285212672 bytes
Redo Buffers 1048576 bytes
ORA-01105: mount is incompatible with mounts by other instances
ORA-19808: recovery destination parameter mismatchit is not the best practice to maintain differnet parameters for each instance in RAC env. also check db_recovery_file_dest and db_recovery_file_dest_size is same on all node. it should be same i.e it should be a shared location.
Anil Malkai -
Archive REDO When One RAC Node is Down
I have a question about how redo log get archived when one of the instances in a two node RAC cluster is down (not open, not mounted).
For example, let's assume instance1 was shutdown and only instance 2 is running.
I have 6 redo logs:
<font face="courier">
SQL> SELECT GROUP#, THREAD#, SEQUENCE#, STATUS FROM V$LOG;
GROUP# THREAD# SEQUENCE# STATUS
1 1 3390 INACTIVE
2 1 3389 INACTIVE
3 1 3391 ACTIVE
5 2 3886 INACTIVE
4 2 3887 INACTIVE
6 2 3888 CURRENT
</font>
If I run the following statement what will happen?
<font face="courier">
SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;
</font>
The documentation says +Specify CURRENT to manually archive the current redo log file group of the specified thread, forcing a log switch. If you omit the THREAD parameter, then Oracle Database archives all redo log file groups from all enabled threads, including logs previous to current logs. You can specify CURRENT only when the database is open.+
Would Oracle archive sequence# 3391 from thread 1 even though the instance is not open?When your instance are not working, it means that it doesn't have any CURRENT redo log. So when you issue switching logfile - it launches archiving only current redo log, for working instance only. That 3391 redo log is not current, it has Active status, that means that it could be needed for recovery purpose, but it had to be archived earlier.
-
Getting Error in starting VIP in 3 NODE RAC Cluster in VMWARE
hi
please can some one help me to have solution for why VIPCA is failing to start VIP on RAC Node 3 it gives the ERROR: CRS-1006; CRS-0215 no more members. Network Configuration is like:
/etc/hosts
127.0.0.1 localhost.localdomain localhost
#Public IP
192.168.2.131 rac1.sun.com rac1
192.168.2.132 rac2.sun.com rac2
192.168.2.133 rac3.sun.com rac3
#Private IP
10.10.10.31 rac1-priv rac1-priv
10.10.10.32 rac2-priv rac2-priv
10.10.10.33 rac3-priv rac3-priv
#Virtual IP
192.168.2.131 rac1-vip.sun.com rac1-vip
192.168.2.132 rac2-vip.sun.com rac2-vip
192.168.2.133 rac3-vip.sun.com rac3-vip
/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=rac1.sun.com
GATEWAY=192.168.2.1
Thanks in Advanceyou should have to user some other new ips for VIP.
PLEASE CHANGE THE VIP IP's and try again.
192.168.2.131 rac1-vip.sun.com rac1-vip
192.168.2.132 rac2-vip.sun.com rac2-vip
192.168.2.133 rac3-vip.sun.com rac3-vipchange the ips to some other ip not used by any machines.
sample /etc/hosts file
127.0.0.1 localhost.localdomain localhost
# Public
10.1.10.201 rac1.localdomain rac1
10.1.10.202 rac2.localdomain rac2
#Private
10.1.9.201 rac1-priv.localdomain rac1-priv
10.1.9.202 rac2-priv.localdomain rac2-priv
#Virtual
*10.1.10.203 rac1-vip.localdomain rac1-vip*
*10.1.10.204 rac2-vip.localdomain rac2-vip* -
I have done all the steps to remove one RAC node but got stuck at the step of running rootdelete.sh file from $CRS_HOME/install directory as I don't have this file in windows environment.
What is the equivalent file for rootdelete.sh on windows platform. I want to run this to remove the node info from the clusterware entry.
Is there a good document that explains about removing the node on windows platform.Hello,
You need to run the following steps to remove a node from a RAC cluster on Windows platform:
Perform the following steps on a node other than the node you want to delete:
1. Run the Database Configuration Assistant (DBCA) utility to delete the instance.
2. Then run the Net Configuration Assistant (NetCA) to delete the listener.
3. If the node that you are deleting has ASM instance, then delete the ASM instance using the srvctl stop asm and srvctl remove asm commands.
4. Run the command srvctl stop nodeapps -n nodename of the node to be deleted to stop the node applications.
5. Run the command srvctl remove nodeapps -n nodename of the node to be deleted to remove the node applications.
6. Stop isqlplus if it is running.
7. Run the command setup.exe -updateNodeList ORACLE_HOME=Oracle_home ORACLE_HOME_NAME=Oracle_home_name CLUSTER_NODES=remaining
nodes where remaining nodes is a list of the nodes that are to remain part of the cluster.
Perform the following steps on the deleted RAC node:
1. Run the command setup.exe -updateNodeList -local -noClusterEnabled ORACLE_HOME=Oracle_home ORACLE_HOME_NAME=Oracle_home_name CLUSTER_NODES="".
Note that you do not need a value for "" after the CLUSTER_NODES= entry in this command. If you delete more than one node, then you must run this command on every deleted node to remove the Oracle home if you have a non-shared Oracle home (non-cluster file system) installation.
2. On the same node, delete the Windows Registry entries and ASM services using Oradim.
3. From the deleted RAC node, run the command Oracle_home\oui\bin\setup.exe to start the Oracle Universal Installer (OUI). Select Deinstall Products and select the Oracle home that you want to de-install.
4. Then to delete the CRS node, from a remaining node run the command crssetup del -nn node_name of the deleted node, node number
5. Then run the command setup.exe -updateNodeList ORACLE_HOME=CRS home ORACLE_HOME_NAME=CRS home name CLUSTER_NODES=remaining nodes where remaining nodes is a list of the nodes that are to remain in the cluster.
6. Then on the deleted CRS node, run the command setup.exe -updateNodeList -local -noClusterEnabled ORACLE_HOME=CRS home ORACLE_HOME_NAME=CRS home name CLUSTER_NODES=""
7. Remove the Oracle home manually from the new node if the home is not shared and then manually remove the HKLM/software/Oracle registry keys and the Oracle services. 7
8. After adding or deleting nodes from your Oracle Database 10g with RAC environment, and after you are sure that your system is functioning properly, make a backup of the contents of the voting disk using the dd.exe utility. The dd.exe utility is part of the MKS toolkit.
ASM Instance Cleanup Procedures after Node Deletion on Windows-Based Platforms
The delete node procedure requires the following additional steps on Windows-based systems to remove the ASM instances:
1. If this is the Oracle home from which the node-specific listener named LISTENER_nodename runs, then use NetCA to remove this listener and its CRS resources. If necessary, re-create this listener in another home.
2. If this is the Oracle home from which the ASM instance runs, then remove the ASM configuration by running the following command for all nodes on which this Oracle home exists:
srvctl stop asm -n node
Then run the following command for the nodes that you are removing:
srvctl remove asm -n node
3. If you are using a cluster file system for your ASM Oracle home, then run the following commands on the local node:
4. rd -s -q %ORACLE_BASE%\admin\+ASM
delete %ORACLE_HOME%\database\*ASM*
5. If you are not using a cluster file system for your ASM Oracle home, then run the delete command mentioned in the previous step on each node on which the Oracle home exists.
6. Run the following command on each node that has an ASM instance:
oradim -delete -asmsid +ASMnode_number
Source:
Oracle® Real Application Clusters Administrator's Guide
10g Release 1 (10.1)
Part Number B10765-02
Chapter 5: Adding and Deleting Nodes and Instances
Hope this helps,
Ben Prusinski, Oracle 10g OCP
http://oracle-magician.blogspot.com -
Until recently I have been surfing the net ok, and then my browser just failed to start up one day. I have tried and tried to get it working but its failing. I even did a virus scan but this didn't help. Please help...
m8,
Backup your phone & do a factory reset, or better still format it!
Knowledge not shared is knowledge wasted!
If you find it helpfull, it's not hard to click the STAR..
Maybe you are looking for
-
How to retrieve multiple data from table and represent it in jsp page
Hi The below JavaScript code is used to add row in the table when I want to add multiple row data into table for single entry no field. <html> function addRow() i++; var newRow = document.all("tblGrid").insertRow();
-
I just updated itunes and started to update my phone to iSo5 when wireless connection was lost on my laptop, I can't get it back and it says there are no networks available-router is working fine, am using it on my iPhone now. Help!
-
ITunes on OS X vs Microsoft Windows 8.1
iTunes: 12.0.1.26 I recently completely switched to a Mac from a PC. I was told by an Apple store 'genius' that I could simply copy my music library from my old pc to my new MBP (from the copy I made of my iTunes library on my old pc). Ever since I
-
MSS Configuration Documentation needed.
Hello Gurus, Can somebody please send me the configuration documentation on MSS. We are currently implementing Business package for MSS 60.1 on ECC 5.0. Thanks a ton, Sasi.
-
Hello! I purchased a new 160 GB ipod classic and while trying to connect to itunes for the first time,the iPod said I need to restore it and when I plugged it in it started making the ticking sound it makes when you roll the click wheel and I tried t