Failed to start GSD on local node
I am trying to install Oracle 9.2 Real Application Cluster on Linux 7.3.
Everything is OK until DBCA asks to run gsdctl. After running gsdctl start in
several minutes I can see "Failed to start GSD on local node" without any
explanations.
What should I do?
To get more information about the problem, I've seen the following used in the "gsd.sh" script:
exec $JRE -nojit -DPROGRAM=gsd -DTRACING.ENABLED=true -DTRACING.LEVEL=0 -classpath $CLASSPATH oracle.ops.mgmt.daemon.OPSMDaemon $MY_OHOME
Start the gsd with: "gsd.sh" rather than "gsdctl start".
The "TRACING.ENABLED" and "TRACING.LEVEL" will cause the JRE to emit a great deal of debug information. The "nojit" option turns off the "just in time" feature, but I don't know if that's too important (same error, either way). I found this information through some newsgroup. However, I haven't actually gotten the "gsd" daemon to run for more than a few minutes. After a minute or two, the error is as follows:
[main] [20:19:6:880] [OCRTree.<init>:80] Going to load the ocr librarysrvmocr
[main] [20:19:6:883] [OCRTree.<init>:80] loaded ocr libraries
[main] [20:19:6:884] [OCRTree.<init>:80] in OCR constructor
[main] [20:19:6:885] [OCRTree.<init>:80] Initializing OCR: NEW
initializeDaemon: localNodeName = orac01
Native Code Debug is ONinitializeOCR: procr_init retval = 0
SIGSEGV 11* segmentation violation
stackbase=BFFFF308, stackpointer=BFFFC6F4
Full thread dump:
"process reaper" (TID:0x404a4800, sys_thread_t:0x815c8f8, state:CW, thread_t: t@49159, sp:0x0
threadID:0xbfa, stack_base:0x413fba50, stack_size:0x200000) prio=0
java.lang.Thread.run(Thread.java)
"KeepAlive" (TID:0x404a4200, sys_thread_t:0x8148be8, state:CW, thread_t: t@40966, sp:0x0 threadID:0xbf9, stack_base:0x413daa50, stack_size:0x200000) prio=0
sun.rmi.transport.KeepAlive.run(ObjectTable.java:182)
java.lang.Thread.run(Thread.java)
"Reaper" (TID:0x404a3e38, sys_thread_t:0x814ea48, state:CW, thread_t: t@32773, sp:0x0 threadID:0xbf8, stack_base:0x413b9a50, stack_size:0x200000) prio=-1073756771
sun.rmi.transport.Reaper.run(ObjectTable.java:199)
java.lang.Thread.run(Thread.java)
"TCP Accept-1" (TID:0x404a3d88, sys_thread_t:0x8148898, state:R, thread_t: t@24580, sp:0x0 threadID:0xbf7, stack_base:0x41398a50, stack_size:0x200000) prio=-1073756774
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:379)
java.net.ServerSocket.implAccept(ServerSocket.java:199)
java.net.ServerSocket.accept(ServerSocket.java:181)
sun.rmi.transport.proxy.HttpAwareServerSocket.accept(HttpAwareServerSocket.java:70)
sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:376)
java.lang.Thread.run(Thread.java)
"SIGQUIT handler" (TID:0x404972a0, sys_thread_t:0x807d768, state:R, thread_t: t@16387, sp:0x0
threadID:0xbf6, stack_base:0x411a7a50, stack_size:0x200000) prio=-1073756763
"Finalizer thread" (TID:0x40497088, sys_thread_t:0x807d640, state:CW, thread_t: t@8194, sp:0x0 threadID:0xbf5, stack_base:0x41186a50, stack_size:0x200000) prio=-1073756763
"main" (TID:0x404970b0, sys_thread_t:0x8064810, state:R, thread_t: t@8192, sp:0x0 threadID:0xbe9, stack_base:0xbffff308, stack_size:0x200000) prio=-1073756763 current thread
oracle.ops.mgmt.rawdevice.OCR.<init>(OCR.java:101)
oracle.ops.mgmt.rawdevice.OCRTree.<init>(OCRTree.java:80)
oracle.ops.mgmt.rawdevice.OCRTree.init(OCRTree.java:93)
oracle.ops.mgmt.rawdevice.RawDeviceConfig.<init>(RawDeviceConfig.java:98)
oracle.ops.mgmt.rawdevice.RawDeviceConfig.init(RawDeviceConfig.java:113)
oracle.ops.mgmt.daemon.OPSMDaemon.<init>(OPSMDaemon.java:207)
oracle.ops.mgmt.daemon.OPSMDaemon.main(OPSMDaemon.java:726)
Monitor Cache Dump:
java.net.PlainSocketImpl@1078606656/1079100848: owner "TCP Accept-1" (0x8148898, 1 entry)
<unknown key> (0x0x8148be8): <unowned>
Waiting to be notified:
"KeepAlive" (0x8148be8)
java.lang.Class@1078622888/1079239192: owner "main" (0x8064810, 1 entry)
<unknown key> (0x0x814ea48): <unowned>
Waiting to be notified:
"Reaper" (0x814ea48)
java.lang.Class@1078623280/1079245648: owner "main" (0x8064810, 1 entry)
Registered Monitor Dump:
Fork_Wait_monitor: <unowned>
Waiting to be notified:
"process reaper" (0x815c8f8)
Thread queue lock: <unowned>
Name and type hash table lock: <unowned>
String intern lock: <unowned>
JNI pinning lock: <unowned>
JNI global reference lock: <unowned>
BinClass lock: <unowned>
Class loading lock: <unowned>
Java stack lock: <unowned>
Code rewrite lock: <unowned>
Heap lock: <unowned>
Has finalization queue lock: <unowned>
Finalize me queue lock: <unowned>
Waiting to be notified:
"Finalizer thread" (0x807d640)
Monitor registry: owner "main" (0x8064810, 1 entry)
Aborted
This error was generated with RedHat Linux 8, the normal edition, and Oracle 9iR2 ORAC. Is this the error you generated? Perhaps we are missing something special about RedHad AS? I would certainly appreciate it if someone could offer specifics.
Similar Messages
-
Unable to start gsd on local node
Hi all, I am working in a live enviorment and in of help. I am wokring in a RAC enviorment and unable to start "gsd" on one of the nodes, all the rest of the services are online. Following is the visual display. Can some one tell me what I am doing wrong.
<fms-db2:/oracle/app/oracle/product/10.2>crs_stat -t
Name Type Target State Host
ora....DB1.srv application ONLINE ONLINE fms-db1
ora....MSDB.cs application ONLINE ONLINE fms-db2
ora....B1.inst application ONLINE ONLINE fms-db1
ora....B2.inst application ONLINE ONLINE fms-db2
ora.FMSDB.db application ONLINE ONLINE fms-db2
ora....B1.lsnr application ONLINE ONLINE fms-db1
ora....db1.gsd application ONLINE ONLINE fms-db1
ora....db1.ons application ONLINE ONLINE fms-db1
ora....db1.vip application ONLINE ONLINE fms-db1
ora....B2.lsnr application ONLINE ONLINE fms-db2
ora....db2.gsd application ONLINE OFFLINE
ora....db2.ons application ONLINE ONLINE fms-db2
ora....db2.vip application ONLINE ONLINE fms-db2
<fms-db2:/oracle/app/oracle/product/10.2>gsd stat
Could not start the daemon on the local node.
<fms-db2:/oracle/app/oracle/product/10.2>gsd start
Could not start the daemon on the local node.
<fms-db2:/oracle/app/oracle/product/10.2>Hi,
Check the crsd.log under $ORA_CRS_HOME/log/$(hostname)/crsd
That will show you what the error was. At that point, you can look up the error in the documentation (not before, as others posters seem to think). All gsd does is coordinate access amongst the various tools such as dbca, so it isn't "life threatening", anyway. Usually if one dies, though, others may very likely follow.
HTH,
Steve -
RAC, ASM failed to start up on second node , ORA-03113: end-of-file on comm
i'm installing an RAC with 2 nodes on top of ASM
when creating ASM Diskgroup , it failed and reported error CRS-0215 failed to start asm on node2
Oracle 10.2.0.1
linux CentOs 4.x
u01/app/oracle/product/10.2.0/db_1/bin/dbca -progress_only -configureASM -templateName NO_VALUE -gdbName NO -sid NO -emConf
iguration NONE -diskList /dev/raw/raw2,/dev/raw/raw3 -diskGroupName DATA -datafileJarLocation /u01/app/oracle/product/10.2.0/db_
1/assistants/dbca/templates -responseFile NO_VALUE -nodeinfo node1,node2 -obfuscatedPasswords true -oratabLocation /u01/app/o
racle/product/10.2.0/db_1/install/oratab -asmSysPassword 05dbb0be38ecf8cca822cf3cf99e675448 -redundancy EXTERNA
[oracle@node2 bin]$ ./crs_stat -t -v
Name Type R/RA F/FT Target State Host
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE node1
ora....E1.lsnr application 0/5 0/0 ONLINE ONLINE node1
ora.node1.gsd application 0/5 0/0 ONLINE ONLINE node1
ora.node1.ons application 0/3 0/0 ONLINE ONLINE node1
ora.node1.vip application 0/0 0/0 ONLINE ONLINE node1
ora....SM2.asm application 0/5 0/0 OFFLINE OFFLINE
ora....E2.lsnr application 0/5 0/0 ONLINE ONLINE node2
ora.node2.gsd application 0/5 0/0 ONLINE ONLINE node2
ora.node2.ons application 0/3 0/0 ONLINE ONLINE node2
ora.node2.vip application 0/0 0/0 ONLINE ONLINE node2
i checked the status , asm is able to start on both nodes if not at the same time ,
when trying to start the second node , with srvctl or sqlplus , each give the error 03113
can anyone suggest me of how to bring up both instances ,
thanks~
[oracle@node2 bin]$ srvctl stop asm -n node1
[oracle@node2 bin]$ srvctl start asm -n node1
[oracle@node2 bin]$ srvctl start asm -n node2
PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Wed May 27 16:14:50 2009
node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:Copyright (c) 1982, 2005, Oracle. All rights reserved.
node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:Enter user-name: Connected to an idle instance.
node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:SQL> ORA-03113: end-of-file on communication channel
node2:ora.node2.ASM2.asm:SQL> Disconnected
node2:ora.node2.ASM2.asm:
[code/]
Edited by: zs_hzh on May 27, 2009 1:25 AMIs it possible to start ASM on second node with SQL*Plus in NOMOUNT state?
-
Failed to start Oracle Weblogic Server node in 11g
Hello friends,
I have a problem starting the Weblogic Server node Oracle 11g (10.3.5).
Any suggestions to solve the problem
Logs----------------------------------------------------------------------------------
<Nov 3, 2011 12:33:03 PM> <INFO> <NodeManager> <Server output log file is '/opt/middleware/user_projects/domains/DomainOWL/servers/Server01/logs/Server01.out'>
<Nov 3, 2011 12:33:07 PM PET> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Nov 3, 2011 12:33:07 PM PET> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<Nov 3, 2011 12:33:11 PM PET> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) Server VM Version 19.1-b02 from Sun Microsystems Inc.>
<Nov 3, 2011 12:33:18 PM PET> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0 Fri Apr 1 20:20:06 PDT 2011 1398638 >
<Nov 3, 2011 12:33:24 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Nov 3, 2011 12:33:24 PM PET> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Nov 3, 2011 12:33:25 PM PET> <Notice> <Log Management> <BEA-170019> <The server log file /opt/middleware/user_projects/domains/DomainOWL/servers/Server01/logs/Server01.log is opened. All server side log events will be written to this file.>
<Nov 3, 2011 12:33:25 PM PET> <Warning> <NodeManager> <BEA-300043> <Node manager native library not found - server process id not saved.>
<Nov 3, 2011 12:33:26 PM PET> <Error> <Socket> <BEA-000438> <Unable to load performance pack. Using Java I/O instead. Please ensure that a native performance library is in: '"/opt/jdk1.6.0/jre/lib/sparcv9/server:/opt/jdk1.6.0/jre/lib/sparcv9:/opt/jdk1.6.0/jre/../lib/sparcv9:/opt/middleware/patch_wls1035/profiles/default/native:/opt/middleware/patch_ocp360/profiles/default/native:/opt/middleware/wlserver_10.3/server/native/solaris/sparc64:/opt/middleware/wlserver_10.3/server/native/solaris/sparc64:/opt/middleware/wlserver_10.3/server/native/solaris/sparc64/oci920_8:/usr/jdk/packages/lib/sparcv9:/lib/64:/usr/lib/64"'
>
<Nov 3, 2011 12:33:37 PM PET> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Nov 3, 2011 12:33:40 PM PET> <Warning> <Store> <BEA-280101> <The persistent file store "_WLS_Server01" is forced to use buffered I/O and so may have significantly degraded performance. Either the OS/hardware environment does not support the chosen write policy or the native wlfileio library is missing. See store open log messages for the requested and final write policies. See the documentation on store synchronous write policy configuration for advice.>
<Nov 3, 2011 12:33:49 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Nov 3, 2011 12:33:49 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Nov 3, 2011 12:33:56 PM PET> <Notice> <Log Management> <BEA-170027> <The Server has established connection with the Domain level Diagnostic Service successfully.>
<Nov 3, 2011 12:33:56 PM PET> <Notice> <Cluster> <BEA-000197> <Listening for announcements from cluster using unicast cluster messaging>
<Nov 3, 2011 12:33:56 PM PET> <Notice> <Cluster> <BEA-000133> <Waiting to synchronize with other running members of oimClusterT.>
<Nov 3, 2011 12:34:26 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN>
<Nov 3, 2011 12:34:27 PM PET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING>
<Nov 3, 2011 12:34:27 PM PET> <Notice> <Cluster> <BEA-000162> <Starting "async" replication service with remote cluster address "null">
<Nov 3, 2011 12:34:31 PM PET> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias DemoIdentity from the jks keystore file /opt/middleware/wlserver_10.3/server/lib/DemoIdentity.jks.>
<Nov 3, 2011 12:34:31 PM PET> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/middleware/wlserver_10.3/server/lib/DemoTrust.jks.>
<Nov 3, 2011 12:34:31 PM PET> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/jdk1.6.0/jre/lib/security/cacerts.>
<Nov 3, 2011 12:34:33 PM PET> <Alert> <Security> <BEA-090152> <Demo trusted CA certificate is being used in production mode: [
Version: V3
Subject: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4
Key: SunPKCS11-Solaris RSA public key, 512 bits (id 58638024, session object)
modulus: 9550192877869244258838480703390456015046425375252278279190673063544122510925482179963329236052146047356415957587628011282484772458983977898996276815440753
public exponent: 65537
Validity: [From: Thu Mar 21 15:12:27 PET 2002,
To: Tue Mar 22 15:12:27 PET 2022]
Issuer: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
SerialNumber: [ 33f10648 fcde0deb 4199921f d64537f4]
Certificate Extensions: 1
[1]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
Algorithm: [MD5withRSA]
Signature:
0000: 9D 26 4C 29 C8 91 C3 A7 06 C3 24 6F AE B4 F8 82 .&L)......$o....
0010: 80 4D AA CB 7C 79 46 84 81 C4 66 95 F4 1E D8 C4 .M...yF...f.....
0020: E9 B7 D9 7C E2 23 33 A4 B7 21 E0 AA 54 2B 4A FF .....#3..!..T+J.
0030: CB 21 20 88 81 21 DB AC 90 54 D8 7D 79 63 23 3C .! ..!...T..yc#<
] The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.>
<Nov 3, 2011 12:34:33 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=Entrust Root Certification Authority - G2,OU=(c) 2009 Entrust\, Inc. - for authorized use only,OU=See www.entrust.net/legal-terms,O=Entrust\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:33 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=thawte Primary Root CA - G3,OU=(c) 2008 thawte\, Inc. - For authorized use only,OU=Certification Services Division,O=thawte\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 3,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 2,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=GlobalSign,O=GlobalSign,OU=GlobalSign Root CA - R3". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "OU=Security Communication RootCA2,O=SECOM Trust Systems CO.\,LTD.,C=JP". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=VeriSign Universal Root Certification Authority,OU=(c) 2008 VeriSign\, Inc. - For authorized use only,OU=VeriSign Trust Network,O=VeriSign\, Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=KEYNECTIS ROOT CA,OU=ROOT,O=KEYNECTIS,C=FR". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=GeoTrust Primary Certification Authority - G3,OU=(c) 2008 GeoTrust Inc. - For authorized use only,O=GeoTrust Inc.,C=US". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Server> <BEA-002613> <Channel "DefaultSecure" is now listening on 192.168.1.10:7006 for protocols iiops, t3s, CLUSTER-BROADCAST-SECURE, ldaps, https.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 192.168.1.10:7005 for protocols iiop, t3, CLUSTER-BROADCAST, ldap, snmp, http.>
<Nov 3, 2011 12:34:34 PM PET> <Notice> <WebLogicServer> <BEA-000330> <Started WebLogic Managed Server "Server01" for domain "DomainOWL" running in Production Mode>
Exception in thread "[STANDBY] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:217)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Exception in thread "[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:36:27 PM PET> <Error> <HTTP> <BEA-101020> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Servlet failed with Exception
java.lang.OutOfMemoryError: PermGen space
>
Exception in thread "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "Timer-1" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:36:53 PM PET> <Error> <HTTP> <BEA-101020> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Servlet failed with Exception
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:36:24 PM PET> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: PermGen space.
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:36:33 PM PET> <Error> <JMX> <BEA-149501> <An exception occurred while registering the MBean com.bea:Name=idmAdminServerT,Type=Server at property WebService.
java.lang.OutOfMemoryError: PermGen space
>
Exception in thread "[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:37:16 PM PET> <Warning> <Socket> <BEA-000402> <There are: 5 active sockets, but the maximum number of socket reader threads allowed by the configuration is: 4. You may want to alter your configuration.>
Exception in thread "[ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:39:41 PM PET> <Error> <HTTP> <BEA-101216> <Servlet: "CoordinatorPortTypeServlethttp" failed to preload on startup in Web application: "wls-wsat.war".
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:42:32 PM PET> <Error> <HTTP> <BEA-101017> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Root cause of ServletException.
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:43:15 PM PET> <Error> <HTTP> <BEA-101017> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Root cause of ServletException.
java.lang.OutOfMemoryError: PermGen space
>
<Nov 3, 2011 12:43:26 PM PET> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: PermGen space.
java.lang.OutOfMemoryError: PermGen space
>
Exception in thread "[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
Exception in thread "[STANDBY] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 3, 2011 12:45:53 PM PET> <Error> <HTTP> <BEA-101017> <[ServletContext@15375304[app:bea_wls_deployment_internal module:bea_wls_deployment_internal.war path:/bea_wls_deployment_internal spec-version:null]] Root cause of ServletException.
java.lang.OutOfMemoryError: PermGen space
>
Exception in thread "[ACTIVE] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "weblogic.timers.TimerThread" java.lang.OutOfMemoryError: PermGen space
Thanks<Nov 4, 2011 11:42:08 AM PET> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: PermGen space.
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
Truncated. see log file for complete stacktrace
>
<Nov 4, 2011 11:42:34 AM PET> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
at java.lang.Class.getDeclaredMethod(Class.java:1935)
at java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1349)
at java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:52)
Truncated. see log file for complete stacktrace
>
The WebLogic Server encountered a critical failure
Reason: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "main" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 4, 2011 11:44:08 AM PET> <Warning> <Socket> <BEA-000402> <There are: 5 active sockets, but the maximum number of socket reader threads allowed by the configuration is: 4. You may want to alter your configuration.>
Exception in thread "[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
Exception in thread "[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
<Nov 4, 2011 11:44:43 AM PET> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: PermGen space.
java.lang.OutOfMemoryError: PermGen space
I made the change and continue with the same problem, you can tell me which file should make this change, very grateful -
Dbconsole failed to start on one RAC node
Hi
I have 2 RAC nodes (RHEL 4) and 10.2.0.1. On one dbconsole is running and on other I get the following. Earlier dbconsole
on both the nodes used to run perfectly fine. I will appreacite any suggestions to rectify this problem.
Regards
oracle@rac01<18>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> emctl start dbconsole
TZ set to Canada/Newfoundland
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
http://rac01:1158/em/console/aboutApplication
Agent Version : 10.1.0.4.1
OMS Version : Unknown
Protocol Version : 10.1.0.2.0
Agent Home : /u01/app/oracle/product/10.2/db_1/rac01_RACDB1
Agent binaries : /u01/app/oracle/product/10.2/db_1
Agent Process ID : 23329
Parent Process ID : 21132
Agent URL : http://rac01:3938/emd/main
Started at : 2007-07-25 11:37:32
Started by user : oracle
Last Reload : 2007-07-25 11:37:32
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 371
Size of XML files pending upload(MB) : 7.66
Available disk space on upload filesystem : 44.78%
Agent is already started. Will restart the agent
Stopping agent ... stopped.
Starting Oracle Enterprise Manager 10g Database Control ............................................................................................. failed.
Logs are generated in directory /u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log
oracle@rac01<19>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log>
ON OTHER NODE:
oracle@rac02<2>:/u01/app/oracle> emctl start dbconsole
TZ set to Canada/Newfoundland
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
http://rac01:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 10g Database Control .................................... started.
Logs are generated in directory /u01/app/oracle/product/10.2/db_1/rac02_RACDB2/sysman/log
oracle@rac02<3>:/u01/app/oracle>Thanks for your time and reply .
Well, here is what I got, couldn't make out from here.
Regards
oracle@rac01<19>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> ls -lart
total 13500
drwxr----- 7 oracle dba 4096 Jul 14 10:48 ..
-rw-r----- 1 oracle dba 0 Jul 14 10:48 emdctl.log
drwxrwx--- 2 oracle dba 4096 Jul 14 10:54 nmcRACDB11521
-rw-r----- 1 oracle dba 4655792 Jul 24 23:01 emoms.trc
-rw-r----- 1 oracle dba 4655792 Jul 24 23:01 emoms.log
drwxr----- 3 oracle dba 4096 Jul 25 11:35 .
-rw-r----- 1 oracle dba 4096 Jul 25 12:05 emdb.nohup.lr
-rw-r----- 1 oracle dba 1074 Jul 25 12:05 emagent_perl.trc
-rw-r----- 1 oracle dba 1731 Jul 25 12:06 emagent.log
-rw-r----- 1 oracle dba 1080 Jul 25 12:07 emagentfetchlet.trc
-rw-r----- 1 oracle dba 1080 Jul 25 12:07 emagentfetchlet.log
-rw-r----- 1 oracle dba 81089 Jul 25 13:28 emdctl.trc
-rw-r----- 1 oracle dba 3309143 Jul 25 13:28 emdb.nohup
-rw-r----- 1 oracle dba 1044518 Jul 25 13:28 emagent.trc
oracle@rac01<20>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> cat emagent.log
2007-07-14 10:50:44 Thread-3086936288 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-14 10:51:16 Thread-3086936288 EMAgent started successfully (00702)
2007-07-14 14:38:21 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-14 14:39:00 Thread-3086935744 EMAgent started successfully (00702)
2007-07-24 07:05:06 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-24 07:07:11 Thread-3086935744 target {+ASM1_rac01, osm_instance} is broken: cannot compute dynamic properties in time. (00155)
2007-07-24 07:07:14 Thread-3086935744 EMAgent started successfully (00702)
2007-07-24 12:06:27 Thread-3086935744 EMAgent normal shutdown (00703)
2007-07-24 12:08:26 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-24 12:08:51 Thread-3086935744 EMAgent started successfully (00702)
2007-07-25 11:35:35 Thread-3086935744 EMAgent normal shutdown (00703)
2007-07-25 11:37:32 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-25 11:39:29 Thread-3086935744 target {+ASM1_rac01, osm_instance} is broken: cannot compute dynamic properties in time. (00155)
2007-07-25 11:39:30 Thread-3086935744 EMAgent started successfully (00702)
2007-07-25 12:03:36 Thread-3086935744 EMAgent normal shutdown (00703)
2007-07-25 12:05:15 Thread-3086935744 Starting Agent 10.1.0.4.1 from /u01/app/oracle/product/10.2/db_1 (00701)
2007-07-25 12:06:23 Thread-3086935744 target {+ASM1_rac01, osm_instance} is broken: cannot compute dynamic properties in time. (00155)
2007-07-25 12:06:24 Thread-3086935744 EMAgent started successfully (00702)
oracle@rac01<21>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> cat emagentfetchlet.log
2007-07-14 11:01:44,208 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-14 14:40:29,096 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-24 07:10:44,123 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-24 12:12:48,187 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-25 11:41:25,628 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-25 12:07:30,335 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
oracle@rac01<22>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log>
oracle@rac01<22>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> tail -40 emagentfetchlet.trc
2007-07-14 11:01:44,208 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-14 14:40:29,096 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-24 07:10:44,123 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-24 12:12:48,187 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-25 11:41:25,628 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
2007-07-25 12:07:30,335 [main] WARN track.OracleInventory collectInventory.439 - ECM: The inventory location file for the special Windows NT case does not exist or is unreadable.
oracle@rac01<25>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> tail -10 emdctl.trc
2007-07-25 13:01:02 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:04:41 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:07:12 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:10:50 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:14:32 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:18:09 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:20:40 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:24:27 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:28:06 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:31:43 Thread-3086935744 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
oracle@rac01<28>:/u01/app/oracle/product/10.2/db_1/rac01_RACDB1/sysman/log> tail -10 emagent.trc
2007-07-25 13:31:44 Thread-43162528 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:31:44 Thread-43162528 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
2007-07-25 13:32:14 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:32:14 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
2007-07-25 13:32:14 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:32:14 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
2007-07-25 13:32:44 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:32:44 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
2007-07-25 13:32:44 Thread-74791840 WARN http: snmehl_connect: connect failed to (rac01:1158): Connection refused (error = 111)
2007-07-25 13:32:44 Thread-74791840 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://rac01:1158/em/upload/: retStatus=-32
Message was edited by:
Singh -
Ons gsd of one node offline,RAC
crs_start ora.whdb02.ons
Attempting to start `ora.whdb02.ons` on member `whdb02`
Start of `ora.whdb02.ons` on member `whdb02` failed.
whdb01 : CRS-1019: Resource ora.whdb02.ons (application) cannot run on whdb01
CRS-0215: Could not start resource 'ora.whdb02.ons'
oracle@whdb02:/oracle/app/oracle/product/10.2.0/db1/network/admin>$CRS_HOME/bin/onsctl start
ksh: /bin/onsctl: not found.
oracle@whdb02:/oracle/app/oracle/product/10.2.0/db1/network/admin>$ORA_CRS_HOME/bin/onsctl start
clsrons_init failed, stat = 504, ocrerr = 32
clsrons_init failed, stat = 504, ocrerr = 32
onsctl: ons failed to start
oracle@whdb02:/oracle/app/oracle/product/10.2.0/db1/network/admin>
oracle@whdb02:/oracle/app/oracle/product/10.2.0/db1/network/admin>
oracle@whdb02:/oracle/app/oracle/product/10.2.0/db1/network/admin>A_CRS_HOME/bin/onsctl STOP
ksh: A_CRS_HOME/bin/onsctl: not found.
oracle@whdb02:/oracle/app/oracle/product/10.2.0/db1/network/admin>$ORA_CRS_HOME/bin/onsctl stop
onsctl: shutting down ons daemon ...
clsrons_init failed, stat = 504, ocrerr = 32
onsctl: shutdown of ons failed!
oracle@whdb02:/oracle/app/oracle/product/10.2.0/db1/network/admin>$ORA_CRS_HOME/bin/onsctl start
clsrons_init failed, stat = 504, ocrerr = 32
clsrons_init failed, stat = 504, ocrerr = 32
onsctl: ons failed to startoracle@whdb02:/oracle/app/oracle/product/10.2.0/db1/network/adgsd log
2010-11-10 13:16:45.515: [ RACG][1] [274628][1][ora.whdb02.gsd]: clsrcexecut: cmd = /oracle/app/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /oracle/app/oracle/product/10.2.0/crs/bin/gsdctl stat
2010-11-10 13:16:45.515: [ RACG][1] [274628][1][ora.whdb02.gsd]: clsrcexecut: rc = 1, time = 2.703s
2010-11-10 13:16:45.515: [ RACG][1] [274628][1][ora.whdb02.gsd]: end for resource = ora.whdb02.gsd, action = start, status = 1, time = 54.109s
2010-11-10 13:16:45.989: [ RACG][1] [274634][1][ora.whdb02.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2010-11-10 13:16:48.694: [ RACG][1] [274634][1][ora.whdb02.gsd]: GSD is not running on the local node
2010-11-10 14:09:27.008: [ RACG][1] [573578][1][ora.whdb02.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2010-11-10 14:10:18.016: [ RACG][1] [573578][1][ora.whdb02.gsd]: Failed to start GSD on local node
2010-11-10 14:10:18.016: [ RACG][1] [573578][1][ora.whdb02.gsd]: clsrcexecut: cmd = /oracle/app/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /oracle/app/oracle/product/10.2.0/crs/bin/gsdctl start
2010-11-10 14:10:18.016: [ RACG][1] [573578][1][ora.whdb02.gsd]: clsrcexecut: rc = 1, time = 51.008s
2010-11-10 14:10:20.720: [ RACG][1] [573578][1][ora.whdb02.gsd]: GSD is not running on the local node
2010-11-10 14:10:20.720: [ RACG][1] [573578][1][ora.whdb02.gsd]: clsrcexecut: cmd = /oracle/app/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /oracle/app/oracle/product/10.2.0/crs/bin/gsdctl stat
2010-11-10 14:10:20.720: [ RACG][1] [573578][1][ora.whdb02.gsd]: clsrcexecut: rc = 1, time = 2.704s
2010-11-10 14:10:20.720: [ RACG][1] [573578][1][ora.whdb02.gsd]: end for resource = ora.whdb02.gsd, action = start, status = 1, time = 54.109s
2010-11-10 14:10:21.195: [ RACG][1] [573584][1][ora.whdb02.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2010-11-10 14:10:23.900: [ RACG][1] [573584][1][ora.whdb02.gsd]: GSD is not running on the local node
2010-11-10 15:33:02.434: [ RACG][1] [618716][1][ora.whdb02.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2010-11-10 15:33:56.467: [ RACG][1] [618716][1][ora.whdb02.gsd]: Failed to start GSD on local node
2010-11-10 15:33:56.467: [ RACG][1] [618716][1][ora.whdb02.gsd]: clsrcexecut: cmd = /oracle/app/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /oracle/app/oracle/product/10.2.0/crs/bin/gsdctl start
2010-11-10 15:33:56.467: [ RACG][1] [618716][1][ora.whdb02.gsd]: clsrcexecut: rc = 1, time = 54.024s
2010-11-10 15:33:59.171: [ RACG][1] [618716][1][ora.whdb02.gsd]: GSD is not running on the local node
2010-11-10 15:33:59.171: [ RACG][1] [618716][1][ora.whdb02.gsd]: clsrcexecut: cmd = /oracle/app/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /oracle/app/oracle/product/10.2.0/crs/bin/gsdctl stat
2010-11-10 15:33:59.171: [ RACG][1] [618716][1][ora.whdb02.gsd]: clsrcexecut: rc = 1, time = 2.704s
2010-11-10 15:33:59.171: [ RACG][1] [618716][1][ora.whdb02.gsd]: end for resource = ora.whdb02.gsd, action = start, status = 1, time = 57.130s
2010-11-10 15:33:59.646: [ RACG][1] [618722][1][ora.whdb02.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2010-11-10 15:34:02.351: [ RACG][1] [618722][1][ora.whdb02.gsd]: GSD is not running on the local node
2010-11-10 15:40:29.176: [ RACG][1] [503954][1][ora.whdb02.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2010-11-10 15:41:20.184: [ RACG][1] [503954][1][ora.whdb02.gsd]: Failed to start GSD on local node
2010-11-10 15:41:20.184: [ RACG][1] [503954][1][ora.whdb02.gsd]: clsrcexecut: cmd = /oracle/app/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /oracle/app/oracle/product/10.2.0/crs/bin/gsdctl start
2010-11-10 15:41:20.184: [ RACG][1] [503954][1][ora.whdb02.gsd]: clsrcexecut: rc = 1, time = 51.007s
2010-11-10 15:41:22.888: [ RACG][1] [503954][1][ora.whdb02.gsd]: GSD is not running on the local node
2010-11-10 15:41:22.889: [ RACG][1] [503954][1][ora.whdb02.gsd]: clsrcexecut: cmd = /oracle/app/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /oracle/app/oracle/product/10.2.0/crs/bin/gsdctl stat
2010-11-10 15:41:22.889: [ RACG][1] [503954][1][ora.whdb02.gsd]: clsrcexecut: rc = 1, time = 2.703s
2010-11-10 15:41:22.889: [ RACG][1] [503954][1][ora.whdb02.gsd]: end for resource = ora.whdb02.gsd, action = start, status = 1, time = 54.106s
2010-11-10 15:41:23.373: [ RACG][1] [290992][1][ora.whdb02.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2010-11-10 15:41:26.078: [ RACG][1] [290992][1][ora.whdb02.gsd]: GSD is not running on the local node
2010-11-10 15:50:06.328: [ RACG][1] [442492][1][ora.whdb02.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2010-11-10 15:50:57.336: [ RACG][1] [442492][1][ora.whdb02.gsd]: Failed to start GSD on local node
2010-11-10 15:50:57.336: [ RACG][1] [442492][1][ora.whdb02.gsd]: clsrcexecut: cmd = /oracle/app/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /oracle/app/oracle/product/10.2.0/crs/bin/gsdctl start
2010-11-10 15:50:57.336: [ RACG][1] [442492][1][ora.whdb02.gsd]: clsrcexecut: rc = 1, time = 51.008s
2010-11-10 15:51:00.043: [ RACG][1] [442492][1][ora.whdb02.gsd]: GSD is not running on the local node
2010-11-10 15:51:00.043: [ RACG][1] [442492][1][ora.whdb02.gsd]: clsrcexecut: cmd = /oracle/app/oracle/product/10.2.0/crs/bin/racgeut -e USRORA_DEBUG=0 540 /oracle/app/oracle/product/10.2.0/crs/bin/gsdctl stat
2010-11-10 15:51:00.043: [ RACG][1] [442492][1][ora.whdb02.gsd]: clsrcexecut: rc = 1, time = 2.706s
2010-11-10 15:51:00.043: [ RACG][1] [442492][1][ora.whdb02.gsd]: end for resource = ora.whdb02.gsd, action = start, status = 1, time = 54.114s
2010-11-10 15:51:01.361: [ RACG][1] [618710][1][ora.whdb02.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2010-11-10 15:51:04.066: [ RACG][1] [618710][1][ora.whdb02.gsd]: GSD is not running on the local node -
Getting Error in starting VIP in 3 NODE RAC Cluster in VMWARE
hi
please can some one help me to have solution for why VIPCA is failing to start VIP on RAC Node 3 it gives the ERROR: CRS-1006; CRS-0215 no more members. Network Configuration is like:
/etc/hosts
127.0.0.1 localhost.localdomain localhost
#Public IP
192.168.2.131 rac1.sun.com rac1
192.168.2.132 rac2.sun.com rac2
192.168.2.133 rac3.sun.com rac3
#Private IP
10.10.10.31 rac1-priv rac1-priv
10.10.10.32 rac2-priv rac2-priv
10.10.10.33 rac3-priv rac3-priv
#Virtual IP
192.168.2.131 rac1-vip.sun.com rac1-vip
192.168.2.132 rac2-vip.sun.com rac2-vip
192.168.2.133 rac3-vip.sun.com rac3-vip
/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=rac1.sun.com
GATEWAY=192.168.2.1
Thanks in Advanceyou should have to user some other new ips for VIP.
PLEASE CHANGE THE VIP IP's and try again.
192.168.2.131 rac1-vip.sun.com rac1-vip
192.168.2.132 rac2-vip.sun.com rac2-vip
192.168.2.133 rac3-vip.sun.com rac3-vipchange the ips to some other ip not used by any machines.
sample /etc/hosts file
127.0.0.1 localhost.localdomain localhost
# Public
10.1.10.201 rac1.localdomain rac1
10.1.10.202 rac2.localdomain rac2
#Private
10.1.9.201 rac1-priv.localdomain rac1-priv
10.1.9.202 rac2-priv.localdomain rac2-priv
#Virtual
*10.1.10.203 rac1-vip.localdomain rac1-vip*
*10.1.10.204 rac2-vip.localdomain rac2-vip* -
Hi DBA's.
Im, running
Finalizing Installation 96% the following Warning:
[Thread-288] [ 2010-01-21 14:28:57.456 ARST ] [CRSNative.internalStartResource:352] Failed to start resource: Name: ora.racdb.db, node: null, filter: null, msg CRS-2674:
Start of 'ora.racdb.db' on 'linux2' failed
CRS-2678: 'ora.racdb.db' on 'linux2' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-5807: Agent failed to process the message
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
[Thread-288] [ 2010-01-21 14:28:57.457 ARST ] [PostDBCreationStep.executeImpl:828] Exception while Starting with HA Database Resource PRCR-1079 : Failed to start resourc
e ora.racdb.db
CRS-2674: Start of 'ora.racdb.db' on 'linux2' failed
CRS-2678: 'ora.racdb.db' on 'linux2' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-5807: Agent failed to process the message
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
oracle$ dbcaHi...
Now is Ok.
I did:
srvctl start instance -d racdb -i racdb2
[oracle@linux1 oracle]$ su - grid -c "crsctl status resource -w \"TYPE co 'ora'\" -t"
Password:
NAME TARGET STATE SERVER STATE_DETAILS
Local Resources
ora.CRS.dg
ONLINE ONLINE linux1
ONLINE ONLINE linux2
ora.FRA.dg
ONLINE ONLINE linux1
ONLINE ONLINE linux2
ora.LISTENER.lsnr
ONLINE ONLINE linux1
ONLINE ONLINE linux2
ora.RACDB_DATA.dg
ONLINE ONLINE linux1
ONLINE ONLINE linux2
ora.asm
ONLINE ONLINE linux1 Started
ONLINE ONLINE linux2 Started
ora.eons
ONLINE ONLINE linux1
ONLINE ONLINE linux2
ora.gsd
OFFLINE OFFLINE linux1
OFFLINE OFFLINE linux2
ora.net1.network
ONLINE ONLINE linux1
ONLINE ONLINE linux2
ora.ons
ONLINE ONLINE linux1
ONLINE ONLINE linux2
Cluster Resources
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE linux1
ora.linux1.vip
1 ONLINE ONLINE linux1
ora.linux2.vip
1 ONLINE ONLINE linux2
ora.oc4j
1 OFFLINE OFFLINE
ora.racdb.db
1 ONLINE ONLINE linux1 Open
2 ONLINE ONLINE linux2 Open
ora.scan1.vip
1 ONLINE ONLINE linux1
Thanks. -
Rs-ora:resource group failed to start on chosen node; it may end up failing
I have configured two node failover cluster environment using netra a/d 1000 storage. When I try to deploy oracle server application it throws the following error
rs-ora: resource group failed to start on chosen node; it may end up failing over to other node(s)
I created metaset and gave one raw did disk to that metaset.
I created logical hostname resource, ha-storage plus resource. Later I brought the resource group to online using following command
#clrg online emM rg-ora
Later I created oracle cluster resource using following command.
#clrs create -g rg-ora -t SUNW.oracle_server -p ORACLE_HOME=/global/oracle/product/10.2.0/db_1 -p ORACLE_SID=infra -p Alert_log_file=/global/oracle/product/10.2.0/db_1/admin/infra/bdump/alert_infra.log -p Connect_string=sysdba/dbadmin1@infra -p Resource_dependencies=rs-ora-has rs-ora
node1 - Validation failed. ORACLE_HOME /global/oracle/product/10.2.0/db_1 does not exist
node1 - ALERT_LOG_FILE /global/oracle/product/10.2.0/db_1/admin/infra/bdump/alert_infra.log doesn't exist
node1 - PARAMETER_FILE: /global/oracle/product/10.2.0/db_1/dbs/initinfra.ora nor server PARAMETER_FILE: /global/oracle/product/10.2.0/db_1/dbs/spfileinfra.ora exists
node1 - This resource depends on a HAStoragePlus resouce that is not online on this node. Ignoring validation errors.
rs-ora: resource group failed to start on chosen node; it may end up failing over to other node(s)
The status of oracle resource shows as follows.
Resource Name Node Name State Status Message
rs-ora node1 Start failed Faulted
I used solaris 10 update 6 patch level is Generic_137137-09, Oracle version 10.2.0, Sun clusters 3.2 update1. Following are the vfstab and /var/adm/messages of both nodes.
Node1#grep ora /etc/vfstab
/dev/md/oradg/dsk/d300 /dev/md/oradg/rdsk/d300 /global/oracle ufs 5 no logging
Node2#grep ora /etc/vfstab
/dev/md/oradg/dsk/d300 /dev/md/oradg/rdsk/d300 /global/oracle ufs 5 no logging
Node1#more /var/adm/messages
Oct 17 05:19:17 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_prenet_start> for resource <ha-
host-1>, resource group <rg-ora>, node <node1>, timeout <300> seconds
Oct 17 05:19:17 node1 Cluster.RGM.rgmd: [ID 751138 daemon.notice] 47 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/
lib/rgm/rt/hafoip/hafoip_prenet_start>:tag=<rg-ora.ha-host-1.10>: Calling security_clnt_connect(..., host=<node1>, sec_typ
e {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 17 05:19:17 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_prenet_start> completed successfully for
resource <ha-host-1>, resource group <rg-ora>, node <node1>, time used: 0% of timeout <300 seconds>
Oct 17 05:19:17 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_prenet_start> for resour
ce <rs-ora-has>, resource group <rg-ora>, node <node1>, timeout <1800> seconds
Oct 17 05:19:17 node1 Cluster.RGM.rgmd: [ID 751138 daemon.notice] 47 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/
lib/rgm/rt/hastorageplus/hastorageplus_prenet_start>:tag=<rg-ora.rs-ora-has.10>: Calling security_clnt_connect(..., host=<tes
tlab5>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 17 05:19:18 node1 Cluster.RGM.rgmd: [ID 375444 daemon.notice] 8 fe_rpc_command: cmd_type(enum):<2>:cmd=<null>:tag=<rg-
ora.rs-ora-has.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<0>, ...)
Oct 17 05:19:18 node1 Cluster.RGM.rgmd: [ID 316625 daemon.notice] Timeout monitoring on method tag <rg-ora.rs-ora-has.10>
has been suspended.
Oct 17 05:19:20 node1 Cluster.Framework: [ID 801593 daemon.notice] stdout: becoming primary for oradg
Oct 17 05:19:21 node1 Cluster.RGM.rgmd: [ID 375444 daemon.notice] 8 fe_rpc_command: cmd_type(enum):<3>:cmd=<null>:tag=<rg-
ora.rs-ora-has.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<0>, ...)
Oct 17 05:19:21 node1 Cluster.RGM.rgmd: [ID 316625 daemon.notice] Timeout monitoring on method tag <rg-ora.rs-ora-has.10>
has been resumed.
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_prenet_start> completed successful
ly for resource <rs-ora-has>, resource group <rg-ora>, node <node1>, time used: 0% of timeout <1800 seconds>
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_start> for resource <ha-host-1>
, resource group <rg-ora>, node <node1>, timeout <500> seconds
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 751138 daemon.notice] 47 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/
lib/rgm/rt/hafoip/hafoip_start>:tag=<rg-ora.ha-host-1.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEA
K, 1:STRONG, 2:DES} =<1>, ...)
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_start> completed successfully for resourc
e <ha-host-1>, resource group <rg-ora>, node <node1>, time used: 0% of timeout <500 seconds>
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_start> for resource <ha
-host-1>, resource group <rg-ora>, node <node1>, timeout <300> seconds
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_start> for resource <rs-
ora-has>, resource group <rg-ora>, node <node1>, timeout <90> seconds
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/
lib/rgm/rt/hafoip/hafoip_monitor_start>:tag=<rg-ora.ha-host-1.7>: Calling security_clnt_connect(..., host=<node1>, sec_typ
e {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 751138 daemon.notice] 47 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/
lib/rgm/rt/hastorageplus/hastorageplus_start>:tag=<rg-ora.rs-ora-has.0>: Calling security_clnt_connect(..., host=<node1>,
sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_start> completed successfully for
resource <ha-host-1>, resource group <rg-ora>, node <node1>, time used: 0% of timeout <300 seconds>
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_start> completed successfully for
resource <rs-ora-has>, resource group <rg-ora>, node <node1>, time used: 0% of timeout <90 seconds>
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_start> for resou
rce <rs-ora-has>, resource group <rg-ora>, node <node1>, timeout <90> seconds
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 751138 daemon.notice] 47 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/
lib/rgm/rt/hastorageplus/hastorageplus_monitor_start>:tag=<rg-ora.rs-ora-has.7>: Calling security_clnt_connect(..., host=<tes
tlab5>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 17 05:19:25 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_start> completed successfu
lly for resource <rs-ora-has>, resource group <rg-ora>, node <node1>, time used: 0% of timeout <90 seconds>
Oct 17 05:19:38 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_validate> for resour
ce <rs-ora>, resource group <rg-ora>, node <node1>, timeout <120> secondsOct 17 05:19:38 node1 Cluster.RGM.rgmd: [ID 375444 daemon.notice] 8 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/
oracle_server/bin/oracle_server_validate>:tag=<rg-ora.rs-ora.2>: Calling security_clnt_connect(..., host=<node1>, sec_type
{0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 17 05:19:38 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_validate> completed successful
ly for resource <rs-ora>, resource group <rg-ora>, node <node1>, time used: 0% of timeout <120 seconds>
Oct 17 05:19:38 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_init> for resource <
rs-ora>, resource group <rg-ora>, node <node1>, timeout <30> seconds
Oct 17 05:19:38 node1 Cluster.RGM.rgmd: [ID 751138 daemon.notice] 47 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor
/oracle_server/bin/oracle_server_init>:tag=<rg-ora.rs-ora.4>: Calling security_clnt_connect(..., host=<node1>, sec_type {0
:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 17 05:19:38 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_init> completed successfully f
or resource <rs-ora>, resource group <rg-ora>, node <node1>, time used: 0% of timeout <30 seconds>
Oct 17 05:19:38 node1 Cluster.CCR: [ID 973933 daemon.notice] resource rs-ora added.
Oct 17 05:19:39 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_start> for resource
<rs-ora>, resource group <rg-ora>, node <node1>, timeout <600> seconds
Oct 17 05:19:39 node1 Cluster.RGM.rgmd: [ID 751138 daemon.notice] 47 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor
/oracle_server/bin/oracle_server_start>:tag=<rg-ora.rs-ora.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {
0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 17 05:19:48 node1 SC[SUNWscor.oracle_server.start]:rg-ora:rs-ora: [ID 876834 daemon.error] Could not start server
Oct 17 05:19:48 node1 Cluster.RGM.rgmd: [ID 938318 daemon.error] Method <bin/oracle_server_start> failed on resource <rs-o
ra> in resource group <rg-ora> [exit code <1>, time used: 1% of timeout <600 seconds>]
Node2# more /var/adm/messages
Oct 14 20:20:04 node2 Cluster.RGM.rgmd: [ID 529407 daemon.notice] resource group rg-ora state on node node2 change to RG_PENDIN
G_OFFLINE
Oct 14 20:20:04 node2 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource rs-ora-has state on node node2 change to R_MON_STOPP
ING
Oct 14 20:20:04 node2 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource ha-host-1 state on node node2 change to R_MON_STOPPI
NG
Oct 14 20:20:04 node2 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_stop> for resource <ha-host
-1>, resource group <rg-ora>, node <node2>, timeout <300> seconds
Oct 14 20:20:04 node2 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_stop> for resource <
rs-ora-has>, resource group <rg-ora>, node <node2>, timeout <90> seconds
Oct 14 20:20:04 node2 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/
rgm/rt/hafoip/hafoip_monitor_stop>:tag=<rg-ora.ha-host-1.8>: Calling security_clnt_connect(..., host=<node2>, sec_type {0:WEAK
, 1:STRONG, 2:DES} =<1>, ...)
Oct 14 20:20:04 node2 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/
rgm/rt/hastorageplus/hastorageplus_monitor_stop>:tag=<rg-ora.rs-ora-has.8>: Calling security_clnt_connect(..., host=<node2>, s
ec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_stop> completed successfully f
or resource <rs-ora-has>, resource group <rg-ora>, node <node2>, time used: 0% of timeout <90 seconds>
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource rs-ora-has state on node node2 change to R_ONLINE_UN
MON
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource rs-ora-has state on node node2 change to R_STOPPING
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 784560 daemon.notice] resource rs-ora-has status on node node2 change to R_FM_UNKNO
WN
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 922363 daemon.notice] resource rs-ora-has status msg on node node2 change to <Stopp
ing>
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_stop> for resource <rs-ora-h
as>, resource group <rg-ora>, node <node2>, timeout <1800> seconds
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/
rgm/rt/hastorageplus/hastorageplus_stop>:tag=<rg-ora.rs-ora-has.1>: Calling security_clnt_connect(..., host=<node2>, sec_type
{0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_stop> completed successfully for reso
urce <ha-host-1>, resource group <rg-ora>, node <node2>, time used: 0% of timeout <300 seconds>
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource ha-host-1 state on node node2 change to R_ONLINE_UNM
ON
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_stop> completed successfully for resou
rce <rs-ora-has>, resource group <rg-ora>, node <node2>, time used: 0% of timeout <1800 seconds>
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource rs-ora-has state on node node2 change to R_STOPPED
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource ha-host-1 state on node node2 change to R_STOPPING
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_stop> for resource <ha-host-1>, res
ource group <rg-ora>, node <node2>, timeout <300> seconds
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 784560 daemon.notice] resource ha-host-1 status on node node2 change to R_FM_UNKNOW
N
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 922363 daemon.notice] resource ha-host-1 status msg on node node2 change to <Stoppi
ng>
Oct 14 20:20:05 node2 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/
rgm/rt/hafoip/hafoip_stop>:tag=<rg-ora.ha-host-1.1>: Calling security_clnt_connect(..., host=<node2>, sec_type {0:WEAK, 1:STRO
NG, 2:DES} =<1>, ...)
Oct 14 20:20:06 node2 ip: [ID 678092 kern.notice] TCP_IOC_ABORT_CONN: local = 192.168.032.244:0, remote = 000.000.000.000:0, s
tart = -2, end = 6
Oct 14 20:20:06 node2 ip: [ID 302654 kern.notice] TCP_IOC_ABORT_CONN: aborted 0 connection
Oct 14 20:20:06 node2 Cluster.RGM.rgmd: [ID 784560 daemon.notice] resource ha-host-1 status on node node2 change to R_FM_OFFLIN
E
Oct 14 20:20:06 node2 Cluster.RGM.rgmd: [ID 922363 daemon.notice] resource ha-host-1 status msg on node node2 change to <Logica
lHostname offline.>
Oct 14 20:20:06 node2 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_stop> completed successfully for resource <ha
-host-1>, resource group <rg-ora>, node <node2>, time used: 0% of timeout <300 seconds>
Oct 14 20:20:06 node2 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource ha-host-1 state on node node2 change to R_OFFLINE
Oct 14 20:20:06 node2 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource rs-ora-has state on node node2 change to R_POSTNET_S
TOPPING -
Server state failed when starting through node manager
Hi,
I'm getting the below error while starting the managed server through node manager
<Sep 14, 2010 3:49:34 PM> <Warning> <Exception while starting server 'ms01': java.io.IOException: Server failed to start up. See server
output log for more details.>
java.io.IOException: Server failed to start up. See server output log for more details.
at weblogic.nodemanager.server.ServerManager.start(ServerManager.java:303)
at weblogic.nodemanager.server.Handler.handleStart(Handler.java:542)
at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:119)
at weblogic.nodemanager.server.Handler.run(Handler.java:66)
at java.lang.Thread.run(Thread.java:619)
I have checked the server logs and it shows the below error code
./admin-stdout.log:<Sep 15, 2010 4:15:56 PM PDT> <Error> <NodeManager> <BEA-300048> <Unable to start the server ms01 : Exception while starting server 'ms01': java.io.IOException: Server failed to start up. See server output log for more details.>
I have also checked WebLogic documentation for BEA-300048. It says "Cause: The most likely cause is a configuration error in the Shell/SSH/RSH NodeManager configuration".
but I'm using Java based node manager
All this is coming up when I enable StartScriptEnabled to true in nodemanager.properties.
What could be reason for failure?Hi.
If you are not using the latest service pack (sp2) please do so. If you are and are still seeing this
problem please open a case with support.
Thanks
Michael
Tim Dawson wrote:
I get this message whenever I try to use the console to start a managed server
on a remote system.
<Jan 12, 2002 8:18:26 AM PST> <Emergency> <WebLogicServer> <Unable to create a
server socket for: weblogic2.dev.wamnet.com/172.17.27.84, port: 7001. java.net.BindException:
Cannot assign requested address: JVM_Bind Perhaps the address weblogic2.dev.wamnet.com/172.17.27.84
is incorrect or another process is using port 7001.>
I've checked and checked and the only thing running on that box is the node manager!
The console will start a managed server on the local system (i.e. the system
that the admin server is running on) without any complaints.
Any ideas? Thanks,
Tim--
Michael Young
Developer Relations Engineer
BEA Support -
Pre-installation cluvfy check fails on local nodes
I am setting up a RAC cluster in Oracle 11.2.0.3 on two Windows 2008R2 nodes.
I have completed all the pre-installation tasks (I hope) and would like to verify this by running:
cluvfy comp sys -p database -n all
using the standalone download of cluvfy - I haven't installed any Oracle software as yet.
This completes successfully on the remote node but fails on the local node (whichever one I run it from).
There is a lot of output but these seem to be the key problems:
[1852@PRDAT217] [Worker 3] [ 2013-10-01 14:44:45.190 BST ] [WindowsSystem.deleteService:876] _WS_ deleteService2: node PRDAT217 Service OracleRemExecService result: 0|Access is denied.
[1852@PRDAT217] [Worker 3] [ 2013-10-01 14:44:45.190 BST ] [NativeResult.<init>:91] NativeResult: The String obtained is0|Access is denied.
[1852@PRDAT217] [Worker 3] [ 2013-10-01 14:44:45.190 BST ] [NativeResult.<init>:99] The status string is: 0
[1852@PRDAT217] [Worker 3] [ 2013-10-01 14:44:45.190 BST ] [NativeResult.<init>:112] The result string is: Access is denied. 1
[1852@PRDAT217] [Worker 3] [ 2013-10-01 14:44:45.190 BST ] [WindowsSystem.startRemoteExecServer:2156] _WS_ Failed to delete Service OracleRemExecService on PRDAT217
[1852@PRDAT217] [Worker 3] [ 2013-10-01 14:44:45.206 BST ] [RemoteExecCommand.setupRemoteExecService:748] Start remoteExecServer failed with Execptionoracle.ops.mgmt.nativesystem.NativeException: PRKN-1016 : Failed to create service "OracleRemExecService" on node "PRDAT217", Error: "Access is denied.".
[1852@PRDAT217] [Worker 3] [ 2013-10-01 14:44:45.206 BST ] [RemoteExecCommand.execute:421] NativeException occured while setting up RemoteExecService. err msg:PRKN-1016 : Failed to create service "OracleRemExecService" on node "PRDAT217", Error: "Access is denied.
However, there is a folder in the temporary destination call oraremservice and there are DLL files and the RemoteExecService.exe in there.
In the temp directory there are a number of log files called e.g. sprvmcli_9885533.log (the number is generated uniquely each time),
When running against the local node, these contain the following:
10/01/13 14:44:42 Trying to open a named pipe
10/01/13 14:44:42 About to open pipe
10/01/13 14:44:42 calling create file
10/01/13 14:44:42 Error opening the pipe 5
10/01/13 14:44:42 sprvcli: retval = 7
When the cluvfy runs successfully against the remote node, this has much more information in, and the errors do not occur.
Am I using the cluvfy utility incorrectly for this stage of the operation, or is there something wrong with my setup?first Confirm the connectivity between all of the nodes:
Cluvfy comp nodecon –n all –verbose
Thanks,
gssdba.wordpress.com -
Failed to start ASM instance "+ASM1" on node "rac1"
I have a problem, because when I start RAC and write command crs_stat -t
column State have 2 wrong parameter..
Name Type Target State Host
ora.....CRM.cs application ONLINE ONLINE rac2
ora....db1.srv application ONLINE ONLINE rac2
ora.devdb.db application ONLINE ONLINE rac2
ora....b1.inst application ONLINE OFFLINE
ora....b2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE UNKNOWN rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
When I try
srvctl start asm -n rac1 then is wrong:
PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac1", [PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac1", [CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.rac1.ASM1.asm' has placement error.]]
[PRKS-1009 : Failed to start ASM instance "+ASM1" on node "rac1", [CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.rac1.ASM1.asm' has placement error.]]
and when I try start instance manualy then
PRKP-1001 : Error starting instance devdb1 on node rac1
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.devdb.devdb1.inst' has placement error.
:( Where is my problem??hi, i have exactly the same error
but your suggestions of remove an recreate the asm resource not working
./srvctl remove asm -n dbs2 -i +ASM2 -f
PRKS-1023 : Failed to remove CRS resource for ASM instance "+ASM2" on node "dbs2", [CRS-0214: Could not unregister resource 'ora.dbs2.ASM2.asm'.]
./srvctl start asm -n dbs2
PRKS-1009 : Failed to start ASM instance "+ASM2" on node "dbs2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "dbs2", [CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.dbs2.ASM2.asm' has placement error.]]
[PRKS-1009 : Failed to start ASM instance "+ASM2" on node "dbs2", [CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.dbs2.ASM2.asm' has placement error.]]
how do i proceed?
iam using solaris 10 with t2000 and t5210 server and oracle 10.2.0.4 -
ONS failed to start on second node
Hi,
I have a problem with ons on 10g rac running on linux 5.3
on node 1 it is running without problem but on second node i got this error
2009-04-08 16:30:41.318: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the
2009-04-08 16:30:41.319: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
2009-04-08 16:30:41.319: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = rac1, port = 6200}
Adding remote host rac1:6200
onscfg[1]
{node = rac2, port = 6
2009-04-08 16:30:41.319: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: 200}
Adding remote host rac2:6200
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission d
2009-04-08 16:30:41.319: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: enied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server loca
2009-04-08 16:30:41.319: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: l port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = rac1, port = 6200}
Adding remote host rac1:6200
o
2009-04-08 16:30:41.319: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: nscfg[1]
{node = rac2, port = 6200}
Adding remote host rac2:6200
onsctl: ons failed to start
2009-04-08 16:30:41.319: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/app/crs
2009-04-08 16:30:41.319: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: cmd = /u01/app/crs/bin/racgeut -e USRORA_DEBUG=0 540 /u01/app/crs/bin/onsctl start
2009-04-08 16:30:41.320: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: rc = 1, time = 2.580s
2009-04-08 16:30:42.148: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the
2009-04-08 16:30:42.150: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
2009-04-08 16:30:42.150: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: RCV: Permission denied
Communication error with the OPMN server local port.
Check the OPMN log files
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = rac1, port = 6200}
Adding remote host rac1:6200
onscfg[1]
{node = rac2, port = 6
2009-04-08 16:30:42.150: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: 200}
Adding remote host rac2:6200
ons is not running ...
2009-04-08 16:30:42.151: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/u01/app/crs
2009-04-08 16:30:42.151: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: cmd = /u01/app/crs/bin/racgeut -e USRORA_DEBUG=0 540 /u01/app/crs/bin/onsctl ping
2009-04-08 16:30:42.151: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: clsrcexecut: rc = 1, time = 0.840s
2009-04-08 16:30:42.153: [ RACG][3065611968] [16553][3065611968][ora.rac2.ons]: end for resource = ora.rac2.ons, action = start, status = 1, time = 3.620s
2009-04-08 16:30:44.376: [ RACG][3066242752] [17061][3066242752][ora.rac2.ons]: onsctl: shutting down ons daemon ...
Number of onsconfiguration retrieved, numcfg = 2
onscfg[0]
{node = rac1, port = 6200}
Adding remote host rac1:6200
onscfg[1]
{node = rac2, port = 6200}
Adding remote host rac2:6200
Any idea how to fix this?
Thankscheck the output for crs_getperm for the resource from both nodes. If you could, post them here.
Regards,
Ganesh -
Today I installed Adobe Director 11.5. I received a couple of errors in the FLEXnet Licensing Service. It looks It could not start. I tried to download the Licence fix what I found on Google but that doesn't work, so I checked the 'Service' at service (windows).
It's status is empty. When I double click on the FLEXnet Licensing Service, clicked on start at the opened window and an error is coming:
Windows could not start the FLEXnet Licensing Service service on Local Computer
Error 1068: The dependency service or group failed to start.
I don't understand what it says with this message, only that it is impossible to start. However, that is not what I am trying of course, I want to start the FLEXnet Licensing Service, service.
Re-installing of the software, doesn't make any change. How can I start it so Director is also working?Hi Don1233,
Please consider seek help at the software vendor side.
For the services didn't start, follow the suggestions posted by Elton in the thread below:
https://social.technet.microsoft.com/Forums/en-US/e35da253-f0df-41d1-8df2-b73fa54742a0/windows-could-not-start-the-flexnet-licensing-service-service-on-local-computer-error-1068-the?forum=w7itproinstall
Best regards
Michael Shao
TechNet Community Support -
Start a cluster locally with 2 nodes
Hi,
I just want to ask that how to start a cluster locally with 2 nodes?
By default i think that it starts only 1.
ThanksHi,
Just start another instance of the same process using the same coherence configuration and they will form a cluster.
mark
Maybe you are looking for
-
How do I use a regular printer with my iPhone?
How do I use a regular printer with my iPhone?
-
Error - Business partner 30000011 does not exist in role TR0151
Hello, I am receiving an error "Business partner 30000011 does not exist in role TR0151" while creating a Fixed Deposit Transaction through FTR_CREATE. Whilst testing Fixed Deposit accounting in SAP Treasury, i was testing a scenario of a MINOR and h
-
Will Graphics Card Help Aperture on Mac Pro?
I've been using Aperture for many years on a variety of Mac hardware. I have 5 or 6 image libraries, ranging in size from 50gb to 250gb. I have no referenced images, do not use "faces" or other CPU-intensive features, and often perform tasks which re
-
I thought that Apple had it set up in such a manner that apps written for 10.x would run on an Intel based machine. Forrrrrrr getttttt itttttttt! After moving my copy of Pac The Man over to my mini I discovered what S L O W really was. What ever happ
-
I can open files directly from my Finder window (Mac OS, running latest installment). And I can copy/paste or drag/drop graphics into an open AI file for editing. But I can't use the AI>File>Open... or AI>File>Place... (or any shortcuts). Absolutely