Oracle RAC Nodes getting reboot in case of preferred controller failed

When we are disconnecting both Fiber cable from preferred Controller A or plugging out Controller A card from Disk Array(IBM DS 4300), After 90 seconds both the servers are rebooting.
In this time complete RAC network is going out of service for approx 5 minutes.After reboot both servers are coming with both instances without any manual intervention
It’s a critical issue for us because we are loosing High Availability, Let us know how we can resolve this critical issue.
Detail of Network:
1. Software- Oracle 10g Release2
2. OS- Redhat Linux 3 (Kernel Version-2.4.21-27.ELsmp)
3. Shared Storage- IBM DS 4300.
4. Multipathing Driver - RDAC (rdac-LINUX-09.00 A5.13)
4. Nodes- IBM 346
5. Databse on ASM
6. ASM,OCR & Voting Disk Preferred controller is A.
7. Hangcheck timer value is 210 seconds.
8. Both Server available with 2 HBA port . I HBA port is connected with Controller A and Seconfd HBA port is connected with Controller B of SAN Disk Array.
As per my understanding,
Voting disk resides in Disk Array and Controller A is preferred owner of Voting Disk LUN.. When i am disconnecting both fiber cable from preferred controller A , then Both Nodes Clusterware software trying to contact with Voting Disk, When they are unable to contact with Voting disk in specfic time period, they are going for reboot.
I tested Controller failure testing with Oracle RAC software as well without Oracle. Without Oracle its working fine and reason behind, in that time Disk Array is waiting for approx 300 seconds for changing preferred controlller from A to B.
But With Oracle, Clusterware Software reboot both nodes before Controller can shift from A to B.
So if i conclude,the tech who has good understanding of Oracle Clusterware on Linux OS & IBM RDAC multipath driver can help me.
when we install Oracle RAC on Linux, it is required to configure hangcheck timer.
Oracle recomends 180 second.
It means if one of node is hanging, then second node will wait for 180 seconds, if within 180 seconds ,it is not able to resolve this situation then it will reboot hung node.
I think Hangcheck timer configuration reuired only with Linux OS.
Configuration File
cat >> /etc/rc.d/rc.local << EOF
modprobe hangcheck-timer hangcheck_tick=15 hangcheck_margin=60

Sorry
Hangcheck timer is
Configuration File
cat >> /etc/rc.d/rc.local << EOF
modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

Similar Messages

  • If use MSSQ , when oracle rac node reboot, client get TPEOS error

    Hi, all
    in my tuxedo applicaton, if we use Single Server, Single Queue mode , when reboot any Oracle RAC node, our application is ok, client can get correct result. but if we use MSSQ（Multi Server, Single Queue) , if Oracle RAC node is ok , our application also is ok. but if we reboot any Oracle RAC node, client program can continue run, get correct result, but always get TPEOS error , for this situation， server can get client request, but client can not get server reply, only get TPEOS error.
    our enviroment is :
    oracle RAC ,10g 10.2.0.4 , two instances ,rac1 rac2, and two DTP services s1 and s2, set s1 and s2 services TAF is basic
    tuxedo 10R3 , two nodes ,work in MP model ，use XA access oracle rac database，services have Transaction and not Transaction
    OS is linux AS4 U5, 64bits
    service program use OCI
    can any one encounter this problem ?

    Hi, first thanks you
    in ULOG file , only have failover information, not any other error message, in client side also has no other error.
    not use MSSQ, ubb file about MSSQ config
    SERVERS
    DEFAULT:
    CLOPT="-A "
    sinUpdate_server SRVGRP=GROUP11 SRVID=80 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinUpdate_server SRVGRP=GROUP12 SRVID=160 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinCount_server SRVGRP=GROUP11 SRVID=240 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinCount_server SRVGRP=GROUP12 SRVID=320 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinSelect_server SRVGRP=GROUP11 SRVID=360 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinSelect_server SRVGRP=GROUP12 SRVID=400 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinInsert_server SRVGRP=GROUP11 SRVID=520 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinInsert_server SRVGRP=GROUP12 SRVID=560 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinDelete_server SRVGRP=GROUP11 SRVID=600 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinDelete_server SRVGRP=GROUP12 SRVID=640 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinDdl_server SRVGRP=GROUP11 SRVID=700 MIN=5 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinDdl_server SRVGRP=GROUP12 SRVID=740 MIN=5 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    lockselect_server SRVGRP=GROUP11 SRVID=800 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    lockselect_server SRVGRP=GROUP12 SRVID=840 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    #mulup_server SRVGRP=GROUP11 SRVID=1 MIN=2 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    #mulup_server SRVGRP=GROUP12 SRVID=60 MIN=2 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinUpdate_server SRVGRP=GROUP13 SRVID=83 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinUpdate_server SRVGRP=GROUP14 SRVID=164 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinCount_server SRVGRP=GROUP13 SRVID=243 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinCount_server SRVGRP=GROUP14 SRVID=324 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinSelect_server SRVGRP=GROUP13 SRVID=363 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinSelect_server SRVGRP=GROUP14 SRVID=404 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinInsert_server SRVGRP=GROUP13 SRVID=523 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinInsert_server SRVGRP=GROUP14 SRVID=564 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinDelete_server SRVGRP=GROUP13 SRVID=603 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinDelete_server SRVGRP=GROUP14 SRVID=644 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinDdl_server SRVGRP=GROUP13 SRVID=703 MIN=5 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    sinDdl_server SRVGRP=GROUP14 SRVID=744 MIN=5 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    lockselect_server SRVGRP=GROUP13 SRVID=803 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    lockselect_server SRVGRP=GROUP14 SRVID=844 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    #mulup_server SRVGRP=GROUP13 SRVID=13 MIN=2 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    #mulup_server SRVGRP=GROUP14 SRVID=64 MIN=2 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y
    WSL SRVGRP=GROUP11 SRVID=1000
    CLOPT="-A -- -n//120.3.8.237:7200 -I 60 -T 60 -w WSH -m 50 -M 100 -x 6 -N 3600"
    WSL SRVGRP=GROUP12 SRVID=1001
    CLOPT="-A -- -n//120.3.8.238:7200 -I 60 -T 60 -w WSH -m 50 -M 100 -x 6 -N 3600"
    WSL SRVGRP=GROUP13 SRVID=1003
    CLOPT="-A -- -n//120.3.8.237:7203 -I 60 -T 60 -w WSH -m 50 -M 100 -x 6 -N 3600"
    WSL SRVGRP=GROUP14 SRVID=1004
    CLOPT="-A -- -n//120.3.8.238:7204 -I 60 -T 60 -w WSH -m 50 -M 100 -x 6 -N 3600"
    if we use MSSQ ,ubb file about MSSQ config is
    *SERVERS
    DEFAULT:
    CLOPT="-A -p 1,60:1,30"
    sinUpdate_server SRVGRP=GROUP11 SRVID=80 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinUpdate11 REPLYQ=Y
    sinUpdate_server SRVGRP=GROUP12 SRVID=160 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinUpdate12 REPLYQ=Y
    sinCount_server SRVGRP=GROUP11 SRVID=240 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinCount11 REPLYQ=Y
    sinCount_server SRVGRP=GROUP12 SRVID=320 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinCount12 REPLYQ=Y
    sinSelect_server SRVGRP=GROUP11 SRVID=360 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinSelec11 REPLYQ=Y
    sinSelect_server SRVGRP=GROUP12 SRVID=400 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinSelect12 REPLYQ=Y
    sinInsert_server SRVGRP=GROUP11 SRVID=520 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinInsert11 REPLYQ=Y
    sinInsert_server SRVGRP=GROUP12 SRVID=560 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinInsert12 REPLYQ=Y
    sinDelete_server SRVGRP=GROUP11 SRVID=600 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinDelete11 REPLYQ=Y
    sinDelete_server SRVGRP=GROUP12 SRVID=640 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinDelete12 REPLYQ=Y
    sinDdl_server SRVGRP=GROUP11 SRVID=700 MIN=5 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinDdl11 REPLYQ=Y
    sinDdl_server SRVGRP=GROUP12 SRVID=740 MIN=5 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinDdl12 REPLYQ=Y
    lockselect_server SRVGRP=GROUP11 SRVID=800 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=lockselect11 REPLYQ=Y
    lockselect_server SRVGRP=GROUP12 SRVID=840 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=lockselect12 REPLYQ=Y
    #mulup_server SRVGRP=GROUP11 SRVID=1 MIN=2 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=mulup11 REPLYQ=Y
    #mulup_server SRVGRP=GROUP12 SRVID=60 MIN=2 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=mulup12 REPLYQ=Y
    sinUpdate_server SRVGRP=GROUP13 SRVID=83 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinUpdate13 REPLYQ=Y
    sinUpdate_server SRVGRP=GROUP14 SRVID=164 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinUpdate14 REPLYQ=Y
    sinCount_server SRVGRP=GROUP13 SRVID=243 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinCount13 REPLYQ=Y
    sinCount_server SRVGRP=GROUP14 SRVID=324 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinCount14 REPLYQ=Y
    sinSelect_server SRVGRP=GROUP13 SRVID=363 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinSelec13 REPLYQ=Y
    sinSelect_server SRVGRP=GROUP14 SRVID=404 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinSelect14 REPLYQ=Y
    sinInsert_server SRVGRP=GROUP13 SRVID=523 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinInsert13 REPLYQ=Y
    sinInsert_server SRVGRP=GROUP14 SRVID=564 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinInsert14 REPLYQ=Y
    sinDelete_server SRVGRP=GROUP13 SRVID=603 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinDelete13 REPLYQ=Y
    sinDelete_server SRVGRP=GROUP14 SRVID=644 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinDelete14 REPLYQ=Y
    sinDdl_server SRVGRP=GROUP13 SRVID=703 MIN=5 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinDdl13 REPLYQ=Y
    sinDdl_server SRVGRP=GROUP14 SRVID=744 MIN=5 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=sinDdl14 REPLYQ=Y
    lockselect_server SRVGRP=GROUP13 SRVID=803 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=lockselect13 REPLYQ=Y
    lockselect_server SRVGRP=GROUP14 SRVID=844 MIN=10 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=lockselect14 REPLYQ=Y
    #mulup_server SRVGRP=GROUP13 SRVID=13 MIN=2 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=mulup13 REPLYQ=Y
    #mulup_server SRVGRP=GROUP14 SRVID=64 MIN=2 MAX=30 MAXGEN=10 GRACE=10 RESTART=Y RQADDR=mulup14 REPLYQ=Y
    WSL SRVGRP=GROUP11 SRVID=1000
    CLOPT="-A -- -n//120.3.8.237:7200 -I 60 -T 60 -w WSH -m 50 -M 100 -x 6 -N 3600"
    WSL SRVGRP=GROUP12 SRVID=1001
    CLOPT="-A -- -n//120.3.8.238:7200 -I 60 -T 60 -w WSH -m 50 -M 100 -x 6 -N 3600"
    WSL SRVGRP=GROUP13 SRVID=1003
    CLOPT="-A -- -n//120.3.8.237:7203 -I 60 -T 60 -w WSH -m 50 -M 100 -x 6 -N 3600"
    WSL SRVGRP=GROUP14 SRVID=1004
    CLOPT="-A -- -n//120.3.8.238:7204 -I 60 -T 60 -w WSH -m 50 -M 100 -x 6 -N 3600"
    about above ubb file ,has any error ? or not correct use MSSQ
    look forward to you answer,thanks.

  • Huge number of idle connections from loopback ip on oracle RAC node

    Hi,
    We have a 2node 11gR2(11.2.0.3) oracle RAC node. We are seeing huge number of idle connection(more than 5000 in each node) on both the nodes and increasing day by day. All the idle connections are from VIP and loopback address(127.0.0.1.47971 )
    netstat -an |grep -i idle|more
    127.0.0.1.47971 Idle
    any insight will be helpful.
    The server is suffering memory issues occasionally (once in a month).
    ORA-27300: OS system dependent operation:fork failed with status: 11
    ORA-27301: OS failure message: Resource temporarily unavailable
    Thanks

    user12959884 wrote:
    Hi,
    We have a 2node 11gR2(11.2.0.3) oracle RAC node. We are seeing huge number of idle connection(more than 5000 in each node) on both the nodes and increasing day by day. All the idle connections are from VIP and loopback address(127.0.0.1.47971 )
    netstat -an |grep -i idle|more
    127.0.0.1.47971 Idle
    any insight will be helpful.
    The server is suffering memory issues occasionally (once in a month).
    ORA-27300: OS system dependent operation:fork failed with status: 11
    ORA-27301: OS failure message: Resource temporarily unavailable
    Thankswe can not control what occurs on your DB Server.
    How do I ask a question on the forums?
    SQL and PL/SQL FAQ
    post results from following SQL
    SELECT * FROM V$VERSION;

  • Oracle RAC 10.2G reboots node every 45 minutes

    Hello:
    - We have installed Oracle RAC 10.2G for Solaris X86 ( 64 bit ).
    - On one node, there are no issues. But the other node ( I think )
    is being rebooted by CRS every 45 minutes or so.
    - Is this issue caused by some misconfiguration I did during the install ?
    - Or is there a patch available to fix this ?
    - Has anyone else encountered this problem ?
    Thanks
    jlem

    Hello:
    - I re-installed Oracle RAC. The nodes were only rebooted once so far.
    So, the second install may be ok. If not, I have provided answers to the first email reply.
    - Any help given is most welcome. In meantime, I will continue searching the oracle forums
    for solutions.
    - My environment is:
    - both nodes are running under vmware ESX server version 3.0.1
    - the shared storage for OCR and Voting Disk is a raw shared device under vmware
    - both nodes are using Solaris X86 5.10 update 5
    - Oracle version is: 10.2.0.3 ( patched from version 10.2.0.1 )
    - My public network configuration is:
    node 1:
    e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
    inet 10.20.1.74 netmask ffff0000 broadcast 10.20.255.255
    ether 0:c:29:3a:45:a9
    e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
    inet 10.20.1.77 netmask ffff0000 broadcast 10.20.255.255
    node 2:
    e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
    inet 10.20.1.75 netmask ffff0000 broadcast 10.20.255.255
    ether 0:c:29:2b:db:90
    e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
    inet 10.20.1.78 netmask ffff0000 broadcast 10.20.255.255
    - My private network configuration is:
    node 1:
    e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
    inet 192.168.0.1 netmask ffffff00 broadcast 192.168.0.255
    ether 0:c:29:3a:45:b3
    node 2:
    e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
    inet 192.168.0.2 netmask ffffff00 broadcast 192.168.0.255
    ether 0:c:29:2b:db:9a
    - My storage solution is:
    - 3 virtual shared SCSI hard disks ( each 500 MB in size )
    - My log files are:
    - /var/adm/messages
    - doesn't report much only the following:
    Nov 12 10:57:05 saucer nfs4cbd[328]: [ID 867284 daemon.notice] nfsv4 cannot determine local hostname binding for transport
    tcp6 - delegations will not be available on this transport
    Nov 12 10:57:21 saucer savecore: [ID 570001 auth.error] reboot after panic: forced crash dump initiated at user requestNov 12 10:57:21 saucer savecore: [ID 748169 auth.error] saving system crash dump in /var/crash/saucer/*.2Nov 12 10:57:41 saucer root: [ID 702911 user.error] Oracle Cluster Ready Services disabled by administrator.Nov 12 10:57:54 saucer rootnex: [ID 349649 kern.info] xsvc0 at rootNov 12 10:57:54 saucer genunix: [ID 936769 kern.info] xsvc0 is /xsvc
    - ocssd.log file for node1 indicates that node2 was evicted for impeding a reconfig. Details are:
    [    CSSD]2008-11-12 10:55:43.700 [15] >TRACE: clssnmPollingThread: node saucer (2) is impending reconfig
    [    CSSD]2008-11-12 10:55:43.700 [15] >WARNING: clssnmPollingThread: node saucer (2) at 90% heartbeat fatal, eviction in 0
    .973 seconds
    [    CSSD]2008-11-12 10:55:44.679 [15] >TRACE: clssnmPollingThread: node saucer (2) is impending reconfig
    [    CSSD]2008-11-12 10:55:44.679 [15] >TRACE: clssnmPollingThread: Eviction started for node saucer (2), flags 0x000d, s
    tate 3, wt4c 0
    [    CSSD]2008-11-12 10:55:44.690 [17] >TRACE: clssnmDoSyncUpdate: Initiating sync 3
    [    CSSD]2008-11-12 10:55:44.690 [17] >TRACE: clssnmDoSyncUpdate: diskTimeout set to (27000)ms
    [    CSSD]2008-11-12 10:55:44.691 [17] >TRACE: clssnmSetupAckWait: Ack message type (11)
    [    CSSD]2008-11-12 10:55:44.691 [17] >TRACE: clssnmSetupAckWait: node(1) is ALIVE
    [    CSSD]2008-11-12 10:55:44.691 [17] >TRACE: clssnmSetupAckWait: node(2) is ALIVE
    [    CSSD]2008-11-12 10:55:44.691 [17] >TRACE: clssnmSendSync: syncSeqNo(3)
    - node2 ocssd.log does not indicate the problem. See below for details:
    [    CSSD]2008-11-12 10:52:34.731 [11] >TRACE: clssgmClientConnectMsg: Connect from con(da8410) proc(dab900) pid() proto(
    10:2:1:1)
    [    CSSD]2008-11-12 10:53:37.305 [11] >TRACE: clssgmClientConnectMsg: Connect from con(da8410) proc(dab900) pid() proto(
    10:2:1:1)
    [    CSSD]2008-11-12 10:54:40.515 [11] >TRACE: clssgmClientConnectMsg: Connect from con(da8410) proc(dab900) pid() proto(
    10:2:1:1)
    [    CSSD]2008-11-12 11:18:09.997 >USER: Oracle Database 10g CSS Release 10.2.0.3.0 Production Copyright 1996, 2004 Orac
    le. All rights reserved.
    [    CSSD]2008-11-12 11:18:09.997 >USER: CSS daemon log for node saucer, number 2, in cluster crs
    [  clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=saucerDBG_CSSD))
    [    CSSD]2008-11-12 11:18:10.016 [1] >TRACE: clssscmain: local-only set to false
    [    CSSD]2008-11-12 11:18:10.031 [1] >TRACE: clssnmReadNodeInfo: added node 1 (flying) to cluster
    [    CSSD]2008-11-12 11:18:10.042 [1] >TRACE: clssnmReadNodeInfo: added node 2 (saucer) to cluster
    [    CSSD]2008-11-12 11:18:10.057 [5] >TRACE: clssnm_skgxnmon: skgxn init failed
    [    CSSD]2008-11-12 11:18:10.057 [1] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor
    - ORACLE VERIFY: cluvfy was run on node2 resulting with the following:
    bash-3.00$ ./cluvfy comp ocr -n all -verbose
    Verifying OCR integrity
    Checking OCR integrity...
    Checking the absence of a non-clustered configuration...
    All nodes free of non-clustered, local-only configurations.
    Uniqueness check for OCR device passed.
    Checking the version of OCR...
    OCR of correct Version "2" exists.
    Checking data integrity of OCR...
    Data integrity check for OCR passed.
    OCR integrity check passed.
    Verification of OCR integrity was successful.
    bash-3.00$
    Thanks
    jlem

  • Oracle RAC - Not getting performance(TPS) as we expect on insert/update

    Hi All,
    We got a problem while executing insert/update and delete queries with Oracle RAC system, we are not getting the TPS as we expected in Oracle RAC. The TPS of Oracle RAC (for insert/update and delete ) is less than as that of
    single oracle system.
    But while executing select queries, we are getting almost double TPS as that of Single Oracle System.
    We have done server side and client side load balancing.
    Can anyone knows to solve this strange behaviour? Shall we need to perform any other settings in ASM/ Oracle Nodes
    for better performance on insert/update and delete queries.
    The following is the Oracle RAC configuration
    OS & Hardware :Windows 2008 R2 , Core 2 Du0 2.66GHz , 4 GB
    Software : Oracle 11g 64 Bit R2 , Oracle Clusterware & ASM , Microsoft iSCSI initiator.
    Storage Simulation : Xeon 4GB , 240 GB ,Win 2008 R2, Microsoft iSCSI Traget
    Please help me to solve this. We are almost stuck with this situation.
    Thanks
    Roy

    Load Profile Per Second Per Transaction Per Exec Per Call
    ~~~~~~~~~~~~ ------------------ ----------------- ----------- -----------
    DB time(s): 48.3 0.3 0.26 0.10
    DB CPU(s): 0.1 0.0 0.00 0.00
    Redo size: 523,787.9 3,158.4
    Logical reads: 6,134.6 37.0
    Block changes: 3,247.1 19.6
    Physical reads: 3.5 0.0
    Physical writes: 50.7 0.3
    User calls: 497.6 3.0
    Parses: 182.0 1.1
    Hard parses: 0.1 0.0
    W/A MB processed: 0.1 0.0
    Logons: 0.1 0.0
    Executes: 184.0 1.1
    Rollbacks: 0.0 0.0
    Transactions: 165.8
    Instance Efficiency Indicators
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Buffer Nowait %: 93.74 Redo NoWait %: 99.96
    Buffer Hit %: 99.99 Optimal W/A Exec %: 100.00
    Library Hit %: 100.19 Soft Parse %: 99.96
    Execute to Parse %: 1.09 Latch Hit %: 99.63
    Parse CPU to Parse Elapsd %: 16.44 % Non-Parse CPU: 84.62
    Shared Pool Statistics Begin End
    Memory Usage %: 75.89 77.67
    % SQL with executions>1: 71.75 69.88
    % Memory for SQL w/exec>1: 75.63 71.38

  • Domain migration of Oracle RAC nodes on windows 2008

    Hello masters,
    We are planning to migrate servers from old domain to a new domain due to company name change. That is, our DB servers named like db-node-x.old-domain.com will be migrated to db-node-x.new-domain.com. The servers are all on Windows 2008 R2 and Oracle RAC 11g2. As dba, i have no idea about what will happen to DBs. Please let me know any possible solutions?
    Thanks in advance.

    user2033016 wrote:
    Hi Everyone,
    I installed Oracle RAC 11g on Windows 2008 64 bit, The installation was successfull crs_state -t executed successfully.
    Bt the machine got crashed ,when executed a select query with 100 sessions( For performance evaluation using TG). A blue screen is appeared on two nodes. So i am not able to evaluate the performance.
    Can you please tell any suggesstions to solve this issue? Is our hardware is not sufficient? Or any software settings has to be done?. I tried most of hte technique ,bt the problem still exists.
    The following machine configuration is used for Oracle RAC setup
    Intel Core 2 Duo 2.66 GHz,
    Ram :4GB.
    OS : windows 2008 Server 64 Bit
    Oracle RAC , Clusterware 11g R2 64 Bit.i guess you have installed/configured RAC using VMware. i guess, the problem is with your RAM size. how much disk space and RAM size you have allocated for each machine?
    which documentation you have followed?

  • Oracle RAC Node Crashing

    Hi,
    We recently upgraded to Oracle RAC database to support Siebel Call Center APplication (8.1.1.11.12 [IP2013]).Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    Frequently we are facing problem of crashing with the Nodes in RAC (Kernal Panic). Due to this users using siebel application are unstable and facing multiple issues Once we restore Siebel Gateway and restart of AOM server fixing the issue.
    Below are the SQL's and Crashing Signature we are seeing du to this crash.
    DBCLog DBCLogError 1 00081f1254bb0ad4:1423830 2015-01-18 11:49:20 OCIStmtExecute: SELECT
    T2.CONFLICT_ID,
    T2.LAST_UPD,
    T2.CREATED,
    T2.LAST_UPD_BY,
    T2.CREATED_BY,
    T2.MODIFICATION_NUM,
    T2.ROW_ID,
    T2.EFF_START_TS,
    T2.EFF_END_TS,
    T2.NAME,
    T2.BUSOBJ_NAME,
    T2.VERSION,
    T2.DEPLOY_STATUS_CD,
    T2.MODE_CD
    FROM
    SIEBEL.S_LST_OF_VAL T1,
    SIEBEL.S_WFA_DPLOY_DEF T2
    WHERE
    T2.DEPLOY_STATUS_CD = T1.NAME AND T1.TYPE = 'WFA_DPLY_STAT_CD' AND T1.LANG_ID = :1 AND
    (T2.TYPE_CD IN ( :2 )) AND
    (T2.NAME = :3 AND T2.DEPLOY_STATUS_CD IN ( :4 ) AND (T2.EFF_END_TS > TO_DATE(:5,'MM/DD/YYYY HH24:MI:SS') OR T2.EFF_END_TS IS NULL))
    ORDER BY
    T2.NAME, T1.VAL
    ObjMgrLog Error 1 00081f1254bb0ad4:1423830 2015-01-18 11:49:20 (oracon.cpp (3896)) SBL-DBC-00105: An error has occurred executing a Sql statement.
    Please continue or ask your systems administrator to check your application configuration if the problem persists.
    and Crash Signature Looks like below
    Exception 0xc0000005 at 0x006500f6
    Thread: 0x00002700, Process 0x00000254
    Thread: 9984, Process 596
    - CONTEXT -
    EIP: 0x00000000, EFL: 0x00010246, FS:  0x0000003b, GS:  0x00000000
    CS:  0x0000001b, DS:  0x00000023, SS:  0x00000023, ES:  0x00000023
    EAX: 0x1a511968, EBX: 0x294ca048, ECX: 0x4cc50008, EDX: 0x3a2ad200
    ESI: 0x4cc50008, EDI: 0x00000000, EBP: 0x00000000, ESP: 0x3a2afff4
    - CALL STACK -
    sslcscr +0xf6
    sscfcmn +0x19050 = CSSObjectBase::CopyErrorMsg() +0xf0
    sscfcmn +0x1950d = CSSObjectBase::DoSetErrorMsg() +0x13d
    sscfcmn +0x19bb5 = CSSObjectBase::SetErrorMsg() +0xa5
    SSFACMBC +0x34a6 = CSSFABCVRec::IsBusinessProcessActive() +0x3b6
    SSFACMBC +0x5133 = CSSFABCVRec::VBCCreate() +0x1e3
    sscfom +0x71d1e = CSSBCVRec::SqlCreate() +0xde
    sscfom +0x4aea = CSSModel::ConstructBusComp() +0xaa
    sscfom +0x653f5 = CSSBusObj::GetBusComp() +0x265
    sscfom +0x65a18 = CSSBusObj::GetBusComp() +0x18
    sscfom +0x13ad8 = CSSModel::BeginSetupView() +0x418
    sscfom +0x13dee = CSSModel::BeginSetupView() +0x5e
    sscaswbc +0x8b070 = CSSSWEUdfLoader::BeginSetupView() +0x30
    sscaswbc +0xe3d88 = CSSSWEFrameMgrInternal::BuildView() +0x1618
    sscaswbc +0xe5646 = CSSSWEFrameMgrInternal::BuildView() +0x36
    sscaswbc +0xb8343 = CSSSWEActionMgr::ActionBuildViewAsync() +0x293
    sscaswbc +0xb873b = CSSSWEActionMgr::DoPostedAction() +0xab
    sscaswbc +0xb9287 = CSSSWEActionMgr::CheckPostedActions() +0x3d7
    sscaswbc +0xe6d41 = CSSSWEFrameMgrInternal::InvokeAppletMethod() +0x11b1
    sscaswbc +0xbbcdc = CSSSWECmdProcessor::InvokeMethod() +0x74c
    sscaswbc +0xc1495 = CSSSWECmdProcessor::_ProcessCommand() +0x725
    sscaswbc +0xc27f8 = CSSSWECmdProcessor::ProcessCommand() +0xab8
    sscaswbc +0xc2d04 = CSSSWECmdProcessor::ProcessCommand() +0xc4
    sscaswbc +0x265559 = CSSServiceSWEIface::Request() +0x309
    sscaswbc +0x267391 = CSSServiceSWEIface::DoInvokeMethod() +0xed1
    sscfom +0x1976c = CSSService::InvokeMethod() +0x24c
    sstcsiom +0x6d6a = CSSSIOMSession::ModInvokeSrvcMethod() +0x10a
    sstcsiom +0x7613 = CSSSIOMSession::RPCMiscModel() +0x513
    sstcsiom +0xe9dd = CSSSIOMSession::HandleRPC() +0x3ad
    sssasos +0x6616 = CompCleanup() +0x3be6
    sssasos +0x5a95 = CompCleanup() +0x3065
    sssasos +0x1a35 = CSSSISObject::operator=() +0x9f5
    sssasos +0x2135 = CompHandleMsg() +0x485
    siebmtshmw +0x494e
    siebmtshmw +0x20051 = GetSmiTaskAPI() +0x6d21
    siebmtshmw +0x239d1 = SmiInProcMsgHandler() +0x2051
    siebmtshmw +0x15f15 = SmiBeginTrace() +0x3195
    siebmtshmw +0x174cf = GetSmiWorkQObj() +0x148f
    siebmtshmw +0xc7f7 = SmiCleanupDetTask() +0x3be7
    sslcosd +0x2152 = OSDThreadPrivIsInit() +0x1d2
    sslcosd +0x21dc = OSDThreadPrivIsInit() +0x25c
    MSVCR71 +0x9565 = endthreadex() +0xa0
    kernel32 +0x2484f = GetModuleHandleA() +0xdf
    - STACK DUMP -
    - MODULE LIST -
    00400000-00445000: C:\siebel\siebsrvr\BIN\siebmtshmw.exe, 811.11.822.4
    7c800000-7c8c3000: C:\WINDOWS\system32\ntdll.dll, 5.2.3790.4937
    77e40000-77f44000: C:\WINDOWS\system32\kernel32.dll, 5.2.3790.5295
    10000000-10025000: C:\siebel\siebsrvr\BIN\ssscsci.dll, 811.11.822.4
    00320000-003cc000: C:\siebel\siebsrvr\BIN\ssscscf.dll, 811.11.822.4
    003d0000-003d7000: C:\siebel\siebsrvr\BIN\sslcosa.dll, 811.11.822.4
    7c340000-7c396000: C:\siebel\siebsrvr\BIN\MSVCR71.dll, 7.10.3052.4
    00450000-0054d000: C:\siebel\siebsrvr\BIN\sslcshar.dll, 811.11.822.4
    77380000-77411000: C:\WINDOWS\system32\USER32.dll, 5.2.3790.4033
    77c00000-77c49000: C:\WINDOWS\system32\GDI32.dll, 5.2.3790.5398
    7d1e0000-7d27c000: C:\WINDOWS\system32\ADVAPI32.dll, 5.2.3790.4455
    77c50000-77cf1000: C:\WINDOWS\system32\RPCRT4.dll, 5.2.3790.5254
    76f50000-76f63000: C:\WINDOWS\system32\Secur32.dll, 5.2.3790.4455
    77b90000-77b98000: C:\WINDOWS\system32\VERSION.dll, 5.2.3790.3959
    77ba0000-77bfa000: C:\WINDOWS\system32\msvcrt.dll, 7.0.3790.3959
    71bb0000-71bb9000: C:\WINDOWS\system32\WSOCK32.dll, 5.2.3790.0
    71c00000-71c17000: C:\WINDOWS\system32\WS2_32.dll, 5.2.3790.3959
    71bf0000-71bf8000: C:\WINDOWS\system32\WS2HELP.dll, 5.2.3790.3959
    003e0000-003e6000: C:\siebel\siebsrvr\BIN\libarm.dll, 811.10.822.3
    00550000-005d5000: C:\siebel\siebsrvr\BIN\sslcsym.dll, 811.11.822.4
    7c3a0000-7c41b000: C:\siebel\siebsrvr\BIN\MSVCP71.dll, 7.10.3077.0
    005e0000-005f1000: C:\siebel\siebsrvr\BIN\sslccore.dll, 811.11.822.4
    00600000-00618000: C:\siebel\siebsrvr\BIN\sslcosd.dll, 811.11.822.4
    00620000-00645000: C:\siebel\siebsrvr\BIN\sslcrsa.dll, 811.11.822.4
    00650000-0067f000: C:\siebel\siebsrvr\BIN\sslcscr.dll, 811.11.822.4
    00680000-007cf000: C:\siebel\siebsrvr\BIN\sslcsrd.dll, 811.11.822.4
    007d0000-0082b000: C:\siebel\siebsrvr\BIN\sslcscc.dll, 811.11.822.4
    003f0000-003ff000: C:\siebel\siebsrvr\BIN\sslcevt.dll, 811.11.822.4
    00830000-00840000: C:\siebel\siebsrvr\BIN\sslcsobj.dll, 811.11.822.4
    00840000-00850000: C:\siebel\siebsrvr\BIN\sslcsnns.dll, 811.11.822.4
    00850000-00931000: C:\siebel\siebsrvr\BIN\sslcnapi.dll, 811.11.822.4
    00940000-00954000: C:\siebel\siebsrvr\BIN\sslcns.dll, 811.11.822.4
    00960000-00968000: C:\siebel\siebsrvr\BIN\sslcsnsc.dll, 811.11.822.4
    00970000-0097b000: C:\siebel\siebsrvr\BIN\ssscsmi.dll, 811.11.822.4
    00980000-00991000: C:\siebel\siebsrvr\BIN\sslcsrvr.dll, 811.11.822.4
    009a0000-009bb000: C:\siebel\siebsrvr\BIN\sslcos.dll, 811.11.822.4
    009c0000-00a5f000: C:\siebel\siebsrvr\BIN\sscfcmn.dll, 811.11.822.4
    7c8d0000-7d0cf000: C:\WINDOWS\system32\SHELL32.dll, 6.0.3790.5018
    7d180000-7d1d2000: C:\WINDOWS\system32\SHLWAPI.dll, 6.0.3790.5318
    77670000-777aa000: C:\WINDOWS\system32\ole32.dll, 5.2.3790.5209
    7d0e0000-7d16c000: C:\WINDOWS\system32\OLEAUT32.dll, 5.2.3790.5464
    761b0000-76245000: C:\WINDOWS\system32\CRYPT32.dll, 5.131.3790.5235
    76190000-761a2000: C:\WINDOWS\system32\MSASN1.dll, 5.2.3790.4584
    77210000-772bb000: C:\WINDOWS\system32\WININET.dll, 6.0.3790.5467
    00a60000-00a70000: C:\siebel\siebsrvr\BIN\sslczlib.dll, 811.11.822.4
    00a70000-00b72000: C:\siebel\siebsrvr\BIN\MFC71U.DLL, 7.10.3077.0
    00b80000-00b88000: C:\siebel\siebsrvr\BIN\sslcsnsr.dll, 811.11.822.4
    00b90000-00b99000: C:\siebel\siebsrvr\BIN\sslcsrtr.dll, 811.11.822.4
    00ba0000-00baf000: C:\siebel\siebsrvr\BIN\sslcsrms.dll, 811.11.822.4
    00bb0000-00bb6000: C:\siebel\siebsrvr\BIN\sslcsssm.dll, 811.11.822.4
    00bc0000-00bd1000: C:\siebel\siebsrvr\BIN\sslcsrcn.dll, 811.11.822.4
    00be0000-00be7000: C:\siebel\siebsrvr\BIN\sslcsysstat.dll, 811.11.822.4
    00fe0000-0100d000: C:\siebel\siebsrvr\BIN\sslcrsa256.dll, 8.1.1.5
    77420000-77523000: C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.5190_x-ww_319264BE\comctl32.dll, 6.0.3790.5190
    5d360000-5d36e000: C:\siebel\siebsrvr\BIN\MFC71ENU.DLL, 7.10.3077.0
    76b70000-76b7b000: C:\WINDOWS\system32\psapi.dll, 5.2.3790.3959
    012b0000-012c0000: C:\siebel\siebsrvr\BIN\sslcasrv.dll, 811.11.822.4
    012c0000-01328000: C:\siebel\siebsrvr\BIN\sslcsrs.dll, 811.11.822.4
    01330000-0133a000: C:\siebel\siebsrvr\BIN\sslcntfy.dll, 811.11.822.4
    01740000-01746000: C:\siebel\siebsrvr\BIN\enu\srcf.dll, 811.9.822.2
    01750000-01756000: C:\siebel\siebsrvr\BIN\enu\srcfapps.dll, 811.9.822.2
    01760000-01766000: C:\siebel\siebsrvr\BIN\sbcf.dll, 811.9.822.2
    01770000-01776000: C:\siebel\siebsrvr\BIN\sbcfapps.dll, 811.9.822.2
    01780000-01791000: C:\siebel\siebsrvr\BIN\sssasos.dll, 811.11.822.4
    017a0000-017a8000: C:\siebel\siebsrvr\BIN\sslcom.dll, 811.11.822.4
    017b0000-017be000: C:\siebel\siebsrvr\BIN\sslcsnom.dll, 811.11.822.4
    017c0000-017f8000: C:\siebel\siebsrvr\BIN\sstcsiom.dll, 811.11.822.4
    020f0000-021f0000: C:\siebel\siebsrvr\BIN\sscfom.dll, 811.11.822.4
    021f0000-022ef000: C:\siebel\siebsrvr\BIN\sslcsc.dll, 811.11.822.4
    022f0000-0236f000: C:\siebel\siebsrvr\BIN\sslcdb.dll, 811.11.822.4
    4bf70000-4bfae000: C:\WINDOWS\system32\ODBC32.dll, 3.526.4795.0
    77530000-775c7000: C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_5.82.3790.5190_x-ww_D21E1F39\COMCTL32.dll, 5.82.3790.5190
    762b0000-762f9000: C:\WINDOWS\system32\comdlg32.dll, 6.0.3790.3959
    01800000-0180b000: C:\siebel\siebsrvr\BIN\sscadsecr.dll, 811.11.822.4
    02370000-0237d000: C:\siebel\siebsrvr\BIN\sslclm.dll, 811.11.822.4
    02380000-0239c000: C:\siebel\siebsrvr\BIN\sslcupg.dll, 811.11.822.4
    023a0000-023ba000: C:\siebel\siebsrvr\BIN\sslcsecm.dll, 811.11.822.4
    023c0000-023ef000: C:\siebel\siebsrvr\BIN\sscfel.dll, 811.11.822.4
    023f0000-026b0000: C:\siebel\siebsrvr\BIN\sscfdm.dll, 811.11.822.4
    026b0000-026bd000: C:\siebel\siebsrvr\BIN\sslcfsm.dll, 811.11.822.4
    026c0000-026c8000: C:\siebel\siebsrvr\BIN\sslcsecc.dll, 811.11.822.4
    026d0000-026e1000: C:\siebel\siebsrvr\BIN\sslccachev2.dll, 811.11.822.4
    026f0000-026f9000: C:\siebel\siebsrvr\BIN\sslcmsgagent.dll, 811.11.822.4
    02bd0000-02be7000: C:\WINDOWS\system32\odbcint.dll, 3.526.3959.0
    02bf0000-02c23000: C:\siebel\siebsrvr\BIN\sslcrsa56.dll, 811.11.822.4
    71b20000-71b61000: C:\WINDOWS\System32\mswsock.dll, 5.2.3790.4318
    76ed0000-76efa000: C:\WINDOWS\system32\DNSAPI.dll, 5.2.3790.4840
    76f70000-76f77000: C:\WINDOWS\System32\winrnr.dll, 5.2.3790.3959
    76f10000-76f3e000: C:\WINDOWS\system32\WLDAP32.dll, 5.2.3790.3959
    76f80000-76f85000: C:\WINDOWS\system32\rasadhlp.dll, 5.2.3790.3959
    5f270000-5f2ca000: C:\WINDOWS\system32\hnetcfg.dll, 5.2.3790.3959
    71ae0000-71ae8000: C:\WINDOWS\System32\wshtcpip.dll, 5.2.3790.3959
    0ae80000-0af6a000: C:\siebel\siebsrvr\BIN\eBaySecAda.dll
    76920000-769e2000: C:\WINDOWS\system32\userenv.dll, 5.2.3790.3959
    71c40000-71c97000: C:\WINDOWS\system32\netapi32.dll, 5.2.3790.5030
    76bb0000-76bdc000: C:\WINDOWS\system32\wintrust.dll, 5.131.3790.5060
    76c10000-76c3a000: C:\WINDOWS\system32\imagehlp.dll, 5.2.3790.5240
    76750000-76779000: C:\WINDOWS\system32\schannel.dll, 5.2.3790.5462
    76e90000-76ecf000: C:\WINDOWS\system32\RASAPI32.DLL, 5.2.3790.3959
    76e40000-76e52000: C:\WINDOWS\system32\rasman.dll, 5.2.3790.3959
    76e60000-76e8f000: C:\WINDOWS\system32\TAPI32.dll, 5.2.3790.3959
    76e30000-76e3c000: C:\WINDOWS\system32\rtutils.dll, 5.2.3790.3959
    76aa0000-76acd000: C:\WINDOWS\system32\WINMM.dll, 5.2.3790.4916
    722f0000-722f5000: C:\WINDOWS\system32\sensapi.dll, 5.2.3790.3959
    772c0000-77371000: C:\WINDOWS\system32\urlmon.dll, 6.0.3790.5467
    68000000-68035000: C:\WINDOWS\system32\rsaenh.dll, 5.2.3790.3959
    68100000-68127000: C:\WINDOWS\system32\dssenh.dll, 5.2.3790.3959
    0b300000-0b31a000: C:\siebel\siebsrvr\BIN\sscdo90.dll, 811.11.822.4
    0b320000-0b3a2000: C:\oracle\product\11.1.0\client_1\bin\OCI.dll, 11.1.0.1
    61c20000-61f81000: C:\oracle\product\11.1.0\client_1\bin\OraClient11.Dll, 11.1.0.7
    60870000-6095b000: C:\oracle\product\11.1.0\client_1\bin\oracore11.dll, 11.1.0.7
    60a80000-60b4e000: C:\oracle\product\11.1.0\client_1\bin\oranls11.dll, 11.1.0.7
    63690000-636a8000: C:\oracle\product\11.1.0\client_1\bin\oraunls11.dll, 11.1.0.7
    60eb0000-60eb7000: C:\oracle\product\11.1.0\client_1\bin\orauts.dll, 11.1.0.7
    636b0000-636b6000: C:\oracle\product\11.1.0\client_1\bin\oravsn11.dll, 11.1.0.7
    60fa0000-61158000: C:\oracle\product\11.1.0\client_1\bin\oracommon11.dll, 11.1.0.7
    63430000-6345d000: C:\oracle\product\11.1.0\client_1\bin\orasnls11.dll, 11.1.0.7
    0b3b0000-0be77000: C:\oracle\product\11.1.0\client_1\bin\orageneric11.dll, 11.1.0.7
    0be80000-0c0dc000: C:\oracle\product\11.1.0\client_1\bin\oraxml11.dll, 11.1.0.7
    60020000-60030000: C:\WINDOWS\system32\MSVCIRT.dll, 7.0.3790.0
    0c0e0000-0c4b2000: C:\oracle\product\11.1.0\client_1\bin\oran11.dll, 11.1.0.6
    62740000-627a3000: C:\oracle\product\11.1.0\client_1\bin\oranl11.dll, 11.1.0.6
    62980000-629b1000: C:\oracle\product\11.1.0\client_1\bin\orantcp11.dll, 11.1.0.6
    0c4c0000-0c507000: C:\oracle\product\11.1.0\client_1\bin\oranldap11.dll, 11.1.0.6
    627f0000-62905000: C:\oracle\product\11.1.0\client_1\bin\orannzsbb11.dll, 11.1.0.6
    62530000-6266d000: C:\oracle\product\11.1.0\client_1\bin\oraldapclnt11.dll, 11.1.0.7
    62670000-6268b000: C:\oracle\product\11.1.0\client_1\bin\orancrypt11.dll, 11.1.0.6
    0c510000-0c643000: C:\oracle\product\11.1.0\client_1\bin\orazt11.dll, 11.1.0.6
    76cf0000-76d0a000: C:\WINDOWS\system32\iphlpapi.dll, 5.2.3790.3959
    62920000-6296d000: C:\oracle\product\11.1.0\client_1\bin\oranro11.dll, 11.1.0.6
    626b0000-626b9000: C:\oracle\product\11.1.0\client_1\bin\oranhost11.dll, 11.1.0.6
    0c650000-0c656000: C:\oracle\product\11.1.0\client_1\bin\orancds11.dll, 11.1.0.6
    629c0000-629cd000: C:\oracle\product\11.1.0\client_1\bin\orantns11.dll, 11.1.0.6
    0c660000-0c6e0000: C:\oracle\product\11.1.0\client_1\bin\oraztkg11.dll, 11.1.0.6
    0c6e0000-0caba000: C:\oracle\product\11.1.0\client_1\bin\orapls11.dll, 11.1.0.7
    63420000-63429000: C:\oracle\product\11.1.0\client_1\bin\oraslax11.dll, 11.1.0.6
    63080000-63296000: C:\oracle\product\11.1.0\client_1\bin\oraplp11.dll, 11.1.0.7
    0cac0000-0cb6d000: C:\oracle\product\11.1.0\client_1\bin\orahasgen11.dll, 11.1.0.6
    62ab0000-62b2f000: C:\oracle\product\11.1.0\client_1\bin\oraocr11.dll, 11.1.0.6
    0cb70000-0cbce000: C:\oracle\product\11.1.0\client_1\bin\oraocrb11.dll, 11.1.0.6
    6d580000-6d628000: C:\WINDOWS\system32\dbghelp.dll, 5.2.3790.3959
    63520000-635c2000: C:\oracle\product\11.1.0\client_1\bin\orasql11.dll, 11.1.0.7
    63060000-63074000: C:\oracle\product\11.1.0\client_1\bin\OraPlc11.Dll, 11.1.0.7
    0cde0000-0cde6000: C:\siebel\siebsrvr\BIN\sslcdbgengw.dll, 811.11.822.4
    0ce30000-0ce4c000: C:\siebel\siebsrvr\BIN\sscfomlg.dll, 811.11.822.4
    0ce60000-0cffb000: C:\siebel\siebsrvr\BIN\sscfcli.dll, 811.11.822.4
    0d070000-0d11d000: C:\siebel\siebsrvr\BIN\sscacmbc.dll, 811.11.822.4
    209c0000-20a7e000: C:\siebel\siebsrvr\BIN\SSCABCBC.DLL, 811.11.822.4
    20cc0000-20d19000: C:\siebel\siebsrvr\BIN\SSCACCBC.DLL, 811.11.822.4
    212d0000-212ec000: C:\siebel\siebsrvr\BIN\sscapsppricersvc.dll, 811.11.822.4
    20060000-2059a000: C:\siebel\siebsrvr\BIN\sscaswbc.dll, 811.11.822.4
    21690000-21704000: C:\siebel\siebsrvr\BIN\SSCMSVC.DLL, 811.11.822.4
    215a0000-2166c000: C:\siebel\siebsrvr\BIN\SSCMCLNT.DLL, 811.11.822.4
    21740000-2179e000: C:\siebel\siebsrvr\BIN\sscmcfgsvc.dll, 811.11.822.4
    0d010000-0d025000: C:\siebel\siebsrvr\BIN\SSCASRMS.DLL, 811.11.822.4
    0d030000-0d070000: C:\siebel\siebsrvr\BIN\sscaadbc.dll, 811.11.822.4
    21590000-2159d000: C:\siebel\siebsrvr\BIN\SSCFHIMS.DLL, 811.11.822.4
    22490000-224aa000: C:\siebel\siebsrvr\BIN\SSCASSMC.DLL, 811.11.822.4
    222d0000-22371000: C:\siebel\siebsrvr\BIN\sscacabc.dll, 811.11.822.4
    224b0000-224ec000: C:\siebel\siebsrvr\BIN\SSCASVBC.DLL, 811.11.822.4
    22be0000-22c24000: C:\siebel\siebsrvr\BIN\swcahcfr.dll, 811.11.822.4
    22aa0000-22b54000: C:\siebel\siebsrvr\BIN\SWCACAFR.DLL, 811.11.822.4
    22c30000-22ca0000: C:\siebel\siebsrvr\BIN\SSCAXMLPWFR.DLL, 811.11.822.4
    22ed0000-22f8c000: C:\siebel\siebsrvr\BIN\SSDTBC.DLL, 811.11.822.4
    22a80000-22a9f000: C:\siebel\siebsrvr\BIN\SSCACTBC.DLL, 811.11.822.4
    23040000-230c1000: C:\siebel\siebsrvr\BIN\SWCASIFR.DLL, 811.11.822.4
    211b0000-211cd000: C:\siebel\siebsrvr\BIN\swcacmfr.dll, 811.11.822.4
    21f60000-21f6d000: C:\siebel\siebsrvr\BIN\SSLCKM.DLL, 811.11.822.4
    22060000-220ca000: C:\siebel\siebsrvr\BIN\SSCANSHSV.DLL, 811.11.822.4
    21b20000-21b9d000: C:\siebel\siebsrvr\BIN\SWCANSHFR.DLL, 811.11.822.4
    21af0000-21b1a000: C:\siebel\siebsrvr\BIN\SSCANSHBC.DLL, 811.11.822.4
    21f80000-21fc9000: C:\siebel\siebsrvr\BIN\sWcasvfr.dll, 811.11.822.4
    22170000-22202000: C:\siebel\siebsrvr\BIN\SSFAFCBC.DLL, 811.11.822.4
    23580000-235c4000: C:\siebel\siebsrvr\BIN\SSCAPRMSV.DLL, 811.11.822.4
    22020000-2203e000: C:\siebel\siebsrvr\BIN\SSCAEITE.DLL, 811.11.822.4
    1ff20000-2005a000: C:\siebel\siebsrvr\BIN\sscafwsv.dll, 811.11.822.4
    22d40000-22ea8000: C:\siebel\siebsrvr\BIN\sscaeiwm.dll, 811.11.822.4
    22d10000-22d3e000: C:\siebel\siebsrvr\BIN\SSCAEIWO.DLL, 811.11.822.4
    23780000-2378b000: C:\siebel\siebsrvr\BIN\SSCAEILM.DLL, 811.11.822.4
    23790000-237e3000: C:\siebel\siebsrvr\BIN\SSCAEISA.DLL, 811.11.822.4
    22560000-22652000: C:\siebel\siebsrvr\BIN\SSCASPBC.DLL, 811.11.822.4
    23880000-23897000: C:\siebel\siebsrvr\BIN\sscafssvactcache.dll, 811.11.822.4
    239a0000-23a89000: C:\siebel\siebsrvr\BIN\SSCOMMSV.DLL, 811.11.822.4
    036c0000-0372a000: C:\siebel\siebsrvr\BIN\SSCHATBS.DLL, 811.11.822.4
    22770000-227c1000: C:\siebel\siebsrvr\BIN\sscasabc.dll, 811.11.822.4
    23b30000-23b4c000: C:\siebel\siebsrvr\BIN\SSFACMBC.DLL, 811.11.822.4
    23b50000-23b64000: C:\siebel\siebsrvr\BIN\SSFACMSV.DLL, 811.11.822.4
    23ea0000-23eef000: C:\siebel\siebsrvr\BIN\sscacmuinsv.dll, 811.11.822.4
    23490000-234f6000: C:\siebel\siebsrvr\BIN\SWCAFSFR.DLL, 811.11.822.4
    21a80000-21aa5000: C:\siebel\siebsrvr\BIN\SSCASHSV.DLL, 811.11.822.4
    21eb0000-21edd000: C:\siebel\siebsrvr\BIN\SSCASHBC.DLL, 811.11.822.4
    21ee0000-21f2f000: C:\siebel\siebsrvr\BIN\SWCASHFR.DLL, 811.11.822.4
    211d0000-211ee000: C:\siebel\siebsrvr\BIN\sscaxmbc.dll, 811.11.822.4
    23ef0000-23f41000: C:\siebel\siebsrvr\BIN\SWCACGFR.DLL, 811.11.822.4
    22140000-22170000: C:\siebel\siebsrvr\BIN\SSCATASV.DLL, 811.11.822.4
    21220000-212c8000: C:\siebel\siebsrvr\BIN\swcabcfr.dll, 811.11.822.4
    23bd0000-23bf4000: C:\siebel\siebsrvr\BIN\sscasasv.dll, 811.11.822.4
    23b90000-23ba6000: C:\siebel\siebsrvr\BIN\SSCAEIXQ.DLL, 811.11.822.4
    24860000-24929000: C:\siebel\siebsrvr\BIN\swcaspsv.dll, 811.11.822.4
    04080000-0408b000: C:\siebel\siebsrvr\BIN\SSCFUREC.DLL, 811.11.822.4
    Heap status          : ok
    Total physical memory: 50320576 KB
    Free physical memory : 35138076 KB
    Paging file size     : 51918740 KB
    Available paging size: 36997428 KB
    Virtual memory info:
    Total              : 2097024 KB
    Available          : 836420 KB
    Available extendent: 0 KB
    Assert failure count : 1595991
    Command line         : siebmtshmw.exe C:\siebel\siebsrvr\admin\siebprodent.pxsblaom026.shm 33 0 1 5 0 ENU 0 0
    Current directory    : C:\siebel\siebsrvr\BIN
    Executable           : C:\siebel\siebsrvr\BIN\siebmtshmw.exe
    Current time         : 01/19/2015 10:52:32:099
    CCF Version          : 8.1.1.10 SIA [23017] LANG_INDEPENDENT
    CCF Compiled On      : Jan 21 2013 18:06:00
    Would approciate if anyone could provide some inputs on the same..
    Thanks
    Ruban

    Hi,
    We recently upgraded to Oracle RAC database to support Siebel Call Center APplication (8.1.1.11.12 [IP2013]).Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    Frequently we are facing problem of crashing with the Nodes in RAC (Kernal Panic). Due to this users using siebel application are unstable and facing multiple issues Once we restore Siebel Gateway and restart of AOM server fixing the issue.
    Below are the SQL's and Crashing Signature we are seeing du to this crash.
    DBCLog DBCLogError 1 00081f1254bb0ad4:1423830 2015-01-18 11:49:20 OCIStmtExecute: SELECT
    T2.CONFLICT_ID,
    T2.LAST_UPD,
    T2.CREATED,
    T2.LAST_UPD_BY,
    T2.CREATED_BY,
    T2.MODIFICATION_NUM,
    T2.ROW_ID,
    T2.EFF_START_TS,
    T2.EFF_END_TS,
    T2.NAME,
    T2.BUSOBJ_NAME,
    T2.VERSION,
    T2.DEPLOY_STATUS_CD,
    T2.MODE_CD
    FROM
    SIEBEL.S_LST_OF_VAL T1,
    SIEBEL.S_WFA_DPLOY_DEF T2
    WHERE
    T2.DEPLOY_STATUS_CD = T1.NAME AND T1.TYPE = 'WFA_DPLY_STAT_CD' AND T1.LANG_ID = :1 AND
    (T2.TYPE_CD IN ( :2 )) AND
    (T2.NAME = :3 AND T2.DEPLOY_STATUS_CD IN ( :4 ) AND (T2.EFF_END_TS > TO_DATE(:5,'MM/DD/YYYY HH24:MI:SS') OR T2.EFF_END_TS IS NULL))
    ORDER BY
    T2.NAME, T1.VAL
    ObjMgrLog Error 1 00081f1254bb0ad4:1423830 2015-01-18 11:49:20 (oracon.cpp (3896)) SBL-DBC-00105: An error has occurred executing a Sql statement.
    Please continue or ask your systems administrator to check your application configuration if the problem persists.
    and Crash Signature Looks like below
    Exception 0xc0000005 at 0x006500f6
    Thread: 0x00002700, Process 0x00000254
    Thread: 9984, Process 596
    - CONTEXT -
    EIP: 0x00000000, EFL: 0x00010246, FS:  0x0000003b, GS:  0x00000000
    CS:  0x0000001b, DS:  0x00000023, SS:  0x00000023, ES:  0x00000023
    EAX: 0x1a511968, EBX: 0x294ca048, ECX: 0x4cc50008, EDX: 0x3a2ad200
    ESI: 0x4cc50008, EDI: 0x00000000, EBP: 0x00000000, ESP: 0x3a2afff4
    - CALL STACK -
    sslcscr +0xf6
    sscfcmn +0x19050 = CSSObjectBase::CopyErrorMsg() +0xf0
    sscfcmn +0x1950d = CSSObjectBase::DoSetErrorMsg() +0x13d
    sscfcmn +0x19bb5 = CSSObjectBase::SetErrorMsg() +0xa5
    SSFACMBC +0x34a6 = CSSFABCVRec::IsBusinessProcessActive() +0x3b6
    SSFACMBC +0x5133 = CSSFABCVRec::VBCCreate() +0x1e3
    sscfom +0x71d1e = CSSBCVRec::SqlCreate() +0xde
    sscfom +0x4aea = CSSModel::ConstructBusComp() +0xaa
    sscfom +0x653f5 = CSSBusObj::GetBusComp() +0x265
    sscfom +0x65a18 = CSSBusObj::GetBusComp() +0x18
    sscfom +0x13ad8 = CSSModel::BeginSetupView() +0x418
    sscfom +0x13dee = CSSModel::BeginSetupView() +0x5e
    sscaswbc +0x8b070 = CSSSWEUdfLoader::BeginSetupView() +0x30
    sscaswbc +0xe3d88 = CSSSWEFrameMgrInternal::BuildView() +0x1618
    sscaswbc +0xe5646 = CSSSWEFrameMgrInternal::BuildView() +0x36
    sscaswbc +0xb8343 = CSSSWEActionMgr::ActionBuildViewAsync() +0x293
    sscaswbc +0xb873b = CSSSWEActionMgr::DoPostedAction() +0xab
    sscaswbc +0xb9287 = CSSSWEActionMgr::CheckPostedActions() +0x3d7
    sscaswbc +0xe6d41 = CSSSWEFrameMgrInternal::InvokeAppletMethod() +0x11b1
    sscaswbc +0xbbcdc = CSSSWECmdProcessor::InvokeMethod() +0x74c
    sscaswbc +0xc1495 = CSSSWECmdProcessor::_ProcessCommand() +0x725
    sscaswbc +0xc27f8 = CSSSWECmdProcessor::ProcessCommand() +0xab8
    sscaswbc +0xc2d04 = CSSSWECmdProcessor::ProcessCommand() +0xc4
    sscaswbc +0x265559 = CSSServiceSWEIface::Request() +0x309
    sscaswbc +0x267391 = CSSServiceSWEIface::DoInvokeMethod() +0xed1
    sscfom +0x1976c = CSSService::InvokeMethod() +0x24c
    sstcsiom +0x6d6a = CSSSIOMSession::ModInvokeSrvcMethod() +0x10a
    sstcsiom +0x7613 = CSSSIOMSession::RPCMiscModel() +0x513
    sstcsiom +0xe9dd = CSSSIOMSession::HandleRPC() +0x3ad
    sssasos +0x6616 = CompCleanup() +0x3be6
    sssasos +0x5a95 = CompCleanup() +0x3065
    sssasos +0x1a35 = CSSSISObject::operator=() +0x9f5
    sssasos +0x2135 = CompHandleMsg() +0x485
    siebmtshmw +0x494e
    siebmtshmw +0x20051 = GetSmiTaskAPI() +0x6d21
    siebmtshmw +0x239d1 = SmiInProcMsgHandler() +0x2051
    siebmtshmw +0x15f15 = SmiBeginTrace() +0x3195
    siebmtshmw +0x174cf = GetSmiWorkQObj() +0x148f
    siebmtshmw +0xc7f7 = SmiCleanupDetTask() +0x3be7
    sslcosd +0x2152 = OSDThreadPrivIsInit() +0x1d2
    sslcosd +0x21dc = OSDThreadPrivIsInit() +0x25c
    MSVCR71 +0x9565 = endthreadex() +0xa0
    kernel32 +0x2484f = GetModuleHandleA() +0xdf
    - STACK DUMP -
    - MODULE LIST -
    00400000-00445000: C:\siebel\siebsrvr\BIN\siebmtshmw.exe, 811.11.822.4
    7c800000-7c8c3000: C:\WINDOWS\system32\ntdll.dll, 5.2.3790.4937
    77e40000-77f44000: C:\WINDOWS\system32\kernel32.dll, 5.2.3790.5295
    10000000-10025000: C:\siebel\siebsrvr\BIN\ssscsci.dll, 811.11.822.4
    00320000-003cc000: C:\siebel\siebsrvr\BIN\ssscscf.dll, 811.11.822.4
    003d0000-003d7000: C:\siebel\siebsrvr\BIN\sslcosa.dll, 811.11.822.4
    7c340000-7c396000: C:\siebel\siebsrvr\BIN\MSVCR71.dll, 7.10.3052.4
    00450000-0054d000: C:\siebel\siebsrvr\BIN\sslcshar.dll, 811.11.822.4
    77380000-77411000: C:\WINDOWS\system32\USER32.dll, 5.2.3790.4033
    77c00000-77c49000: C:\WINDOWS\system32\GDI32.dll, 5.2.3790.5398
    7d1e0000-7d27c000: C:\WINDOWS\system32\ADVAPI32.dll, 5.2.3790.4455
    77c50000-77cf1000: C:\WINDOWS\system32\RPCRT4.dll, 5.2.3790.5254
    76f50000-76f63000: C:\WINDOWS\system32\Secur32.dll, 5.2.3790.4455
    77b90000-77b98000: C:\WINDOWS\system32\VERSION.dll, 5.2.3790.3959
    77ba0000-77bfa000: C:\WINDOWS\system32\msvcrt.dll, 7.0.3790.3959
    71bb0000-71bb9000: C:\WINDOWS\system32\WSOCK32.dll, 5.2.3790.0
    71c00000-71c17000: C:\WINDOWS\system32\WS2_32.dll, 5.2.3790.3959
    71bf0000-71bf8000: C:\WINDOWS\system32\WS2HELP.dll, 5.2.3790.3959
    003e0000-003e6000: C:\siebel\siebsrvr\BIN\libarm.dll, 811.10.822.3
    00550000-005d5000: C:\siebel\siebsrvr\BIN\sslcsym.dll, 811.11.822.4
    7c3a0000-7c41b000: C:\siebel\siebsrvr\BIN\MSVCP71.dll, 7.10.3077.0
    005e0000-005f1000: C:\siebel\siebsrvr\BIN\sslccore.dll, 811.11.822.4
    00600000-00618000: C:\siebel\siebsrvr\BIN\sslcosd.dll, 811.11.822.4
    00620000-00645000: C:\siebel\siebsrvr\BIN\sslcrsa.dll, 811.11.822.4
    00650000-0067f000: C:\siebel\siebsrvr\BIN\sslcscr.dll, 811.11.822.4
    00680000-007cf000: C:\siebel\siebsrvr\BIN\sslcsrd.dll, 811.11.822.4
    007d0000-0082b000: C:\siebel\siebsrvr\BIN\sslcscc.dll, 811.11.822.4
    003f0000-003ff000: C:\siebel\siebsrvr\BIN\sslcevt.dll, 811.11.822.4
    00830000-00840000: C:\siebel\siebsrvr\BIN\sslcsobj.dll, 811.11.822.4
    00840000-00850000: C:\siebel\siebsrvr\BIN\sslcsnns.dll, 811.11.822.4
    00850000-00931000: C:\siebel\siebsrvr\BIN\sslcnapi.dll, 811.11.822.4
    00940000-00954000: C:\siebel\siebsrvr\BIN\sslcns.dll, 811.11.822.4
    00960000-00968000: C:\siebel\siebsrvr\BIN\sslcsnsc.dll, 811.11.822.4
    00970000-0097b000: C:\siebel\siebsrvr\BIN\ssscsmi.dll, 811.11.822.4
    00980000-00991000: C:\siebel\siebsrvr\BIN\sslcsrvr.dll, 811.11.822.4
    009a0000-009bb000: C:\siebel\siebsrvr\BIN\sslcos.dll, 811.11.822.4
    009c0000-00a5f000: C:\siebel\siebsrvr\BIN\sscfcmn.dll, 811.11.822.4
    7c8d0000-7d0cf000: C:\WINDOWS\system32\SHELL32.dll, 6.0.3790.5018
    7d180000-7d1d2000: C:\WINDOWS\system32\SHLWAPI.dll, 6.0.3790.5318
    77670000-777aa000: C:\WINDOWS\system32\ole32.dll, 5.2.3790.5209
    7d0e0000-7d16c000: C:\WINDOWS\system32\OLEAUT32.dll, 5.2.3790.5464
    761b0000-76245000: C:\WINDOWS\system32\CRYPT32.dll, 5.131.3790.5235
    76190000-761a2000: C:\WINDOWS\system32\MSASN1.dll, 5.2.3790.4584
    77210000-772bb000: C:\WINDOWS\system32\WININET.dll, 6.0.3790.5467
    00a60000-00a70000: C:\siebel\siebsrvr\BIN\sslczlib.dll, 811.11.822.4
    00a70000-00b72000: C:\siebel\siebsrvr\BIN\MFC71U.DLL, 7.10.3077.0
    00b80000-00b88000: C:\siebel\siebsrvr\BIN\sslcsnsr.dll, 811.11.822.4
    00b90000-00b99000: C:\siebel\siebsrvr\BIN\sslcsrtr.dll, 811.11.822.4
    00ba0000-00baf000: C:\siebel\siebsrvr\BIN\sslcsrms.dll, 811.11.822.4
    00bb0000-00bb6000: C:\siebel\siebsrvr\BIN\sslcsssm.dll, 811.11.822.4
    00bc0000-00bd1000: C:\siebel\siebsrvr\BIN\sslcsrcn.dll, 811.11.822.4
    00be0000-00be7000: C:\siebel\siebsrvr\BIN\sslcsysstat.dll, 811.11.822.4
    00fe0000-0100d000: C:\siebel\siebsrvr\BIN\sslcrsa256.dll, 8.1.1.5
    77420000-77523000: C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.5190_x-ww_319264BE\comctl32.dll, 6.0.3790.5190
    5d360000-5d36e000: C:\siebel\siebsrvr\BIN\MFC71ENU.DLL, 7.10.3077.0
    76b70000-76b7b000: C:\WINDOWS\system32\psapi.dll, 5.2.3790.3959
    012b0000-012c0000: C:\siebel\siebsrvr\BIN\sslcasrv.dll, 811.11.822.4
    012c0000-01328000: C:\siebel\siebsrvr\BIN\sslcsrs.dll, 811.11.822.4
    01330000-0133a000: C:\siebel\siebsrvr\BIN\sslcntfy.dll, 811.11.822.4
    01740000-01746000: C:\siebel\siebsrvr\BIN\enu\srcf.dll, 811.9.822.2
    01750000-01756000: C:\siebel\siebsrvr\BIN\enu\srcfapps.dll, 811.9.822.2
    01760000-01766000: C:\siebel\siebsrvr\BIN\sbcf.dll, 811.9.822.2
    01770000-01776000: C:\siebel\siebsrvr\BIN\sbcfapps.dll, 811.9.822.2
    01780000-01791000: C:\siebel\siebsrvr\BIN\sssasos.dll, 811.11.822.4
    017a0000-017a8000: C:\siebel\siebsrvr\BIN\sslcom.dll, 811.11.822.4
    017b0000-017be000: C:\siebel\siebsrvr\BIN\sslcsnom.dll, 811.11.822.4
    017c0000-017f8000: C:\siebel\siebsrvr\BIN\sstcsiom.dll, 811.11.822.4
    020f0000-021f0000: C:\siebel\siebsrvr\BIN\sscfom.dll, 811.11.822.4
    021f0000-022ef000: C:\siebel\siebsrvr\BIN\sslcsc.dll, 811.11.822.4
    022f0000-0236f000: C:\siebel\siebsrvr\BIN\sslcdb.dll, 811.11.822.4
    4bf70000-4bfae000: C:\WINDOWS\system32\ODBC32.dll, 3.526.4795.0
    77530000-775c7000: C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_5.82.3790.5190_x-ww_D21E1F39\COMCTL32.dll, 5.82.3790.5190
    762b0000-762f9000: C:\WINDOWS\system32\comdlg32.dll, 6.0.3790.3959
    01800000-0180b000: C:\siebel\siebsrvr\BIN\sscadsecr.dll, 811.11.822.4
    02370000-0237d000: C:\siebel\siebsrvr\BIN\sslclm.dll, 811.11.822.4
    02380000-0239c000: C:\siebel\siebsrvr\BIN\sslcupg.dll, 811.11.822.4
    023a0000-023ba000: C:\siebel\siebsrvr\BIN\sslcsecm.dll, 811.11.822.4
    023c0000-023ef000: C:\siebel\siebsrvr\BIN\sscfel.dll, 811.11.822.4
    023f0000-026b0000: C:\siebel\siebsrvr\BIN\sscfdm.dll, 811.11.822.4
    026b0000-026bd000: C:\siebel\siebsrvr\BIN\sslcfsm.dll, 811.11.822.4
    026c0000-026c8000: C:\siebel\siebsrvr\BIN\sslcsecc.dll, 811.11.822.4
    026d0000-026e1000: C:\siebel\siebsrvr\BIN\sslccachev2.dll, 811.11.822.4
    026f0000-026f9000: C:\siebel\siebsrvr\BIN\sslcmsgagent.dll, 811.11.822.4
    02bd0000-02be7000: C:\WINDOWS\system32\odbcint.dll, 3.526.3959.0
    02bf0000-02c23000: C:\siebel\siebsrvr\BIN\sslcrsa56.dll, 811.11.822.4
    71b20000-71b61000: C:\WINDOWS\System32\mswsock.dll, 5.2.3790.4318
    76ed0000-76efa000: C:\WINDOWS\system32\DNSAPI.dll, 5.2.3790.4840
    76f70000-76f77000: C:\WINDOWS\System32\winrnr.dll, 5.2.3790.3959
    76f10000-76f3e000: C:\WINDOWS\system32\WLDAP32.dll, 5.2.3790.3959
    76f80000-76f85000: C:\WINDOWS\system32\rasadhlp.dll, 5.2.3790.3959
    5f270000-5f2ca000: C:\WINDOWS\system32\hnetcfg.dll, 5.2.3790.3959
    71ae0000-71ae8000: C:\WINDOWS\System32\wshtcpip.dll, 5.2.3790.3959
    0ae80000-0af6a000: C:\siebel\siebsrvr\BIN\eBaySecAda.dll
    76920000-769e2000: C:\WINDOWS\system32\userenv.dll, 5.2.3790.3959
    71c40000-71c97000: C:\WINDOWS\system32\netapi32.dll, 5.2.3790.5030
    76bb0000-76bdc000: C:\WINDOWS\system32\wintrust.dll, 5.131.3790.5060
    76c10000-76c3a000: C:\WINDOWS\system32\imagehlp.dll, 5.2.3790.5240
    76750000-76779000: C:\WINDOWS\system32\schannel.dll, 5.2.3790.5462
    76e90000-76ecf000: C:\WINDOWS\system32\RASAPI32.DLL, 5.2.3790.3959
    76e40000-76e52000: C:\WINDOWS\system32\rasman.dll, 5.2.3790.3959
    76e60000-76e8f000: C:\WINDOWS\system32\TAPI32.dll, 5.2.3790.3959
    76e30000-76e3c000: C:\WINDOWS\system32\rtutils.dll, 5.2.3790.3959
    76aa0000-76acd000: C:\WINDOWS\system32\WINMM.dll, 5.2.3790.4916
    722f0000-722f5000: C:\WINDOWS\system32\sensapi.dll, 5.2.3790.3959
    772c0000-77371000: C:\WINDOWS\system32\urlmon.dll, 6.0.3790.5467
    68000000-68035000: C:\WINDOWS\system32\rsaenh.dll, 5.2.3790.3959
    68100000-68127000: C:\WINDOWS\system32\dssenh.dll, 5.2.3790.3959
    0b300000-0b31a000: C:\siebel\siebsrvr\BIN\sscdo90.dll, 811.11.822.4
    0b320000-0b3a2000: C:\oracle\product\11.1.0\client_1\bin\OCI.dll, 11.1.0.1
    61c20000-61f81000: C:\oracle\product\11.1.0\client_1\bin\OraClient11.Dll, 11.1.0.7
    60870000-6095b000: C:\oracle\product\11.1.0\client_1\bin\oracore11.dll, 11.1.0.7
    60a80000-60b4e000: C:\oracle\product\11.1.0\client_1\bin\oranls11.dll, 11.1.0.7
    63690000-636a8000: C:\oracle\product\11.1.0\client_1\bin\oraunls11.dll, 11.1.0.7
    60eb0000-60eb7000: C:\oracle\product\11.1.0\client_1\bin\orauts.dll, 11.1.0.7
    636b0000-636b6000: C:\oracle\product\11.1.0\client_1\bin\oravsn11.dll, 11.1.0.7
    60fa0000-61158000: C:\oracle\product\11.1.0\client_1\bin\oracommon11.dll, 11.1.0.7
    63430000-6345d000: C:\oracle\product\11.1.0\client_1\bin\orasnls11.dll, 11.1.0.7
    0b3b0000-0be77000: C:\oracle\product\11.1.0\client_1\bin\orageneric11.dll, 11.1.0.7
    0be80000-0c0dc000: C:\oracle\product\11.1.0\client_1\bin\oraxml11.dll, 11.1.0.7
    60020000-60030000: C:\WINDOWS\system32\MSVCIRT.dll, 7.0.3790.0
    0c0e0000-0c4b2000: C:\oracle\product\11.1.0\client_1\bin\oran11.dll, 11.1.0.6
    62740000-627a3000: C:\oracle\product\11.1.0\client_1\bin\oranl11.dll, 11.1.0.6
    62980000-629b1000: C:\oracle\product\11.1.0\client_1\bin\orantcp11.dll, 11.1.0.6
    0c4c0000-0c507000: C:\oracle\product\11.1.0\client_1\bin\oranldap11.dll, 11.1.0.6
    627f0000-62905000: C:\oracle\product\11.1.0\client_1\bin\orannzsbb11.dll, 11.1.0.6
    62530000-6266d000: C:\oracle\product\11.1.0\client_1\bin\oraldapclnt11.dll, 11.1.0.7
    62670000-6268b000: C:\oracle\product\11.1.0\client_1\bin\orancrypt11.dll, 11.1.0.6
    0c510000-0c643000: C:\oracle\product\11.1.0\client_1\bin\orazt11.dll, 11.1.0.6
    76cf0000-76d0a000: C:\WINDOWS\system32\iphlpapi.dll, 5.2.3790.3959
    62920000-6296d000: C:\oracle\product\11.1.0\client_1\bin\oranro11.dll, 11.1.0.6
    626b0000-626b9000: C:\oracle\product\11.1.0\client_1\bin\oranhost11.dll, 11.1.0.6
    0c650000-0c656000: C:\oracle\product\11.1.0\client_1\bin\orancds11.dll, 11.1.0.6
    629c0000-629cd000: C:\oracle\product\11.1.0\client_1\bin\orantns11.dll, 11.1.0.6
    0c660000-0c6e0000: C:\oracle\product\11.1.0\client_1\bin\oraztkg11.dll, 11.1.0.6
    0c6e0000-0caba000: C:\oracle\product\11.1.0\client_1\bin\orapls11.dll, 11.1.0.7
    63420000-63429000: C:\oracle\product\11.1.0\client_1\bin\oraslax11.dll, 11.1.0.6
    63080000-63296000: C:\oracle\product\11.1.0\client_1\bin\oraplp11.dll, 11.1.0.7
    0cac0000-0cb6d000: C:\oracle\product\11.1.0\client_1\bin\orahasgen11.dll, 11.1.0.6
    62ab0000-62b2f000: C:\oracle\product\11.1.0\client_1\bin\oraocr11.dll, 11.1.0.6
    0cb70000-0cbce000: C:\oracle\product\11.1.0\client_1\bin\oraocrb11.dll, 11.1.0.6
    6d580000-6d628000: C:\WINDOWS\system32\dbghelp.dll, 5.2.3790.3959
    63520000-635c2000: C:\oracle\product\11.1.0\client_1\bin\orasql11.dll, 11.1.0.7
    63060000-63074000: C:\oracle\product\11.1.0\client_1\bin\OraPlc11.Dll, 11.1.0.7
    0cde0000-0cde6000: C:\siebel\siebsrvr\BIN\sslcdbgengw.dll, 811.11.822.4
    0ce30000-0ce4c000: C:\siebel\siebsrvr\BIN\sscfomlg.dll, 811.11.822.4
    0ce60000-0cffb000: C:\siebel\siebsrvr\BIN\sscfcli.dll, 811.11.822.4
    0d070000-0d11d000: C:\siebel\siebsrvr\BIN\sscacmbc.dll, 811.11.822.4
    209c0000-20a7e000: C:\siebel\siebsrvr\BIN\SSCABCBC.DLL, 811.11.822.4
    20cc0000-20d19000: C:\siebel\siebsrvr\BIN\SSCACCBC.DLL, 811.11.822.4
    212d0000-212ec000: C:\siebel\siebsrvr\BIN\sscapsppricersvc.dll, 811.11.822.4
    20060000-2059a000: C:\siebel\siebsrvr\BIN\sscaswbc.dll, 811.11.822.4
    21690000-21704000: C:\siebel\siebsrvr\BIN\SSCMSVC.DLL, 811.11.822.4
    215a0000-2166c000: C:\siebel\siebsrvr\BIN\SSCMCLNT.DLL, 811.11.822.4
    21740000-2179e000: C:\siebel\siebsrvr\BIN\sscmcfgsvc.dll, 811.11.822.4
    0d010000-0d025000: C:\siebel\siebsrvr\BIN\SSCASRMS.DLL, 811.11.822.4
    0d030000-0d070000: C:\siebel\siebsrvr\BIN\sscaadbc.dll, 811.11.822.4
    21590000-2159d000: C:\siebel\siebsrvr\BIN\SSCFHIMS.DLL, 811.11.822.4
    22490000-224aa000: C:\siebel\siebsrvr\BIN\SSCASSMC.DLL, 811.11.822.4
    222d0000-22371000: C:\siebel\siebsrvr\BIN\sscacabc.dll, 811.11.822.4
    224b0000-224ec000: C:\siebel\siebsrvr\BIN\SSCASVBC.DLL, 811.11.822.4
    22be0000-22c24000: C:\siebel\siebsrvr\BIN\swcahcfr.dll, 811.11.822.4
    22aa0000-22b54000: C:\siebel\siebsrvr\BIN\SWCACAFR.DLL, 811.11.822.4
    22c30000-22ca0000: C:\siebel\siebsrvr\BIN\SSCAXMLPWFR.DLL, 811.11.822.4
    22ed0000-22f8c000: C:\siebel\siebsrvr\BIN\SSDTBC.DLL, 811.11.822.4
    22a80000-22a9f000: C:\siebel\siebsrvr\BIN\SSCACTBC.DLL, 811.11.822.4
    23040000-230c1000: C:\siebel\siebsrvr\BIN\SWCASIFR.DLL, 811.11.822.4
    211b0000-211cd000: C:\siebel\siebsrvr\BIN\swcacmfr.dll, 811.11.822.4
    21f60000-21f6d000: C:\siebel\siebsrvr\BIN\SSLCKM.DLL, 811.11.822.4
    22060000-220ca000: C:\siebel\siebsrvr\BIN\SSCANSHSV.DLL, 811.11.822.4
    21b20000-21b9d000: C:\siebel\siebsrvr\BIN\SWCANSHFR.DLL, 811.11.822.4
    21af0000-21b1a000: C:\siebel\siebsrvr\BIN\SSCANSHBC.DLL, 811.11.822.4
    21f80000-21fc9000: C:\siebel\siebsrvr\BIN\sWcasvfr.dll, 811.11.822.4
    22170000-22202000: C:\siebel\siebsrvr\BIN\SSFAFCBC.DLL, 811.11.822.4
    23580000-235c4000: C:\siebel\siebsrvr\BIN\SSCAPRMSV.DLL, 811.11.822.4
    22020000-2203e000: C:\siebel\siebsrvr\BIN\SSCAEITE.DLL, 811.11.822.4
    1ff20000-2005a000: C:\siebel\siebsrvr\BIN\sscafwsv.dll, 811.11.822.4
    22d40000-22ea8000: C:\siebel\siebsrvr\BIN\sscaeiwm.dll, 811.11.822.4
    22d10000-22d3e000: C:\siebel\siebsrvr\BIN\SSCAEIWO.DLL, 811.11.822.4
    23780000-2378b000: C:\siebel\siebsrvr\BIN\SSCAEILM.DLL, 811.11.822.4
    23790000-237e3000: C:\siebel\siebsrvr\BIN\SSCAEISA.DLL, 811.11.822.4
    22560000-22652000: C:\siebel\siebsrvr\BIN\SSCASPBC.DLL, 811.11.822.4
    23880000-23897000: C:\siebel\siebsrvr\BIN\sscafssvactcache.dll, 811.11.822.4
    239a0000-23a89000: C:\siebel\siebsrvr\BIN\SSCOMMSV.DLL, 811.11.822.4
    036c0000-0372a000: C:\siebel\siebsrvr\BIN\SSCHATBS.DLL, 811.11.822.4
    22770000-227c1000: C:\siebel\siebsrvr\BIN\sscasabc.dll, 811.11.822.4
    23b30000-23b4c000: C:\siebel\siebsrvr\BIN\SSFACMBC.DLL, 811.11.822.4
    23b50000-23b64000: C:\siebel\siebsrvr\BIN\SSFACMSV.DLL, 811.11.822.4
    23ea0000-23eef000: C:\siebel\siebsrvr\BIN\sscacmuinsv.dll, 811.11.822.4
    23490000-234f6000: C:\siebel\siebsrvr\BIN\SWCAFSFR.DLL, 811.11.822.4
    21a80000-21aa5000: C:\siebel\siebsrvr\BIN\SSCASHSV.DLL, 811.11.822.4
    21eb0000-21edd000: C:\siebel\siebsrvr\BIN\SSCASHBC.DLL, 811.11.822.4
    21ee0000-21f2f000: C:\siebel\siebsrvr\BIN\SWCASHFR.DLL, 811.11.822.4
    211d0000-211ee000: C:\siebel\siebsrvr\BIN\sscaxmbc.dll, 811.11.822.4
    23ef0000-23f41000: C:\siebel\siebsrvr\BIN\SWCACGFR.DLL, 811.11.822.4
    22140000-22170000: C:\siebel\siebsrvr\BIN\SSCATASV.DLL, 811.11.822.4
    21220000-212c8000: C:\siebel\siebsrvr\BIN\swcabcfr.dll, 811.11.822.4
    23bd0000-23bf4000: C:\siebel\siebsrvr\BIN\sscasasv.dll, 811.11.822.4
    23b90000-23ba6000: C:\siebel\siebsrvr\BIN\SSCAEIXQ.DLL, 811.11.822.4
    24860000-24929000: C:\siebel\siebsrvr\BIN\swcaspsv.dll, 811.11.822.4
    04080000-0408b000: C:\siebel\siebsrvr\BIN\SSCFUREC.DLL, 811.11.822.4
    Heap status          : ok
    Total physical memory: 50320576 KB
    Free physical memory : 35138076 KB
    Paging file size     : 51918740 KB
    Available paging size: 36997428 KB
    Virtual memory info:
    Total              : 2097024 KB
    Available          : 836420 KB
    Available extendent: 0 KB
    Assert failure count : 1595991
    Command line         : siebmtshmw.exe C:\siebel\siebsrvr\admin\siebprodent.pxsblaom026.shm 33 0 1 5 0 ENU 0 0
    Current directory    : C:\siebel\siebsrvr\BIN
    Executable           : C:\siebel\siebsrvr\BIN\siebmtshmw.exe
    Current time         : 01/19/2015 10:52:32:099
    CCF Version          : 8.1.1.10 SIA [23017] LANG_INDEPENDENT
    CCF Compiled On      : Jan 21 2013 18:06:00
    Would approciate if anyone could provide some inputs on the same..
    Thanks
    Ruban

  • Oracle Rac Node 1 , Node 2

    *Hi All ,
    I need to ask about RAC :
    I have Oracle RAC ( Node1,Node2) When i check The performance For both nodes , the all load Goes to node 1 without anything happened on node 2 , as i know the the load must be equal on the both Nodes ( node1 , node2 ).
    How Can i do this ? to make the load for both nodes equal.
    Thank you very much.*

    Thank you Guys
    My Load balance Configuration like the following :
    Node 1 "tnsname":
    MARKADB_SRV =
    (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 172.29.11.23)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = 172.29.11.22)(PORT = 1521))
    (LOAD_BALANCE = yes)
    (CONNECT_DATA =
    (SERVER = DEDICATED)
    (SERVICE_NAME = markadb_srv)
    (FAILOVER_MODE =
    (TYPE = SELECT)
    (METHOD = BASIC)
    (RETRIES = 180)
    (DELAY = 5)
    Node 1 "Listener"
    LISTENER_MARKADB1 =
    (DESCRIPTION_LIST =
    (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 172.29.11.22)(PORT = 1521))
    (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 172.29.11.20)(PORT = 1521))
    Node 2 "Tnsname" :
    MARKADB_SRV =
    (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 172.29.11.22)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = 172.29.11.23)(PORT = 1521))
    (LOAD_BALANCE = yes)
    (CONNECT_DATA =
    (SERVER = DEDICATED)
    (SERVICE_NAME = markadb_srv)
    (FAILOVER_MODE =
    (TYPE = SELECT)
    (METHOD = BASIC)
    (RETRIES = 180)
    (DELAY = 5)
    Node 2 "Listener"
    LISTENER_MARKADB2 =
    (DESCRIPTION_LIST =
    (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 172.29.11.23)(PORT = 1521))
    (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 172.29.11.21)(PORT = 1521))
    But all traffic goes to node 1 , and when node 1 goes down its not converting to node 2 as we all know but to another site /Failover ( I got Oracle Stream )
    Any Solution Guys ?

  • Delete oracle RAC node - still appear on addNode script

    Hello ,
    I delete one from oracle RAC have 3 node .
    I try to re-install it again but it give me below error :
    The following error was encountered for new node . the node names rac3 are unique ...
    it still appear on add node ! why ?
    [oracle@rac1 dev]$ /u01/crs/oracle/product/10.2.0/crs/bin/crs_stat -t
    Name           Type           Target    State     Host       
    ora....C1.inst application    ONLINE    OFFLINE              
    ora....C2.inst application    ONLINE    OFFLINE              
    ora.RAC.db     application    ONLINE    OFFLINE              
    ora....SM1.asm application    ONLINE    ONLINE    rac1       
    ora....C1.lsnr application    ONLINE    OFFLINE              
    ora.rac1.gsd   application    ONLINE    OFFLINE              
    ora.rac1.ons   application    ONLINE    ONLINE    rac1       
    ora.rac1.vip   application    ONLINE    ONLINE    rac1       
    ora....SM2.asm application    ONLINE    OFFLINE              
    ora....C2.lsnr application    ONLINE    OFFLINE              
    ora.rac2.gsd   application    ONLINE    OFFLINE              
    ora.rac2.ons   application    ONLINE    OFFLINE              
    ora.rac2.vip   application    ONLINE    ONLINE    rac1
    Can you help please how to solve this issue  ?
    Thanks

    Hello ,
    Finally done
    /u01/crs/oracle/product/10.2.0/crs/install/rootdeletenode.sh rac3,3
    /u01/crs/oracle/product/10.2.0/crs/bin/olsnodes -n now it done appear
    but still same error appear .......
    so after
    /u01/crs/oracle/product/10.2.0/crs/oui/bin/runInstaller -updateNodelist ORACLE_HOME=/u01/crs/oracle/product/10.2.0/crs/ "CLUSTER_NODES={rac1,rac2}" CRS=true
      445  locate racgons
      446  /u01/crs/oracle/product/10.2.0/crs/bin/racgons remove_config rac3
    Regards,
    Mohanad Awad

  • Oracle RAC node recovery

    Hi,All,
    Oracle 10.2.0.4 soalris 10 sun cluster. 2 nodes
    Node 1's OS system and Oracle software was delete by mis-action.
    how can I recovery Node 1 after re-install OS.
    Thanks
    Simon Lai

    Nodes are not databases.
    Instances are not databases.
    Databases are recovered not memory processes and caches.

  • RAC node rebooting frequently

    Hi all,
    I am woserking on two node rac environment.One of my rac node is rebooting so frequently.I am using oracle 10g database and clusterware also(10.2.0.1).
    Ihave checked os logs(linux AS 4),and rac related logs.Not able to find out anything.Posting all logs please suggest.

    Hi i am posting alert log,os log and ocssd logs....
    clusterware alert log....._
    [crsd(5649)]CRS-1201:CRSD started on node ctmisdb1.
    2012-03-21 09:50:38.188
    [cssd(7490)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ctmisdb1 .
    2012-03-21 09:50:46.726
    [crsd(5649)]CRS-1204:Recovering CRS resources for node ctmisdb2.
    2012-03-21 09:55:21.760
    [cssd(7490)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ctmisdb1 ctmisdb2 .
    2012-03-21 12:07:46.681
    [cssd(7426)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /u01/app/oracle/product/crs/log/ctmisdb1/cssd/ocssd.log.
    2012-03-21 12:07:50.432
    [cssd(7426)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ctmisdb1 ctmisdb2 .
    2012-03-21 12:07:50.893
    [crsd(5549)]CRS-1012:The OCR service started on node ctmisdb1.
    2012-03-21 12:07:50.942
    [evmd(7304)]CRS-1401:EVMD started on node ctmisdb1.
    2012-03-21 12:07:52.827
    [crsd(5549)]CRS-1201:CRSD started on node ctmisdb1.
    2012-03-21 12:48:41.908
    [cssd(7448)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /u01/app/oracle/product/crs/log/ctmisdb1/cssd/ocssd.log.
    2012-03-21 12:48:45.741
    [cssd(7448)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ctmisdb1 ctmisdb2 .
    2012-03-21 12:48:49.173
    [crsd(5546)]CRS-1012:The OCR service started on node ctmisdb1.
    2012-03-21 12:48:49.190
    [evmd(7328)]CRS-1401:EVMD started on node ctmisdb1.
    2012-03-21 12:48:50.818
    [crsd(5546)]CRS-1201:CRSD started on node ctmisdb1.
    2012-03-21 13:26:36.398
    [cssd(7343)]CRS-1605:CSSD voting file is online: /dev/raw/raw2. Details in /u01/app/oracle/product/crs/log/ctmisdb1/cssd/ocssd.log.
    2012-03-21 13:26:40.492
    [cssd(7343)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ctmisdb1 ctmisdb2 .
    2012-03-21 13:26:40.939
    [crsd(5542)]CRS-1012:The OCR service started on node ctmisdb1.
    2012-03-21 13:26:40.977
    [evmd(7223)]CRS-1401:EVMD started on node ctmisdb1.
    2012-03-21 13:26:42.772
    [crsd(5542)]CRS-1201:CRSD started on node ctmisdb1.
    node os log....+
    Mar 21 12:06:35 ctmisdb1 rc: Starting readahead: succeeded
    Mar 21 12:06:35 ctmisdb1 messagebus: messagebus startup succeeded
    Mar 21 12:06:36 ctmisdb1 cups-config-daemon: cups-config-daemon startup succeeded
    Mar 21 12:06:36 ctmisdb1 haldaemon: haldaemon startup succeeded
    Mar 21 12:06:37 ctmisdb1 fstab-sync[6267]: removed all generated mount points
    Mar 21 12:06:37 ctmisdb1 fstab-sync[6378]: added mount point /media/cdrecorder for /dev/hde
    Mar 21 12:06:37 ctmisdb1 su(pam_unix)[6323]: session opened for user oracle by (uid=0)
    Mar 21 12:06:37 ctmisdb1 su(pam_unix)[6324]: session opened for user oracle by (uid=0)
    Mar 21 12:06:37 ctmisdb1 su(pam_unix)[6229]: session opened for user oracle by (uid=0)
    Mar 21 12:06:37 ctmisdb1 su(pam_unix)[6229]: session closed for user oracle
    Mar 21 12:06:37 ctmisdb1 su(pam_unix)[6644]: session opened for user oracle by (uid=0)
    Mar 21 12:06:37 ctmisdb1 kernel: matroxfb: cannot set xres to 800, rounded up to 832
    Mar 21 12:06:37 ctmisdb1 last message repeated 2 times
    Mar 21 12:06:41 ctmisdb1 su(pam_unix)[6323]: session closed for user oracle
    Mar 21 12:06:41 ctmisdb1 su(pam_unix)[6644]: session closed for user oracle
    Mar 21 12:06:41 ctmisdb1 su(pam_unix)[6324]: session closed for user oracle
    Mar 21 12:06:41 ctmisdb1 logger: Cluster Ready Services completed waiting on dependencies.
    Mar 21 12:06:41 ctmisdb1 last message repeated 2 times
    Mar 21 12:06:45 ctmisdb1 gdm(pam_unix)[6379]: session opened for user root by (uid=0)
    Mar 21 12:06:46 ctmisdb1 gconfd (root-7052): starting (version 2.8.1), pid 7052 user 'root'
    Mar 21 12:06:47 ctmisdb1 gconfd (root-7052): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
    Mar 21 12:06:47 ctmisdb1 gconfd (root-7052): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 1
    Mar 21 12:06:47 ctmisdb1 gconfd (root-7052): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
    Mar 21 12:06:55 ctmisdb1 gconfd (root-7052): Resolved address "xml:readwrite:/root/.gconf" to a writable configuration source at position 0
    Mar 21 12:07:41 ctmisdb1 su(pam_unix)[5547]: session opened for user oracle by (uid=0)
    Mar 21 12:07:41 ctmisdb1 logger: Running CRSD with TZ =
    Mar 21 12:07:43 ctmisdb1 su(pam_unix)[7399]: session opened for user oracle by (uid=0)
    Mar 21 12:12:49 ctmisdb1 sshd(pam_unix)[15323]: session opened for user root by root(uid=0)
    Mar 21 12:12:57 ctmisdb1 su(pam_unix)[15531]: session opened for user oracle by root(uid=0)
    Mar 21 12:47:05 ctmisdb1 syslogd 1.4.1: restart.
    ocssd log....
    [    CSSD]2012-03-21 11:24:41.045 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800661f0c0) proc(0x8006622560) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 11:24:41.078 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660cfe0) proc(0x800662ba70) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:07:44.564 >USER: Oracle Database 10g CSS Release 10.2.0.1.0 Production Copyright 1996, 2004 Oracle. All rights reserved.
    [  clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=ctmisdb1DBG_CSSD))
    [    CSSD]2012-03-21 12:07:44.564 >USER: CSS daemon log for node ctmisdb1, number 1, in cluster crs
    [    CSSD]2012-03-21 12:07:44.581 [28260544] >TRACE: clssscmain: local-only set to false
    [    CSSD]2012-03-21 12:07:44.603 [28260544] >TRACE: clssnmReadNodeInfo: added node 1 (ctmisdb1) to cluster
    [    CSSD]2012-03-21 12:07:44.621 [28260544] >TRACE: clssnmReadNodeInfo: added node 2 (ctmisdb2) to cluster
    [    CSSD]2012-03-21 12:07:44.627 [72925824] >TRACE: clssnm_skgxnmon: skgxn init failed, rc 1
    [    CSSD]2012-03-21 12:07:44.627 [28260544] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor
    [    CSSD]2012-03-21 12:07:44.641 [28260544] >TRACE: clssnmInitNMInfo: misscount set to 60
    [    CSSD]2012-03-21 12:07:44.655 [28260544] >TRACE: clssnmDiskStateChange: state from 1 to 2 disk (0//dev/raw/raw2)
    [    CSSD]2012-03-21 12:07:46.661 [72925824] >TRACE: clssnmDiskStateChange: state from 2 to 4 disk (0//dev/raw/raw2)
    [    CSSD]2012-03-21 12:07:46.690 [72925824] >TRACE: clssnmReadDskHeartbeat: node(2) is down. rcfg(18) wrtcnt(7920) LATS(0) Disk lastSeqNo(7920)
    [    CSSD]2012-03-21 12:07:46.752 [28260544] >TRACE: clssnmFatalInit: fatal mode enabled
    [    CSSD]2012-03-21 12:07:46.752 [94777984] >TRACE: clssnmconnect: connecting to node 1, flags 0x0001, connector 1
    [    CSSD]2012-03-21 12:07:46.753 [94777984] >TRACE: clssnmconnect: connecting to node 0, flags 0x0000, connector 1
    [    CSSD]2012-03-21 12:07:46.753 [94777984] >TRACE: clssnmClusterListener: Probing node(2)
    [    CSSD]2012-03-21 12:07:46.755 [94777984] >TRACE: clssnmConnComplete: connected to node 2 (con 0x8006601040), state 3 birth 0, unique 1332303918/1332303918 prevConuni(0)
    [    CSSD]2012-03-21 12:07:46.756 [106332800] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_crs_1))
    [    CSSD]2012-03-21 12:07:46.756 [106332800] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ctmisdb1_crs))
    [    CSSD]2012-03-21 12:07:46.757 [151810688] >TRACE: clssnmPollingThread: Connection complete
    [    CSSD]2012-03-21 12:07:46.757 [162296448] >TRACE: clssnmSendingThread: Connection complete
    [    CSSD]2012-03-21 12:07:46.757 [172782208] >TRACE: clssnmRcfgMgrThread: Connection complete
    [    CSSD]2012-03-21 12:07:46.757 [172782208] >TRACE: clssnmRcfgMgrThread: Local Join
    [    CSSD]2012-03-21 12:07:46.757 [172782208] >WARNING: clssnmLocalJoinEvent: takeover aborted due to connected but inactive nodes
    [    CSSD]2012-03-21 12:07:47.339 [94777984] >TRACE: clssnmHandleSync: Acknowledging sync: src[2] srcName[ctmisdb2] seq[5] sync[18]
    [    CSSD]2012-03-21 12:07:47.759 [172782208] >TRACE: clssnmRcfgMgrThread: lastleader(2) unique(1332311864)
    [    CSSD]2012-03-21 12:07:48.341 [94777984] >TRACE: clssnmSendVoteInfo: node(2) syncSeqNo(18)
    [    CSSD]2012-03-21 12:07:50.346 [94777984] >TRACE: clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
    [    CSSD]2012-03-21 12:07:50.346 [94777984] >TRACE: clssnmDeactivateNode: node 0 () left cluster
    [    CSSD]2012-03-21 12:07:50.346 [94777984] >TRACE: clssnmUpdateNodeState: node 1, state (1/2) unique (1332311864/1332311864) prevConuni(0) birth (0/18) (old/new)
    [    CSSD]2012-03-21 12:07:50.346 [94777984] >TRACE: clssnmUpdateNodeState: node 2, state (4/3) unique (1332303918/1332303918) prevConuni(0) birth (0/16) (old/new)
    [    CSSD]2012-03-21 12:07:50.346 [94777984] >USER: clssnmHandleUpdate: SYNC(18) from node(2) completed
    [    CSSD]2012-03-21 12:07:50.346 [94777984] >USER: clssnmHandleUpdate: NODE 1 (ctmisdb1) IS ACTIVE MEMBER OF CLUSTER
    [    CSSD]2012-03-21 12:07:50.346 [94777984] >USER: clssnmHandleUpdate: NODE 2 (ctmisdb2) IS ACTIVE MEMBER OF CLUSTER
    [    CSSD]2012-03-21 12:07:50.429 [28260544] >USER: NMEVENT_SUSPEND [00][00][00][00]
    [    CSSD]2012-03-21 12:07:50.429 [183267968] >TRACE: clssgmReconfigThread: started for reconfig (18)
    [    CSSD]2012-03-21 12:07:50.429 [183267968] >USER: NMEVENT_RECONFIG [00][00][00][06]
    [    CSSD]2012-03-21 12:07:50.429 [183267968] >TRACE: clssgmEstablishConnections: 2 nodes in cluster incarn 18
    [    CSSD]2012-03-21 12:07:50.430 [140255872] >TRACE: clssgmInitialRecv: (0x102a0360) accepted a new connection from node 2 born at 16 active (2, 2), vers (10,3,1,2)
    [    CSSD]2012-03-21 12:07:50.430 [140255872] >TRACE: clssgmInitialRecv: conns done (2/2)
    [    CSSD]2012-03-21 12:07:50.430 [183267968] >TRACE: clssgmEstablishMasterNode: MASTER for 18 is node(2) birth(16)
    [    CSSD]2012-03-21 12:07:50.430 [183267968] >TRACE: clssgmChangeMasterNode: requeued 0 RPCs
    [    CSSD]2012-03-21 12:07:50.432 [140255872] >TRACE: clssgmHandleDBDone(): src/dest (2/65535) size(72) incarn 18
    [    CSSD]CLSS-3000: reconfiguration successful, incarnation 18 with 2 nodes
    [    CSSD]CLSS-3001: local node number 1, master node number 2
    [    CSSD]2012-03-21 12:07:50.433 [183267968] >TRACE: clssgmReconfigThread: completed for reconfig(18), with status(1)
    [    CSSD]2012-03-21 12:07:50.550 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006603bb0) proc(0x8006608b00) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:07:50.551 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x80066066f0) proc(0x8006608d70) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:07:53.569 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660ec70) proc(0x8006611260) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:00.829 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006610990) proc(0x800660de00) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:04.698 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006613030) proc(0x8006612930) pid(8115) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:04.816 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006612950) proc(0x8006613c20) pid(8115) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:04.832 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006612950) proc(0x8006613c20) pid(8115) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:06.615 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006612950) proc(0x8006613c20) pid(8171) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:07.114 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006615960) proc(0x8006616350) pid(8175) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:11.373 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x80066192a0) proc(0x8006619470) pid(8302) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:11.669 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800661bf60) proc(0x800661ee20) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:17.135 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800661bf60) proc(0x800661ee70) pid(8458) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:17.268 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800661fc00) proc(0x80066220d0) pid(8460) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:17.305 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x80066223e0) proc(0x8006625250) pid(8462) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:17.353 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006625560) proc(0x8006628430) pid(8464) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:24.585 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006625560) proc(0x8006628430) pid(8645) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:27.957 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006628740) proc(0x800662b610) pid(8722) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:30.931 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800662cce0) proc(0x800662c860) pid(8801) proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:36.400 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800661c5f0) proc(0x800661eb50) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:37.863 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800662f1c0) proc(0x800661eee0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:38.537 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800662f1c0) proc(0x800661d500) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:39.232 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800661bf60) proc(0x800661d500) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:43.085 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x8006611210) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:08:58.971 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x80066112c0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:09:59.290 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x800660b190) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:10:59.589 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x800660b190) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:11:59.904 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x800660b190) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:13:00.203 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x800660b190) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:13:14.029 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x800660b190) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:14:00.501 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x8006611210) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:15:00.809 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:16:01.117 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:17:01.447 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:01.762 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:39.841 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:42.123 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:42.316 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:42.843 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:42.963 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:43.098 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b260) proc(0x800662bd20) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:44.173 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:44.368 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b260) proc(0x800660b310) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:45.351 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x8006628670) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:46.236 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:47.031 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:47.694 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:47.819 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b260) proc(0x800660b310) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:48.103 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:48.327 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b260) proc(0x800660b310) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:48.484 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x8006611210) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:48.758 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:49.529 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:50.509 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:51.060 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x800660b830) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:18:51.558 [106332800] >TRACE: clssgmClientConnectMsg: Connect from con(0x8006611630) proc(0x800662f0f0) pid() proto(10:2:1:1)
    [    CSSD]2012-03-21 12:48:39.836 >USER: Oracle Database 10g CSS Release 10.2.0.1.0 Production Copyright 1996, 2004 Oracle. All rights reserved.
    [  clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=ctmisdb1DBG_CSSD))
    [    CSSD]2012-03-21 12:48:39.836 >USER: CSS daemon log for node ctmisdb1, number 1, in cluster crs
    [    CSSD]2012-03-21 12:48:39.849 [28260544] >TRACE: clssscmain: local-only set to false
    [    CSSD]2012-03-21 12:48:39.865 [28260544] >TRACE: clssnmReadNodeInfo: added node 1 (ctmisdb1) to cluster
    [    CSSD]2012-03-21 12:48:39.872 [28260544] >TRACE: clssnmReadNodeInfo: added node 2 (ctmisdb2) to cluster
    [    CSSD]2012-03-21 12:48:39.879 [72925824] >TRACE: clssnm_skgxnmon: skgxn init failed, rc 1
    [    CSSD]2012-03-21 12:48:39.879 [28260544] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor
    [    CSSD]2012-03-21 12:48:39.881 [28260544] >TRACE: clssnmInitNMInfo: misscount set to 60
    [    CSSD]2012-03-21 12:48:39.888 [28260544] >TRACE: clssnmDiskStateChange: state from 1 to 2 disk (0//dev/raw/raw2)
    [    CSSD]2012-03-21 12:48:41.892 [72925824] >TRACE: clssnmDiskStateChange: state from 2 to 4 disk (0//dev/raw/raw2)
    [    CSSD]2012-03-21 12:48:41.915 [72925824] >TRACE: clssnmReadDskHeartbeat: node(2) is down. rcfg(20) wrtcnt(10367) LATS(0) Disk lastSeqNo(10367)
    [    CSSD]2012-03-21 12:48:41.959 [28260544] >TRACE: clssnmFatalInit: fatal mode enabled
    [    CSSD]2012-03-21 12:48:41.959 [94777984] >TRACE: clssnmconnect: connecting to node 1, flags 0x0001, connector 1
    [    CSSD]2012-03-21 12:48:41.959 [94777984] >TRACE: clssnmconnect: connecting to node 0, flags 0x0000, connector 1
    [    CSSD]2012-03-21 12:48:41.959 [94777984] >TRACE: clssnmClusterListener: Probing node(2)
    [    CSSD]2012-03-21 12:48:41.961 [94777984] >TRACE: clssnmConnComplete: connected to node 2 (con 0x8006702790), state 3 birth 0, unique 1332303918/1332303918 prevConuni(0)
    [    CSSD]2012-03-21 12:48:41.962 [106332800] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_crs_1))
    [    CSSD]2012-03-21 12:48:41.962 [106332800] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ctmisdb1_crs))
    [    CSSD]2012-03-21 12:48:41.963 [152330880] >TRACE: clssnmPollingThread: Connection complete
    [    CSSD]2012-03-21 12:48:41.963 [162816640] >TRACE: clssnmSendingThread: Connection complete
    [    CSSD]2012-03-21 12:48:41.963 [173302400] >TRACE: clssnmRcfgMgrThread: Connection complete
    [    CSSD]2012-03-21 12:48:41.963 [173302400] >TRACE: clssnmRcfgMgrThread: Local Join
    [    CSSD]2012-03-21 12:48:41.963 [173302400] >WARNING: clssnmLocalJoinEvent: takeover aborted due to connected but inactive nodes
    [    CSSD]2012-03-21 12:48:42.631 [94777984] >TRACE: clssnmHandleSync: Acknowledging sync: src[2] srcName[ctmisdb2] seq[13] sync[20]
    [    CSSD]2012-03-21 12:48:42.965 [173302400] >TRACE: clssnmRcfgMgrThread: lastleader(2) unique(1332314319)
    [    CSSD]2012-03-21 12:48:43.636 [94777984] >TRACE: clssnmSendVoteInfo: node(2) syncSeqNo(20)
    [    CSSD]2012-03-21 12:48:45.640 [94777984] >TRACE: clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
    [    CSSD]2012-03-21 12:48:45.640 [94777984] >TRACE: clssnmDeactivateNode: node 0 () left cluster
    [    CSSD]2012-03-21 12:48:45.640 [94777984] >TRACE: clssnmUpdateNodeState: node 1, state (1/2) unique (1332314319/1332314319) prevConuni(0) birth (0/20) (old/new)
    [    CSSD]2012-03-21 12:48:45.640 [94777984] >TRACE: clssnmUpdateNodeState: node 2, state (4/3) unique (1332303918/1332303918) prevConuni(0) birth (0/16) (old/new)
    [    CSSD]2012-03-21 12:48:45.640 [94777984] >USER: clssnmHandleUpdate: SYNC(20) from node(2) completed
    [    CSSD]2012-03-21 12:48:45.640 [94777984] >USER: clssnmHandleUpdate: NODE 1 (ctmisdb1) IS ACTIVE MEMBER OF CLUSTER
    [    CSSD]2012-03-21 12:48:45.640 [94777984] >USER: clssnmHandleUpdate: NODE 2 (ctmisdb2) IS ACTIVE MEMBER OF CLUSTER
    [    CSSD]2012-03-21 12:48:45.737 [28260544] >USER: NMEVENT_SUSPEND [00][00][00][00]
    [    CSSD]2012-03-21 12:48:45.738 [183788160] >TRACE: clssgmReconfigThread: started for reconfig (20)
    [    CSSD]2012-03-21 12:48:45.738 [183788160] >USER: NMEVENT_RECONFIG [00][00][00][06]
    [    CSSD]2012-03-21 12:48:45.738 [183788160] >TRACE: clssgmEstablishConnections: 2 nodes in cluster incarn 20
    [    CSSD]2012-03-21 12:48:45.739 [140776064] >TRACE: clssgmInitialRecv: (0x102a0370) accepted a new connection from node 2 born at 16 active (2, 2), vers (10,3,1,2)
    [    CSSD]2012-03-21 12:48:45.739 [140776064] >TRACE: clssgmInitialRecv: conns done (2/2)
    [    CSSD]2012-03-21 12:48:45.739 [183788160] >TRACE: clssgmEstablishMasterNode: MASTER for 20 is node(2) birth(16)
    [    CSSD]2012-03-21 12:48:45.739 [183788160] >TRACE: clssgmChangeMasterNode: requeued 0 RPCs
    [    CSSD]2012-03-21 12:48:45.741 [140776064] >TRACE: clssgmHandleDBDone(): src/dest (2/65535) size(72) incarn 20
    [    CSSD]CLSS-3000: reconfiguration successful, incarnation 20 with 2 nodes
    Plz check and help..........

  • Oracle RAC 11g query

    Hi All,
    I am new to Oracle RAC 11g, facing a issue, Request all to help.
    1) What is the instance id means which we get from gv$session in Oracle RAC setup?
    2) What is differnce between Session failure and node failure?
    3) If session fail, then client should connect to node1(to which session previously connected) or node2?
    Thanks

    1) What is the instance id means which we get from gv$session in Oracle RAC setup?Instance ID is unique for each instance in clustered database.
    2) What is differnce between Session failure and node failure?Session Failure - when the connection to an instance is lost, SESSION failover results only in the establishment of a new connection to another Oracle RAC node
    Node Failure/Eviction - we can say hardware fails, the Cluster Manager reports the change in the cluster's membership to Global Resource Directory (GRD)
    Resource directory which consists of both Global enqueue service & cache service
    3) If session fail, then client should connect to node1(to which session previously connected) or node2?You have to configure in SERVICE.

  • RAC node outage causes SOA Suite 10.1.3.4 BPEL  failure

    Using weblogic 9.2 and the SOA Suite 10.1.3.4. We use a 10g Oracle RAC ( 2 nodes ); the WL cluster has a multi data source of 2 pools, each pool pointing to a single node in the rac, each pool deployed to the cluster, and the multi data source in load-balancing mode.
    So the other night, one of the db nodes had a hardware failure ( ironically, with a remote monitoring / management card ). Annoying, but it should not have caused the BPEL servers to be in "FAILED NOT RESTARTABLE" status the next morning.
    Jun 9, 2009 12:10:07 AM EDT> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "esbaqds2": Io exception: The Network Adapter could not establish the connection>
    SEVERE: Destroying JMSDequeuer failed
    oracle.jms.AQjmsException: Connection has been administratively destroyed. Reconnect.
    at oracle.jms.AQjmsSession.preClose(AQjmsSession.java:980)
    at oracle.jms.AQjmsObject.close(AQjmsObject.java:409)
    at oracle.jms.AQjmsSession.close(AQjmsSession.java:1020)
    at oracle.tip.esb.server.dispatch.JMSDequeuer.destroy(JMSDequeuer.java:419)
    at oracle.tip.esb.server.dispatch.JMSDequeuer.destroyWithoutUnsubscribing(JMSDequeuer.java:395)
    at oracle.tip.esb.server.dispatch.JMSDequeuer.dequeue(JMSDequeuer.java:175)
    at oracle.tip.esb.server.dispatch.agent.ESBWork.process(ESBWork.java:174)
    at oracle.tip.esb.server.dispatch.agent.ESBWork.run(ESBWork.java:132)
    at weblogic.connector.security.layer.WorkImpl.runIt(WorkImpl.java:108)
    at weblogic.connector.security.layer.WorkImpl.run(WorkImpl.java:44)
    at weblogic.connector.work.WorkRequest.run(WorkRequest.java:93)
    at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    followed by a 2 GB log file containing 1.3 million iterations of the following within the next 10 minutes before the managed servers failed.
    java.lang.NullPointerException
    at java.lang.String.<init>(String.java:144)
    at oracle.tip.esb.server.dispatch.JMSDequeuer.dequeue(JMSDequeuer.java:168)
    at oracle.tip.esb.server.dispatch.agent.ESBWork.process(ESBWork.java:174)
    at oracle.tip.esb.server.dispatch.agent.ESBWork.run(ESBWork.java:132)
    at weblogic.connector.security.layer.WorkImpl.runIt(WorkImpl.java:108)
    at weblogic.connector.security.layer.WorkImpl.run(WorkImpl.java:44)
    at weblogic.connector.work.WorkRequest.run(WorkRequest.java:93)
    at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    Both managed instances of the BPEL cluster failed, even though the 1st node of the Oracle RAC was still available.
    Our 10.3 cluster, also using multi data sources to the same RAC for the OSB components, simply went on about its business using the remaining rac node pool.
    Seems to be a single point of failure...

    We haven't changed the JDBC connection string yet, but we did run a test in the same environment while Oracle support considers the situation.
    For the test, we simply shutdown one node of the RAC and watched to see what happens. Within the space of a minute, the JDBC "Failed Reserve Request Count" was increasing by thousands on every refresh of the screen. We restarted the RAC node after 5 minutes, by which time the "Failed Reserve Request Count" was over 190,000
    The 2 BPEL managed servers remained in Running status and each created a 660 MB log file within that 5 minutes. In the original outage, the nodes were down for about 15 minutes. Most of the logging is being generated from within the oracle.tip.esb classes, not by the weblogic classes. It looks like that once the pool pointing to the downed RAC node becomes disabled, the Oracle BPEL code is still trying to use it even though the multi-source JNDI is the published lookup:
    INFO: JMSDequeuer::createConnection - AQ Topics
    java.sql.SQLException: weblogic.common.ResourceException:
    esbaqds(esbaqds2): Pool esbaqds2 is disabled, cannot allocate resources to applications..
    esbaqds(esbaqds1): Pool esbaqds1 is disabled, cannot allocate resources to applications..
    at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:250)
    at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:348)
    at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:364)
    at oracle.tip.esb.server.dispatch.JMSDequeuer.createAQConnection(JMSDequeuer.java:559)
    at oracle.tip.esb.server.dispatch.JMSDequeuer.dequeue(JMSDequeuer.java:159)
    at oracle.tip.esb.server.dispatch.agent.ESBWork.process(ESBWork.java:174)
    at oracle.tip.esb.server.dispatch.agent.ESBWork.run(ESBWork.java:132)
    at weblogic.connector.security.layer.WorkImpl.runIt(WorkImpl.java:108)
    at weblogic.connector.security.layer.WorkImpl.run(WorkImpl.java:44)
    at weblogic.connector.work.WorkRequest.run(WorkRequest.java:93)
    at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
    Caused by: weblogic.common.ResourceException:
    esbaqds(esbaqds2): Pool esbaqds2 is disabled, cannot allocate resources to applications..
    esbaqds(esbaqds1): Pool esbaqds1 is disabled, cannot allocate resources to applications..
    at weblogic.jdbc.common.internal.MultiPool.searchLoadBalance(MultiPool.java:331)
    at weblogic.jdbc.common.internal.MultiPool.findPool(MultiPool.java:202)
    at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:77)
    at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:346)
    ... 11 more
    SEVERE: Failed to process deferred message
    oracle.tip.esb.server.dispatch.QueueHandlerException: Error creating "weblogic.common.ResourceException: No good connections available."
    at oracle.tip.esb.server.dispatch.JMSDequeuer.createAQConnection(JMSDequeuer.java:661)
    at oracle.tip.esb.server.dispatch.JMSDequeuer.dequeue(JMSDequeuer.java:159)
    at oracle.tip.esb.server.dispatch.agent.ESBWork.process(ESBWork.java:174)
    at oracle.tip.esb.server.dispatch.agent.ESBWork.run(ESBWork.java:132)
    at weblogic.connector.security.layer.WorkImpl.runIt(WorkImpl.java:108)
    at weblogic.connector.security.layer.WorkImpl.run(WorkImpl.java:44)
    at weblogic.connector.work.WorkRequest.run(WorkRequest.java:93)
    at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

  • Gig Ethernet V/S  SCI as Cluster Private Interconnect for Oracle RAC

    Hello Gurus
    Can any one pls confirm if it's possible to configure 2 or more Gigabit Ethernet interconnects ( Sun Cluster 3.1 Private Interconnects) on a E6900 cluster ?
    It's for a High Availability requirement of Oracle 9i RAC. i need to know ,
    1) can i use gigabit ethernet as Private cluster interconnect for Deploying Oracle RAC on E6900 ?
    2) What is the recommended Private Cluster Interconnect for Oracle RAC ? GiG ethernet or SCI with RSM ?
    3) How about the scenarios where one can have say 3 X Gig Ethernet V/S 2 X SCI , as their cluster's Private Interconnects ?
    4) How the Interconnect traffic gets distributed amongest the multiple GigaBit ethernet Interconnects ( For oracle RAC) , & is anything required to be done at oracle Rac Level to enable Oracle to recognise that there are multiple interconnect cards it needs to start utilizing all of the GigaBit ethernet Interfaces for transfering packets ?
    5) what would happen to Oracle RAC if one of the Gigabit ethernet private interconnects fails
    Have tried searching for this info but could not locate any doc that can precisely clarify these doubts that i have .........
    thanks for the patience
    Regards,
    Nilesh

    Answers inline...
    Tim
    Can any one pls confirm if it's possible to configure
    2 or more Gigabit Ethernet interconnects ( Sun
    Cluster 3.1 Private Interconnects) on a E6900
    cluster ?Yes, absolutely. You can configure up to 6 NICs for the private networks. Traffic is automatically striped across them if you specify clprivnet0 to Oracle RAC (9i or 10g). That is TCP connections and UDP messages.
    It's for a High Availability requirement of Oracle
    9i RAC. i need to know ,
    1) can i use gigabit ethernet as Private cluster
    interconnect for Deploying Oracle RAC on E6900 ? Yes, definitely.
    2) What is the recommended Private Cluster
    Interconnect for Oracle RAC ? GiG ethernet or SCI
    with RSM ? SCI is or is in the process of being EOL'ed. Gigabit is usually sufficient. Longer term you may want to consider Infiniband or 10 Gigabit ethernet with RDS.
    3) How about the scenarios where one can have say 3 X
    Gig Ethernet V/S 2 X SCI , as their cluster's
    Private Interconnects ? I would still go for 3 x GbE because it is usually cheaper and will probably work just as well. The latency and bandwidth differences are often masked by the performance of the software higher up the stack. In short, unless you tuned the heck out of your application and just about everything else, don't worry too much about the difference between GbE and SCI.
    4) How the Interconnect traffic gets distributed
    amongest the multiple GigaBit ethernet Interconnects
    ( For oracle RAC) , & is anything required to be done
    at oracle Rac Level to enable Oracle to recognise
    that there are multiple interconnect cards it needs
    to start utilizing all of the GigaBit ethernet
    Interfaces for transfering packets ?You don't need to do anything at the Oracle level. That's the beauty of using Oracle RAC with Sun Cluster as opposed to RAC on its own. The striping takes place automatically and transparently behind the scenes.
    5) what would happen to Oracle RAC if one of the
    Gigabit ethernet private interconnects fails It's completely transparent. Oracle will never see the failure.
    Have tried searching for this info but could not
    locate any doc that can precisely clarify these
    doubts that i have .........This is all covered in a paper that I have just completed and should be published after Christmas. Unfortunately, I cannot give out the paper yet.
    thanks for the patience
    Regards,
    Nilesh

  • How to establish user equvalance between 2 rac nodes on non default ports??

    Hi Friends,
    Please shed some light on how to establish user equvalance between 2 rac nodes on non default ports such as ssh on 22...
    i want to establish user equvalance on other ports..
    Thanks
    RB

    R12DBA wrote:
    Hi Friends,
    Please shed some light on how to establish user equvalance between 2 rac nodes on non default ports such as ssh on 22...
    i want to establish user equvalance on other ports..
    Thanks
    RBHi RB ,
    22 is default port for ssh. For configuring
    http://yasarmoran.wordpress.com/2010/06/12/configuring-ssh-on-oracle-rac-nodes/
    For non default port , first of all you need to configure ssh on new port . For that refer :
    http://www.itworld.com/nls_unixssh0500506

Maybe you are looking for

  • New iPod Touch 5th Gen- iOS 7.0.6 major battery drain.

    iPod Touch 5th Gen, 32GB Have all the major known drainers off: such as Reduce Motion, Location Services, Notifications, Bluetooth, etc. Was at 100% this a.m. Usage: 2 hours, 9 minutes Standby Mode: 11 hours, 45 minutes. It was at 65% 4 minutes ago.

  • Getting error in mapping execution Invalid CHAR

    hi i am having file to RFC scenario, and the input file is an xml file and i am getting erron in mapping as below but when i download the payload and load it in the integration repository mapping tool ,i am able to execute it  succesfully. i think th

  • Problems with sudo on Solaris 8

    Greetings all, I am trying to set up user accounts with scripts that will restart services, should they fail. I am setting up the default shell as the script that I want to have run when they log in and at the completion it will kick them off. A nice

  • Re-using expression-generated values to input into other expressions (read-out)

    Hi there everyone. I was wondering if there is a way to 'capture' expression generated values and either buffer these values or re-using them directly in other expressions. I am trying to track rotations of parented layers, which are generated from t

  • Dypro table error

    Hello All, we are facing issue while doing sap migration from windows sql to solaries oracle. Import mointor has been successfull but we are issue in table Dynopro. during the istallation phase we stuck facing issue reset DDIC password. when try to l