ASM on one node crashes when we start the other two nodes ASM

Question

We completed database build in Aug 2010
We complete PSU patching in Jan ending
Feb 4th the database crashed
We cannot start ASM on node1
ASM starts good on node2 and node3 but node1 cannot join
If ASM is down on node2, node3 then we can start ASM node1Reconfiguration started (old inc 0, new inc 6)
ASM instance
List of nodes:
0 1 2
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* allocate domain 1, invalid = TRUE
* allocate domain 2, invalid = TRUE
Mon Mar 01 16:53:00 2010
Trace dumping is performing id=[cdmp_20100301165301]
Mon Mar 01 16:53:55 2010
ERROR: LMD0 (ospid: 274638) detects an idle connection to instance 2
Mon Mar 01 16:54:44 2010
Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_860280.trc (incident=116865):
ORA-29740: evicted by member 1, group incarnation 8
Incident details in: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116865/+ASM1_lmon_860280_i116865.trc
Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_860280.trc:
ORA-29740: evicted by member 1, group incarnation 8
LMON (ospid: 860280): terminating the instance due to error 29740
Mon Mar 01 16:54:46 2010
System state dump is made for local instance
Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_614488.trc (incident=116833):
ORA-29740: evicted by member , group incarnation
Incident details in: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116833/+ASM1_diag_614488_i116833.trc
Mon Mar 01 16:54:46 2010
ORA-1092 : opitsk aborting process
Errors in file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_614488.trc:
ORA-29740: evicted by member , group incarnation
Trace dumping is performing id=[cdmp_20100301165446]
Instance terminated by LMON, pid = 860280
Another thing we found that when we start ASM on node1, the cluster interconnect hangs when we try to ping
We did modify the cluster_interconnect parameter to try to start using public interface but the issued remained the same and we were not able to ping public interface
The crs is fine
$ crs_stat -t
Name Type Target State Host
ora....p1.inst application ONLINE OFFLINE
ora....p2.inst application ONLINE ONLINE noden2
ora....p3.inst application ONLINE ONLINE noden3
ora....1p2.srv application ONLINE ONLINE noden2
ora....1p3.srv application ONLINE ONLINE noden3
ora.....net.cs application ONLINE ONLINE noden1
ora.appl.db application ONLINE ONLINE noden1
ora....SM1.asm application ONLINE OFFLINE
ora....N1.lsnr application ONLINE ONLINE noden1
ora....8n1.gsd application ONLINE ONLINE noden1
ora....8n1.ons application ONLINE ONLINE noden1
ora....8n1.vip application ONLINE ONLINE noden1
ora....SM2.asm application ONLINE ONLINE noden2
ora....N2.lsnr application ONLINE ONLINE noden2
ora....8n2.gsd application ONLINE ONLINE noden2
ora....8n2.ons application ONLINE ONLINE noden2
ora....8n2.vip application ONLINE ONLINE noden2
ora....SM3.asm application ONLINE ONLINE noden3
ora....N3.lsnr application ONLINE ONLINE noden3
ora....8n3.gsd application ONLINE ONLINE noden3
ora....8n3.ons application ONLINE ONLINE noden3
ora....8n3.vip application ONLINE ONLINE noden3
Any inpts can help

Answer 1

Env
3-node RAC
oracle version 11.1.0.7
Latest PSU Jan applied
OS is AIX version is 6100-02==========
LMON trace files
==========
Trace file /oradb/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_860280.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /oradb/oracle/product/11.1/asm_1
System name:     AIX
Node name:     host-node1
Release:     1
Version:     6
Machine:     00C39EA44C00
Instance name: +ASM1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 8
Unix process pid: 860280, image: oracle@host-node1 (LMON)
*** 2010-03-01 16:50:23.023
*** SESSION ID:(218.1) 2010-03-01 16:50:23.023
*** CLIENT ID:() 2010-03-01 16:50:23.023
*** SERVICE NAME:() 2010-03-01 16:50:23.023
*** MODULE NAME:() 2010-03-01 16:50:23.023
*** ACTION NAME:() 2010-03-01 16:50:23.023
GES resources 5596 pool 6
GES enqueues 7959
GES IPC: Receivers 2 Senders 2
GES IPC: Buffers Receive 1000 Send (i:1150 b:482) Reserve 402
GES IPC: Msg Size Regular 416 Batch 8192
Batching factor: enqueue replay 201, ack 224
Batching factor: cache replay 126 size per lock 64
kjxggin: CGS tickets = 1000
kjxgrdmpcpu: CPU Total 6 Core 3 Socket -1 OCPU 6
kjxgrdmpcpu: High load threshold 21504
*** 2010-03-01 16:50:23.362
kjxgmrcfg: Reconfiguration started, type 1
kjxgmcs: Setting state to 0 0.
*** 2010-03-01 16:50:23.363
Name Service frozen
kjxgmcs: Setting state to 0 1.
kjxgrdecidever: No old version members in the cluster
kjxgrssvote: reconfig bitmap chksum 0x88477268 cnt 3 master 0 ret 0
ksirValidateModuleInfo: action = 10 startup = 0
Name Service Mode: multi (0x21)
kjfcpiora: published my fusion master weight 5322
kjfcpiora: publish my flogb 9
kjfcpiora: publish my cluster_database_instances parameter=3
kjxggpoll: change poll time to 50 ms
kjxgrpropmsg: SSMEMI: inst 1 - no disk vote
kjxgrpropmsg: SSMEMI: inst 1 - no disk vote
kjxgrpropmsg: SSMEMI: inst 2 - no disk vote
SSVOTE: Master indicates no Disk Voting
kjxgmps: proposing substate 2
kjxgmcs: Setting state to 6 2.
kjfmuin: bitmap 0 1 2
kjfmmhi: received msg from 0 (inc 6)
kjfmmhi: received msg from 1 (inc 2)
kjfmmhi: received msg from 2 (inc 4)
Performed the unique instance identification check
kjxgmps: proposing substate 3
kjxgmcs: Setting state to 6 3.
Name Service recovery started
Deleted all dead-instance name entries
kjxgmps: proposing substate 4
kjxgmcs: Setting state to 6 4.
Multicasted all local name entries for publish
Replayed all pending requests
kjxgmps: proposing substate 5
kjxgmcs: Setting state to 6 5.
Name Service normal
Name Service recovery done
*** 2010-03-01 16:50:23.889
*** 2010-03-01 16:50:23.958
kjxgmps: proposing substate 6
kjxgmcs: Setting state to 6 6.
kjxggpoll: change poll time to 600 ms
2010-03-01 16:50:23.980620 :
********* kjfcrfg() called, BEGIN LMON RCFG *********
kjfcrfg: DRM window size = 0->128 (min lognb = 9)
2010-03-01 16:50:23.980811 :
Reconfiguration started (old inc 0, new inc 6)
ASM instance
Send timeout: 300 secs
Defer Queue timeout: 360 secs
Synchronization timeout: 420 sec
List of nodes:
0 1 2
*** 2010-03-01 16:50:24.023
2010-03-01 16:50:24.034432 : Global Resource Directory frozen
node 0
release 11 1 0 7
node 1
release 11 1 0 7
node 2
release 11 1 0 7
number of mastership buckets = 128
2010-03-01 16:50:24.034959 :
domain attach called for domid 0
* kjbdomalc: domain 0 invalid = TRUE
* kjbdomatt: first attach for domain 0
asby init, 0/0/x1
asby returns, 0/0/x1/false
* Domain maps before reconfiguration:
* DOMAIN 0 (valid 1): 0
* End of domain mappings
* Domain maps after recomputation:
* DOMAIN 0 (valid 1): 0 1 2
* End of domain mappings
Dead inst
Join inst 0 1 2
Exist inst
Active Sendback Threshold = 50 %
Communication channels reestablished
2010-03-01 16:50:24.152688 :
received all domreplay (6.6)
2010-03-01 16:50:24.152732 :
sent master 1 (6.6)
*** 2010-03-01 16:53:00.494
kjfmReceiverHealthCB_Check: Reciever [0] is healthy.
2010-03-01 16:52:56.921800 : Received comm error info from 2 (cnt 1)
kjxgrvalid: valid - 0.1 : (6 6) from 2
kjxgrrcfgchk: Initiating reconfig, reason=3
kjxgrrcfgchk: COMM rcfg - Disk Vote Required
2010-03-01 16:52:57.077877 : kjxgrnetchk: start 0x53001440, end 0x53019ae0
2010-03-01 16:52:57.077906 : kjxgrnetchk: Sending comm check req to 1
2010-03-01 16:52:57.078140 : kjxgrnetchk: Sending comm check req to 2
kjxgrrcfgchk: prev pstate 5 mapsz 512
kjxgrrcfgchk: new bmp: 0 1 2
kjxgrrcfgchk: work bmp: 0 1 2
kjxgrrcfgchk: rr bmp: 0 1 2
*** 2010-03-01 16:53:00.792
kjxgmrcfg: Reconfiguration started, type 3
kjxgmcs: Setting state to 6 0.
*** 2010-03-01 16:53:00.792
Name Service frozen
kjxgmcs: Setting state to 6 1.
kjxgrdecidever: No old version members in the cluster
kjxgrmsghndlr: Queue msg (0x110a21e50->0x110f09b90) type 7 for later
*** 2010-03-01 16:54:43.233
kjxgrssvote: reconfig bitmap chksum 0x88477268 cnt 3 master 2 ret 0
kjxgrrcfgchk: disable CGS timeout
kjxggpoll: change poll time to 50 ms
* kjfcchknested: CGS rcfg detected in step 7.0.0
SSVOTE: Master indicates Disk Voting required
2010-03-01 16:54:37.535518 : kjxgrmsghndlr: evict req from 1 for 0, seq (8, 8) vers 2193970751
2010-03-01 16:54:37.535587 : kjxgrdtrt: Evicted by 1, seq (8, 8)
IMR state information
Member 0, thread -1, state 0x2:c, flags 0x2c48
RR seq commit 6 cur 8
Propstate 3 prv 2 pending 0
rcfg rsn 3, rcfg time 1392514113, mem ct 3
master 2, master rcfg time 1392479783
evicted memcnt 0, starttm 0 chkcnt 0
system load 241 (normal)
Member information:
Member 0, incarn 6, version 0x82c5563f, thrd -1
prev thrd -1, status 0x1203 (JR..), err 0x0000
Member 1, incarn 6, version 0x82c1073b, thrd 2
prev thrd -1, status 0x1007 (JRM.), err 0x0002
Member 2, incarn 6, version 0x82c114ee, thrd 3
prev thrd -1, status 0x0007 (JRM.), err 0x0000
=====================================================
Group name: +ASM
Member id: 0
Cached KGXGN event: 0
Group State:
State: 6 1
Reconfig started start-tm 0x4b8c373c tmout period 0xffffffff state 0x2
Reconfig INPG type 3 inc 6 rsn 0 data 0x0
Reconfig COMP type 1 inc 6 rsn 0 data 0x0
Commited Map: 0 1 2
New Map: 0 1 2
KGXGN Map: 0 1 2
KGXGN Map2: 0 1 2
Master node: 0
Memcnt 3 Rcvcnt 0
Substate Proposal: false
Inc Proposal:
incarn 0 memcnt 0 master 0
proposal false matched false
map:
Master Inc State:
incarn 0 memcnt 0 agrees 0 flag 0x1
wmap:
nmap:
ubmap:
Substate Handler Execution State
substate 0 status done
substate 1 status done
substate 2 status done
substate 3 status done
substate 4 status done
substate 5 status done
substate 6 status done
IMR hist: 20[0x0a00:0x53019b0e] 4[0x0007:0x53019b0e] 3[0x0006:0x53019b0e]
IMR hist: 20[0x0902:0x53019b0e] 20[0x0702:0x53019b0b] 20[0x0702:0x53019b0a]
IMR hist: 20[0x0702:0x53019b0a] 1[0x0006:0x53019b0a] 20[0x0702:0x53019aff]
IMR hist: 10[0x0006:0x52fdbdb1] 20[0x0b00:0x52fdbdb1] 9[0x0006:0x52fdbdaf]
IMR hist: 20[0x0a02:0x52fdbdaf] 20[0x0a01:0x52fdbce1] 20[0x0a00:0x52fdbc8a]
IMR hist: 4[0x0005:0x52fdbc86] 3[0x0004:0x52fdbc4c] 20[0x0900:0x52fdbc4c]
IMR hist: 20[0x0802:0x52fdbc08] 20[0x0801:0x52fdbc08] 20[0x0801:0x52fdbc08]
IMR hist: 20[0x0602:0x52fdbc08] 20[0x0601:0x52fdbc08] 20[0x0601:0x52fdbc08]
IMR hist: 20[0x0800:0x52fdbc08] 20[0x0700:0x52fdbc08] 20[0x0602:0x52fdbc07]
IMR hist: 20[0x0800:0x52fdbc07] 20[0x0700:0x52fdbc07] 1[0x0000:0x52fdbbb8]
IMR hist: 0[0x0000:0x00000000] 0[0x0000:0x00000000]
KJM HIST LMD0:
7:0 6:0 5:7:0 12:97697 7:0 6:0 5:7:0 12:97696 7:0 6:0
5:7:0 12:97703 7:0 6:0 5:7:0 2:0 1:0 12:97713 7:0 6:0
5:7:0 12:97766 7:0 6:0 5:7:0 12:97782 7:0 6:0 5:7:0 12:97778
7:0 6:0 5:7:0 12:97799 7:0 6:0 5:7:0 12:97771 7:0 6:0
5:7:0 12:97784 7:0 6:0 5:7:0 12:97805 7:0 6:0 5:7:0 12:97785
7:0 6:0 5:7:0 12:97757 7:0 6:0 5:7:0 12:97770 7:0 6:0
5:7:0 12:97784 7:0 6:0
KJM HIST LMS0:
7:0 6:0 5:7:0 10:0 12:97697 7:0 6:0 5:7:0 10:0 12:97696
7:0 6:0 5:7:0 10:0 12:97703 7:0 6:0 5:7:0 10:0 12:97713
7:0 6:0 5:7:0 10:0 2:0 12:97766 7:0 6:0 5:7:0 10:0
12:97782 7:0 6:0 5:7:0 10:0 12:97778 7:0 6:0 5:7:0 10:0
12:97799 7:0 6:0 5:7:0 10:0 12:97771 7:0 6:0 5:7:0 10:0
12:97784 7:0 6:0 5:7:0 10:0 12:97805 7:0 6:0 5:7:0 10:0
12:97785 7:0 6:0 5:7:0
DUMP state for lmd0 (ospid 274638)
DUMP IPC context for lmd0 (ospid 274638)
Dumping process 9.274638 info:
*** 2010-03-01 16:54:43.664
Process diagnostic dump for oracle@host-node1 (LMD0), OS id=274638,
pid: 9, proc_ser: 1, sid: 217, sess_ser: 1
loadavg : 1.72 1.07 0.90
swap info: free_mem = 28642.09M rsv = 16.00M
alloc = 21.13M avail = 4096.00M swap_free = 4074.87M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
240001 A oracle 274638 1 0 60 20 12ca9f590 156060 16:50:23 - 0:00 asm_lmd0_+ASM1
Short stack dump:
<-ksedsts()+0254<-ksdxfstk()+0028<-ksdxcb()+05d8<-sspuser()+0074<-4750<-poll()+000c<-sskgxp_select()+00e4<-skgxpiwait()+08a4<-skgxpwait()+06fc<-ksxpwait()+081c<-ksliwat()+0a58<-kslwaitctx()+0150<-kslwait()+006c<-ksxprcvimd()+0368<-kjctr_rksxp()+013c<-kjctrcv()+0160<-kjcsrmg()+005c<-kjmdm()+2454<-ksbrdp()+075c<-opirip()+0444<-opidrv()+0414<-sou2o()+0090<-opimai_real()+0148<-main()+0090<-__start()+0070
Process diagnostic dump actual duration=0.161000 sec
(max dump time=30.000000 sec)
*** 2010-03-01 16:54:43.825
SO: 0x70000001ff913a0, type: 2, owner: 0x0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x70000001ff913a0, name=process, file=ksu.h LINE:10706 ID:, pg=0
(process) Oracle pid:9, ser:1, calls cur/top: 0x70000001f733140/0x70000001f733140
flags : (0x6) SYSTEM
flags2: (0x100), flags3: (0x0)
int error: 0, call error: 0, sess error: 0, txn error 0
ksudlp FALSE at location: 0
(post info) last post received: 0 0 83
last post received-location: kji.h LINE:2369 ID:kjga: clear wait for lmon
last process to post me: 70000001ff903b0 1 6
last post sent: 0 0 25
last post sent-location: ksa2.h LINE:282 ID:ksasnd
last process posted by me: 70000001ff903b0 1 6
(latch info) wait_event=68 bits=0
Process Group: DEFAULT, pseudo proc: 0x70000001f4851d0
O/S info: user: oracle, term: UNKNOWN, ospid: 274638
OSD pid info: Unix process pid: 274638, image: oracle@host-node1 (LMD0)
Dump of memory from 0x070000001FF70038 to 0x070000001FF70240
70000001FF70030 00000000 00000000 [........]
70000001FF70040 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
SO: 0x70000001f6de4a0, type: 4, owner: 0x70000001ff913a0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x70000001ff913a0, name=session, file=ksu.h LINE:10719 ID:, pg=0
(session) sid: 217 ser: 1 trans: 0x0, creator: 0x70000001ff913a0
flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x408) -/-
DID: , short-term DID:
txn branch: 0x0
oct: 0, prv: 0, sql: 0x0, psql: 0x0, user: 0/SYS
ksuxds FALSE at location: 0
service name: SYS$BACKGROUND
Current Wait Stack:
0: waiting for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2613 seq_num=2614 snap_id=1
wait times: snap=0.018269 sec, exc=0.018269 sec, total=0.018269 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
in_wait=1 iflags=0x5a8
Wait State:
auto_close=0 flags=0x22 boundary=0x0/-1
Session Wait History:
0: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2612 seq_num=2613 snap_id=1
wait times: snap=0.160172 sec, exc=0.160172 sec, total=0.160172 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000008 sec of elapsed time
1: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2611 seq_num=2612 snap_id=1
wait times: snap=0.096359 sec, exc=0.096359 sec, total=0.096359 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000008 sec of elapsed time
2: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2610 seq_num=2611 snap_id=1
wait times: snap=0.098065 sec, exc=0.098065 sec, total=0.098065 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000007 sec of elapsed time
3: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2609 seq_num=2610 snap_id=1
wait times: snap=0.097831 sec, exc=0.097831 sec, total=0.097831 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000014 sec of elapsed time
4: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2608 seq_num=2609 snap_id=1
wait times: snap=0.095876 sec, exc=0.095876 sec, total=0.095876 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000008 sec of elapsed time
5: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2607 seq_num=2608 snap_id=1
wait times: snap=0.098788 sec, exc=0.098788 sec, total=0.098788 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000006 sec of elapsed time
6: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2606 seq_num=2607 snap_id=1
wait times: snap=0.098854 sec, exc=0.098854 sec, total=0.098854 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000007 sec of elapsed time
7: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2605 seq_num=2606 snap_id=1
wait times: snap=0.098040 sec, exc=0.098040 sec, total=0.098040 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000008 sec of elapsed time
8: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2604 seq_num=2605 snap_id=1
wait times: snap=0.097322 sec, exc=0.097322 sec, total=0.097322 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000007 sec of elapsed time
9: waited for 'ges remote message'
waittime=40, loop=0, p3=44
wait_id=2603 seq_num=2604 snap_id=1
wait times: snap=0.097334 sec, exc=0.097334 sec, total=0.097334 sec
wait times: max=0.080000 sec
wait counts: calls=1 os=1
occurred after 0.000008 sec of elapsed time
Sampled Session History
The sampled session history is constructed by sampling
the target session every 1 second. The sampling process
captures at each sample if the session is in a non-idle wait,
an idle wait, or not in a wait. If the session is in a
non-idle wait then one interval is shown for all the samples
the session was in the same non-idle wait. If the
session is in an idle wait or not in a wait for
consecutive samples then one interval is shown for all
the consecutive samples. Though we display these consecutive
samples in a single interval the session may NOT be continuously
idle or not in a wait (the sampling process does not know).
The history is displayed in reverse chronological order.
sample interval: 1 sec, max history 120 sec
KSFD PGA DUMPS
Number of completed I/O requests=0 flags=0
END OF PROCESS STATE
LMON IPC context:
ksxpdmp: facility 0 (?) (0x1, 0x0) counts 0, 0
ksxpdmp: Dumping the osd context
SKGXP: SKGXPCTX: 0x1103bfb58 ctx
SKGXP:
SKGXP: WAIT HISTORY
SKGXP: Time(msec)     Wait Type     Return Code
SKGXP: ----------     ---------     ------------
SKGXP: 0          NORMAL          SUCC
SKGXP: 0          NORMAL          SUCC
SKGXP: 0          NORMAL          SUCC
SKGXP: 0          NORMAL          SUCC
SKGXP: 0          NORMAL          TIMEDOUT
SKGXP: 12          NORMAL          TIMEDOUT
SKGXP: 0          NORMAL          TIMEDOUT
SKGXP: 20          NORMAL          TIMEDOUT
SKGXP: 0          NORMAL          TIMEDOUT
SKGXP: 19          NORMAL          TIMEDOUT
SKGXP: 0          NORMAL          TIMEDOUT
SKGXP: 20          NORMAL          TIMEDOUT
SKGXP: 0          NORMAL          TIMEDOUT
SKGXP: 19          NORMAL          TIMEDOUT
SKGXP: 0          NORMAL          TIMEDOUT
SKGXP: 20          NORMAL          TIMEDOUT
SKGXP: wait delta 0 sec (27 msec) ctx ts 0x3e377 last ts 0x3e381
SKGXP: user cpu time since last wait 0 sec 0 ticks
SKGXP: system cpu time since last wait 0 sec 0 ticks
SKGXP: locked 1
SKGXP: blocked 51
SKGXP: timed wait receives 0
SKGXP: admno 0x485303b1 admport:
SKGXP: SSKGXPT 0x103c0a74 flags sockno 12 IP 192.168.253.49 UDP 49777
SKGXP: context timestamp 0x3e377
SKGXP: buffers queued on port 1105aa950
SKGXP:
SKGXP: Dumping Connection Handle Table
SKGXP: sconno accono ertt state seq# RcvPid TotCreditsSKGXP: sent rtrans acks
SKGXP: CNH Table Bucket: 10
SKGXP: 0x339d0248 0x6dd6841c 64 4 32838 589900 8SKGXP: 75d 5d 32838d
SKGXP: CNH Table Bucket: 11
SKGXP: 0x339d0249 0x75ef4c98 32 4 32811 1007758 8SKGXP: 48d 12d 32811d
SKGXP: CNH Table Bucket: 12
SKGXP: 0x339d024a 0x75703ec2 16 4 32763 524518 8SKGXP: 0d 0d 0d
SKGXP: CNH Table Bucket: 13
SKGXP: 0x339d024b 0x41094259 16 4 32763 520260 8SKGXP: 0d 0d 0d
SKGXP: CNH Table Bucket: 14
SKGXP: 0x339d024c 0x7c1c696c 16 4 32763 585808 8SKGXP: 0d 0d 0d
SKGXP: CNH Table Bucket: 15
SKGXP: 0x339d024d 0x138c8c4a 16 4 32763 843952 8SKGXP: 0d 0d 0d
SKGXP:
SKGXP: Dumping Accept Handle Table
SKGXP: ach accono sconno admno state SndPid seq# rcv rtrans acks credits
SKGXP: ACH Table Bucket: 1472
SKGXP: 0x111088010 0x48cb4387 0x3365b236 0x1fe7dc68 40 1007758 32812 49 0 26 8
SKGXP: ACH Table Bucket: 1474
SKGXP: 0x11108b730 0x48cb4389 0x1c69654a 0x7183ff4c 40 589900 32838 75 0 52 8
Incident 116865 created, dump file: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116865/+ASM1_lmon_860280_i116865.trc
ORA-29740: evicted by member 1, group incarnation 8
error 29740 detected in background process
ORA-29740: evicted by member 1, group incarnation 8
*** 2010-03-01 16:54:46.430
LMON (ospid: 860280): terminating the instance due to error 29740
ksuitm: waiting up to [5] seconds before killing DIAG
==========
DIAG trace files
=========
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /oradb/oracle/product/11.1/asm_1
System name:     AIX
Node name:     host-node1
Release:     1
Version:     6
Machine:     00C39EA44C00
Instance name: +ASM1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 4
Unix process pid: 614488, image: oracle@host-node1 (DIAG)
*** 2010-03-01 16:50:22.947
*** SESSION ID:(222.1) 2010-03-01 16:50:22.947
*** CLIENT ID:() 2010-03-01 16:50:22.947
*** SERVICE NAME:() 2010-03-01 16:50:22.947
*** MODULE NAME:() 2010-03-01 16:50:22.947
*** ACTION NAME:() 2010-03-01 16:50:22.947
Node id: 0
List of nodes: 0, 1, 2,
*** 2010-03-01 16:50:22.948
Reconfiguration starts [incarn=0]
*** 2010-03-01 16:50:22.948
I'm the master node
Group reconfiguration cleanup
*** 2010-03-01 16:50:23.602
A rcfg proposal from node 2 is received
*** 2010-03-01 16:50:23.602
A rcfg proposal from node 1 is received
*** 2010-03-01 16:50:23.602
Reconfiguration completes [incarn=3]
*** 2010-03-01 16:53:00.877
A dump event msg is rcv'd
REQUEST:trace dump in directory cdmp_20100301165301
*** 2010-03-01 16:53:00.877
Trace dumping is performing id=[cdmp_20100301165301]....
*** 2010-03-01 16:53:01.041
Trace dumping is done
*** 2010-03-01 16:54:46.560
Instance is terminating by process 860280 [ospid=oracle@host-node1 (LMON)]
Performing diagnostic data dump for this instance
Incident 116833 created, dump file: /oradb/oracle/diag/asm/+asm/+ASM1/incident/incdir_116833/+ASM1_diag_614488_i116833.trc
ORA-29740: evicted by member , group incarnation
Error 29740 encountered during system state dump
*** 2010-03-01 16:54:49.280
----- Error Stack Dump -----
ORA-29740: evicted by member , group incarnation
*** 2010-03-01 16:54:49.281
Trace dumping is performing id=[cdmp_20100301165446]....
*** 2010-03-01 16:54:49.433
Trace dumping is done