How does 1 ASM instance in a 10 node RAC cause all 10 ASM instance to hang

Question

Linux RHEL4
11.2.0.1.0 GI
11.2.0.1.0 RDBMS
11.1.0.7.0 RDBMS
10.2.0.4.4 RDBMS
Yesterday we had one of the ASM instance on our 10-node cluster hang with a latch "ASM file allocation latch". Oracle support has stated we are hitting a bug with is fixed in the latest PSU.
What I am trying to understand is how a latch on one ASM instance caused all 10 ASM instances to hang?
Oracle Supports explanation still does not answer how all 10 ASM instance were affected other then the faulty code that the patch fixes. Bellow is info out of the SAR.
Data Guard Site:
================
ARC1 PID 21508 reports ORA-240 'control file enqueue held for more than 120 seconds' at Thu Oct 28 11:35:54 2010. This message means ARC1 PID 21508 was holding the enqueue past the maximum tine limit of 120 seconds.
Next ARC0 PID 17398 reports ORA-16146: 'control file enqueue unavailable' at Thu Oct 28 11:36:52 2010. This means ARC0 PID 17398 tried to access the controlfile and it could get the lock.
ARC1 PID 21508 is then killed at Thu Oct 28 11:49:57 2010.
Killing enqueue blocker (pid=21508) on resource CF-00000000-00000000 by (pid=23078) by killing session 5.3
We see that RSM has been restarted by Broker. This matches to the alert.log and messages showing RSM being restarted several time. i.e. RSM0 started with pid=84, OS id=27915
Since this started happening around 11:14, the problem may have really started then instead of w/ the CF enqueue block.
The primary appears to be shipping redo to the standby using asynchronous mode. Unless there was a gap, I do no know if ARC1 should have been doing any remote network I/O. Because a log switch to thread 1 sequence 5235 had just happened at 11:33am, I suspect it was a local disk I/O that was started at least. The ORA-240 is reported at 11:35, exactly 120 sec (or 2min) from this log switch. Notice also the entry to show the log thread 1 sequence 5234 was registered to the controlfile didn't happen yet.
ASM Side:
============
From the systemstate dump on ASM, multiple processes are waiting on 'ASM file allocation latch' and we are kind of stuck on stack:
ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1782<-sspuser()+112<-0000003218E0C5B0<-kfuhInsert()+175<-kffilCreate()+601<-kfnsUFG()+3691<-kfnsBackground()+4382<-kfnDispatch()+527<-opiodr()+1149<-ttcpip()+1251<-opitsk()+1633<-opiino()+958<-opiodr()+1149<-opidrv()+570<-sou2o()+103<-opimai_real()+133<-ssthrdmain()+214<-main()+201<-__libc_start_main()+219
This caused the hang on the ASM side and hence ASM was not responding to the database. The CF Enqueue issue on the database was a side effect of this.
This is a direct <BUG:9232266> which is a duplicate of <BUG:8974548>

Answer 1

Thanks Murali,
I was little reluctent to open a tar since my experince says that its better to troubleshoot the issue yourself instead of spending time with the Oracle support :)
Well, I got some more information on this : the sql for the process id of ASM instance is :
Alter Diskgroup mount all;
It looks like it is unable to mount the diskgroup and get hunged at that point , is it due to the new disk added to the system or do we missed something after adding the new disk.
Also, the ASM process can not be killed from unix box while the ASM is down.
Will appreciate your time if we can move in any directions from here ..
Thanks ,
Ankur