Oracleasm listdisks is not listing all disk groups and returning "strange special characters" and disks groups are dismounting unexpected

Question

Hi everybody,
I have a 3 node Oracle Cluster with verstion 11.2.0.3.0 installed using ASM/ACFS/ADVM.
Red Hat 5.7 - Kernel 2.6.18-371.12.1.el5 64bits
OracleASM version:
     oracleasm-support-2.1.8-1.el5
     oracleasm-2.6.18-371.12.1.el5-2.0.5-1.el5
     oracleasmlib-2.0.4-1.el5
On may, 2014 we had a physical failure in our storage system and all our disk partitions were completely lost.
The Oracle Grid Infrastructure and Oracle RDBMS haven't needed to be reinstalled because they were healthy.
The Oracle Cluster Registry (OCR) was lost (it was composed by three ASM disk groups in a redundant structure) and after recreation of the OCR disk groups I recover the Oracle Registry information from backup.
All cluster information were recovered (listener, SCAN, VIP, databases and services)
Databases could not be started because all ASM disk groups have been lost and oracle ASM instance (asmcmd -p) was showing all disk groups, but they were all "empty" - the physical partition related with them were recreated.
All the "old" disk groups were dropped, and the "new" disk groups were recreated and all the backups from the databases were restored.
After all services were up, and the environment seems to be OK, I check the healthiness of it, by executing the oracle cluster post-installation tool and others verify tools (crsctl and srvctl), well all returning messages saying that the environment was completely OK and no errors were found or generated (including in many log files I have inspected)
After 2 weeks from the completely recover of the environment, without any error/failure message been received from it, the disk groups started to dismount unexpectedly, from whatever node and not only one specific disk group stops, but any one of them, any time.
I opened a SR in Metalink (Oracle Support) and they conduct me to upgrade RedHat kernel version and OracleASM version.
Simultaneously I opened a Case Solution in RedHat to help me identify the better way to do the upgrade in advantage of the Oracle specifications.
From:
rpm -qa | grep kernel
kernel-devel-2.6.18-274.el5
kernel-headers-2.6.18-274.el5
kernel-2.6.18-274.el5
rpm -qa | grep oracleasm
oracleasm-support-2.1.8-1.el5
oracleasmlib-2.0.4-1.el5
oracleasm-2.6.18-274.el5-2.0.5-1.el5
To:
rpm -qa | grep kernel
kernel-devel-2.6.18-371.12.1.el5
kernel-2.6.18-371.12.1.el5
kernel-headers-2.6.18-371.12.1.el5
rpm -qa | grep oracleasm
oracleasm-support-2.1.8-1.el5
oracleasm-2.6.18-371.12.1.el5-2.0.5-1.el5
oracleasmlib-2.0.4-1.el5
Well, after this upgrade less times the disk groups dismounted, but they continue dismounting from whatever node, any disk group, any time.
So, I opened another SR in Metalink (3-10143566371 still open) and they are conducting me again to upgrade the RedHat kernel version and OracleASM version.
Information about OracleASM:
# oracleasm listdisks
ACFS
FRA
OCR01
OCR02
OCR03
$ asmcmd -p
ASMCMD [+] > ls -l
State    Type    Rebal  Name
MOUNTED  EXTERN  N      ACFS/
MOUNTED  EXTERN  N      ARCH/
MOUNTED  EXTERN  N      DEV/
MOUNTED  EXTERN  N      DIR_LOG/
MOUNTED  EXTERN  N      JAVA_ARCHIVE/
MOUNTED  EXTERN  N      MSAF/
MOUNTED  NORMAL  N      OCR/
MOUNTED  EXTERN  N      PRD_DATA/
MOUNTED  EXTERN  N      PRD_REDO1/
MOUNTED  EXTERN  N      PRD_REDO2/
MOUNTED  EXTERN  N      PRD_REDO3/
MOUNTED  EXTERN  N      PRD_REDO4/
MOUNTED  EXTERN  N      RASTR/
MOUNTED  EXTERN  N      STAGE/
This is what happens when execute "oracleasm scandisks":
# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ùìÿÿÿÿÿÿÿÿÿÿÿìóæþÿ¥Ï¨®Ð¢"
Unable to instantiate disk "ùìÿÿÿÿÿÿÿÿÿÿÿìóæþÿ¥Ï¨®Ð¢"
Instantiating disk "ü´ñõ
                        ñõúìsö"
Unable to instantiate disk "ü´ñõ
                                ñõúìsö"
~±Ã1~u·}cÿ-Ûg disk "
¾s
~±Ã1~u·}cÿ-Ûinstantiate disk "
¾s
  Instantiating disk "à·Öªß³Ö½Þ®hìëÖßßÞÖÛÜØÖÕ"
Unable to instantiate disk "à·Öªß³Ö½Þ®hìëÖßßÞÖÛÜØÖÕ"
Instantiating disk "êËbkî,c
~,XZ±´b¹u²´biÅr"
Unable to instantiate disk "êËbkî,c
~,XZ±´b¹u²´biÅr"
Instantiating disk "
PuTTYPuTTYUnable to instantiate disk "
kÍ3|úùõ/øÊInstantiating disk "u
i)ïìL"
kÍ3|úùõ/øÊe to instantiate disk "u
My question is: "This will really solve the problem?"
Another: "Oracle really understood what is happening and they can help me to solve this problem?"
Final: "Can anyone please help me this problem?"

Answer 1

Hi,
my two cent.
Wait for the SR feedback.
But for the future I advice you to do an ASM Metadata backup to be able to restore this data if you lost
configurations or complete disks and so on.
asmcmd md_backup -b <Pfad>/Filename
and also an ocrconfig backup via crontab.
ocrconfig -local -manualbackup
regards