使用AMDU工具从无法MOUNT的DISKGROUP中抽取数据文件

AMDU是ORACLE针对ASM开发的源数据转储工具，其全称为ASM Metadata Dump Utility(AMDU)
AMDU具体以下三个主要功能：
1. 将ASM DISK上的元数据转储到文件系统上以便分析
2. 将ASM文件的内容抽取出来并写入到OS文件系统，Diskgroup是否mount均可
3. 打印出块的元数据，以块中C语言结构或16进制的形式
这里我们将用到使用AMDU抽取ASM DISKGROUP中的数据文件； ASM作为近几年最流行的存储解决方案， 大家对他的优缺点都有所了解，其中的问题之一就是ASM是个黑盒。 一旦DISKGROUP无法MOUNT起来就意味着传统方法无法以磁盘为基础导出任何数据。
AMDU解决了这一问题， 这里我们仅讨论在ASM DISKGROUP 无法MOUNT的情况下的范畴，不讨论RDBMS数据文件在ASM下讹误的处理。
注意 AMDU虽然是11g才发布的工具，但是实际对10g的ASM 也有效。
当前你可能遇到的场景是， ORACLE DATABASE的SPFILE、CONTROLFILE、DATAFILE均存放在ASM DISKGROUP中，而由于一些ASM ORA-600错误导致无法MOUNT该DISKGROUP， 你需要的是使用AMDU将这些文件从ASM DISK中转储出来。
场景 1 丢失了 包括SPFILE、CONTROLFILE、DATAFILE
恢复步骤： 从备份中还原出SPFILE ，即便没有SPFILE的话PFILE也可以，总之你需要从参数文件中了解control_files的信息
SQL> show parameter control_files
NAME                                 TYPE        VALUE
control_files                        string      +DATA/prodb/controlfile/curren
                                                 t.260.794687955, +FRA/prodb/co
                                                 ntrolfile/current.256.79468795
                                                 5获得control_files 控制文件在ASM中的位置后事情就好办了，+DATA/prodb/controlfile/current.260.794687955 这里 260是这个控制文件在+DATA 这个DISKGROUP中的FILE NUMBER
此外我们还需要ASM DISK的DISCOVERY PATH信息，这完全可以从ASM的SPFILE中的asm_diskstring 参数获得
[oracle@mlab2 oracle.SupportTools]$ unzip amdu_X86-64.zip
Archive:  amdu_X86-64.zip
  inflating: libskgxp11.so          
  inflating: amdu                   
  inflating: libnnz11.so            
  inflating: libclntsh.so.11.1      
[oracle@mlab2 oracle.SupportTools]$ export LD_LIBRARY_PATH=./
[oracle@mlab2 oracle.SupportTools]$ ./amdu -diskstring '/dev/asm*' -extract data.260
amdu_2009_10_10_20_19_17/
AMDU-00204: Disk N0006 is in currently mounted diskgroup DATA
AMDU-00201: Disk N0006: '/dev/asm-disk10'
AMDU-00204: Disk N0003 is in currently mounted diskgroup DATA
AMDU-00201: Disk N0003: '/dev/asm-disk5'
AMDU-00204: Disk N0002 is in currently mounted diskgroup DATA
AMDU-00201: Disk N0002: '/dev/asm-disk6'
[oracle@mlab2 oracle.SupportTools]$ cd amdu_2009_10_10_20_19_17/
[oracle@mlab2 amdu_2009_10_10_20_19_17]$ ls
DATA_260.f  report.txt
[oracle@mlab2 amdu_2009_10_10_20_19_17]$ ls -l
total 9548
-rw-r--r-- 1 oracle oinstall 9748480 Oct 10 20:19 DATA_260.f
-rw-r--r-- 1 oracle oinstall    9441 Oct 10 20:19 report.txt以上转储出来的DATA_260.f 就是控制文件，我们使用该控制文件startup mount RDBMS实例：
SQL> alter system set control_files='/opt/oracle.SupportTools/amdu_2009_10_10_20_19_17/DATA_260.f' scope=spfile;
System altered.
SQL> startup force mount;
ORACLE instance started.
Total System Global Area 1870647296 bytes
Fixed Size                  2229424 bytes
Variable Size             452987728 bytes
Database Buffers         1409286144 bytes
Redo Buffers                6144000 bytes
Database mounted.
SQL> select name from v$datafile;
NAME
+DATA/prodb/datafile/system.256.794687873
+DATA/prodb/datafile/sysaux.257.794687875
+DATA/prodb/datafile/undotbs1.258.794687875
+DATA/prodb/datafile/users.259.794687875
+DATA/prodb/datafile/example.265.794687995
+DATA/prodb/datafile/mactbs.267.794688457
6 rows selected.startup mount实例后，可以从v$datafile中获得数据文件名，其中就包括了其在DISKGROUP中的FILE NUMBER
再使用./amdu -diskstring '/dev/asm*' -extract 命令即可 导出数据文件到操作系统
[oracle@mlab2 oracle.SupportTools]$ ./amdu -diskstring '/dev/asm*' -extract data.256
amdu_2009_10_10_20_22_21/
AMDU-00204: Disk N0006 is in currently mounted diskgroup DATA
AMDU-00201: Disk N0006: '/dev/asm-disk10'
AMDU-00204: Disk N0003 is in currently mounted diskgroup DATA
AMDU-00201: Disk N0003: '/dev/asm-disk5'
AMDU-00204: Disk N0002 is in currently mounted diskgroup DATA
AMDU-00201: Disk N0002: '/dev/asm-disk6'
[oracle@mlab2 oracle.SupportTools]$ cd amdu_2009_10_10_20_22_21/
[oracle@mlab2 amdu_2009_10_10_20_22_21]$ ls
DATA_256.f  report.txt
[oracle@mlab2 amdu_2009_10_10_20_22_21]$ dbv file=DATA_256.f
DBVERIFY: Release 11.2.0.3.0 - Production on Sat Oct 10 20:23:12 2009
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
DBVERIFY - Verification starting : FILE = /opt/oracle.SupportTools/amdu_2009_10_10_20_22_21/DATA_256.f
DBVERIFY - Verification complete
Total Pages Examined         : 90880
Total Pages Processed (Data) : 59817
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 12609
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 3637
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14817
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 1125305 (0.1125305)@@
如何在OTN中文技术论坛提一个问题？
论坛礼仪需知及学习oracle的方法论
Maclean Liu
Oracle Database Administrator
Oracle Certified 10g/11g Master     
www.askmaclean.com

楼主，你好。我们现在也是按此方 案正在进行，但有一个技术点
./amdu -diskstring '/dev/asm*' -extract data.260
asm 实例不能mount 起来
其中260 这个号是asm filenumber 有的数据文件是名字是这样的
+DG_ORA/mos5200/datafile/pm4h_db94.dbf
从文件名上不能看到260这样的number
可能是因为asm不能mount ,从v$asm_file 里也只能看到部分文件
oracle 提供了一个方案把extract 出来的文件对其block0 dump,可是asm 里的文件太多了归档也放在里面了，不太具有可行性
如何解决呢,ML想个办法,谢谢

Similar Messages

  • Mount Diskgroup without one disk

    Hi All
    We have on database of 6TB on raw devices. it's 10gr2 on IBM-AIX. We have plan to move raw to ASM. While going through recommendations it seems Oracle suggest
    to have 2 Disk Groups. One for datafiles other for flash area.
    We got nearly 25 LUNs assigned to this servers, LUNs are mapped to PV->VG Iwe have one to one mapping ) then VG have multiple logical volume which are being used as raw.
    My question here is, if we have problem with LUN ( some corruption,yes, we have raid but please help to me understand theoretically) then we would loose few datafiles which are mapped to PV->VG.
    But if we use same architecture in ASM where 25 LUNS assigned to one Diskgroup then whole diskgroup will not be mount which means whole db will be down.
    Is there something I am missing ?
    Regards,
    Lyxx

    can you please explain what you refer to as PV and VG?
    If you use ASM to stripe over 25 disks/volumes/luns and you loose any one of them, your database will be unavailable. The slight advantage here is that all your data is striped over all disks which should result in better performance. But you are right, the diskgroup would not mount. Also, the mapping of multiple volumes for one physical disk (or raid) is not a good idea since it will not improve performance.
    But you could also have ASM do some mirroring itself. For example, you could assign 24 disks/volumes to asm and mirror your data across 2 failover groups with 12 disks each. Of course, this would mean you'd have to use even more disks.
    Or, if you want to stay as close to your current deployment as possible, you could setup 25 diskgroups and map them just the way you are mapping them now. In that way, when you loose a disk/volume/lun you will only loose one diskgroup and if the datafiles in that diskgroup are not critical, your database could still be up.
    Bjoern

  • ASM instance wont mount diskgroup..

    HI, I have 10g release 2 installed on CENTOS 4.4, I use ASM striping with with 4 raw disks.
    I had a system crash due to a power failure and now the the ASM wont mount the diskgroup.
    export $ORACLE_HOME=+ASM
    SQL> startup mount;
    ASM instance started
    Total System Global Area 130023424 bytes
    Fixed Size 2071000 bytes
    Variable Size 102786600 bytes
    ASM Cache 25165824 bytes
    ORA-15110: no diskgroups mounted
    SQL> alter diskgroup RESEARCH1 mount;
    alter diskgroup RESEARCH1 mount
    ERROR at line 1:
    ORA-15032: not all alterations performed
    ORA-15063: ASM discovered an insufficient number of disks for diskgroup
    "RESEARCH1"
    now when I use /etc/init.d/oracleasm listdisks I can see all my disks:
    DISK1
    DISK2
    DISK3
    DISK4
    then i tried to change asm_diskstring to point the mounting point, here is my ora file:
    *.asm_diskgroups='RESEARCH1'
    +ASM.asm_diskgroups='RESEARCH1' #Manual Dismount
    *.asm_diskstring='/dev/oracleasm/disks'
    *.background_dump_dest='/home/oracle/product/10.2.0/db_1/admin/+ASM/bdump'
    *.core_dump_dest='/home/oracle/product/10.2.0/db_1/admin/+ASM/cdump'
    *.instance_type='asm'
    *.large_pool_size=12M
    *.remote_login_passwordfile='EXCLUSIVE'
    *.user_dump_dest='/home/oracle/product/10.2.0/db_1/admin/+ASM/udump'
    any ideas?
    Thanks
    Assaf

    Hi,
    by oracleasm lib utility you can configure as below
    # /etc/init.d/oracleasm configure
    Default user to own the driver interface [oracle]: oracle
    Default group to own the driver interface [dba]: dba
    Start Oracle ASM library driver on boot (y/n) [y]: y
    Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
    Writing Oracle ASM library driver configuration: [ OK ]
    Scanning system for ASM disks: [ OK ]
    # /etc/init.d/oracleasm enable
    Thanks

  • Install RAC:ORA-15063 error while mount diskgroup

    I have two nodes(linux1,linux2),but when I run DBCA on the computer of linux2 to create database,occur ORA-15063 error ,the detail information:
    could not mount the diskgroup on remote node linux1 using connection service linux1:21521+ASM1.Ensure that the listener is running on this node and the asm istance is registered to the listener.receive follow errors:
    ORA-15032:NOT ALL ALTERATION PERFORMED
    ORA-15063:ASM DISCOVERED ON INSUFFICIENT NUMBER OF DISK FOR DISKGROUP "ORCL_DATA1"
    who can help me!!!
    thanks in advance;

    there are the two file of the config:
    /u01/app/oracle/product/10.2.0/db_1/dbs/init+ASM1.ora
    SPFILE='/u02/oradata/orcl/dbs/spfile+ASM.ora'
    /u02/oradata/orcl/dbs/spfile+ASM.ora
    .asm_diskgroups=''
    +ASM1.asm_diskgroups='ORCL_DATA1'#Manual Mount
    .asm_diskstring='ORCL:VOL*'
    .background_dump_dest='/u01/app/oracle/admin/+ASM/bdump'
    *.cluster_database=true
    *.core_dump_dest='/u01/app/oracle/admin/+ASM/cdump'
    +ASM1.instance_number=1
    +ASM2.instance_number=2
    *.instance_type='asm'
    *.large_pool_size=12M
    *.remote_login_passwordfile='exclusive'
    *.user_dump_dest='/u01/app/oracle/admin/+ASM/udump'

  • Can't mount diskgroup on second RAC node (ss3)

    I have a newly created 11g (Oracle 11.1.0.6) RAC database consisting of two nodes (SS1 and SS3).
    Installed CRS and everything is working normal: crs_stat -t -v shows:
    Name Type R/RA F/FT Target State Host
    ora....Y1.inst application 0/5 0/0 ONLINE ONLINE ss1
    ora....Y2.inst application    0/5    0/0    ONLINE    OFFLINE
    ora.SY.db application 0/0 0/1 ONLINE ONLINE ss3
    ora....SM1.asm application 0/5 0/0 ONLINE ONLINE ss1
    ora....S1.lsnr application 0/5 0/0 ONLINE ONLINE ss1
    ora.ss1.gsd application 0/5 0/0 ONLINE ONLINE ss1
    ora.ss1.ons application 0/3 0/0 ONLINE ONLINE ss1
    ora.ss1.vip application 0/0 0/0 ONLINE ONLINE ss1
    ora....SM2.asm application 0/5 0/0 ONLINE ONLINE ss3
    ora....S3.lsnr application 0/5 0/0 ONLINE ONLINE ss3
    ora.ss3.gsd application 0/5 0/0 ONLINE ONLINE ss3
    ora.ss3.ons application 0/3 0/0 ONLINE ONLINE ss3
    ora.ss3.vip application 0/0 0/0 ONLINE ONLINE ss3
    Installed ASM in it's own ORACLE_HOME. From ss1 I see:
    SELECT group_number, disk_number, mount_status, header_status, state, path FROM v$asm_disk;
    GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE PATH
    1 0 CACHED MEMBER NORMAL /dev/rdsk/c4t600C0FF0000000000B5DB04B603F9500d0s6
    2 0 CACHED MEMBER NORMAL /dev/rdsk/c4t600C0FF0000000000B5DB04B603F9500d0s7
    SELECT name, label FROM v$asm_disk;
    NAME LABEL
    DATA_0000
    RECOVERY_0000
    But, from ss3 I see:
    SELECT group_number, disk_number, mount_status, header_status, state, path FROM v$asm_disk;
    no rows selected
    SELECT name, label FROM v$asm_disk;
    NAME LABEL
    DATA_0000
    RECOVERY_0000
    ALTER DISKGROUP DATA MOUNT;
    ERROR at line 1:
    ORA-15032: not all alterations performed
    ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
    I am using multipathed I/O ~ which I believe is significant from the info on multiplexed I/O in the Storage Administrator's Guide here:
    [http://download.oracle.com/docs/cd/B28359_01/server.111/b31107/asmprepare.htm#BHCJBJFD]
    /u01/app/product/11.1.0/asm/dbs/init.ora on both nodes has:
    asm_diskstring='/dev/rdsk/*'
    Any troubleshooting help is much appreciated.

    Just for information:
    make sure the owner of the database can read/write in the ASM devices.
    i.o.w. If you have installed ASM under a different user than your database (for example - asm is installed with oracle and your database with oradb), you must make sure user oradb can read/write in the devices as well. Otherwise, you won't be able to startup your database either. Solve this by putting both users in the dba group and give the dba group read/write permission on the devices.
    Arnoud

  • ASM mount diskgroup

    Hi All
    I am using 11gR2 but when I start the grid infrastructure using "crsctl start crs" then the ASM instance get started but the disk group did not get mounted. My question how can I get the ASM disk groups get mounted when I start the grid infrastructure ?
    Thanks

    Hi,
    by oracleasm lib utility you can configure as below
    # /etc/init.d/oracleasm configure
    Default user to own the driver interface [oracle]: oracle
    Default group to own the driver interface [dba]: dba
    Start Oracle ASM library driver on boot (y/n) [y]: y
    Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
    Writing Oracle ASM library driver configuration: [ OK ]
    Scanning system for ASM disks: [ OK ]
    # /etc/init.d/oracleasm enable
    Thanks

  • Diskgroup not mounted during cluster startup

    Hi,
    I have a 2 node RAC(11GR2) on VMWare 7.1.4. OS is Solaris 10
    I have registered 2 instances in the cluster.
    srvctl add database -d dbrac -o /u01/app/oracle/product/11.2.0/dbhome_1 -a "extdg,nordg"
    srvctl add instance -d dbrac -i dbrac2 -n vmsol2
    srvctl add instance -d dbrac -i dbrac1 -n vmsol1
    After after registering the 2 instances, initially the instance were automatically up when ever I execute a ./crsctl start cluster.
    But now the database instances on both nodes are not comming up; only ASM is up.
    While checking ASM disk group, I found EXTDG and NORDG is not mounted. So I mounted the disk group and started the database.
    Then I manually mounted, the disk groups and started the database.
    SQL> alter diskgroup nordg mount;
    Diskgroup altered
    SQL> alter diskgroup extdg mount;
    Later I tried removing the the database configuration from the cluster using SRVCTL and added the database to cluster again.
    srvctl remove instance -d dbrac -i dbrac1
    srvctl remove instance -d dbrac -i dbrac2
    srvctl remove database -d dbrac
    srvctl add database -d dbrac -o /u01/app/oracle/product/11.2.0/dbhome_1 -a "extdg,nordg"
    srvctl add instance -d dbrac -i dbrac2 -n vmsol2
    srvctl add instance -d dbrac -i dbrac1 -n vmsol1
    Still database is not starting during cluster startup.
    Why did the disk group is not getting mounted while cluster is starting? Can someone help me?
    Regards,
    Mat
    Edited by: user11278087 on Mar 9, 2012 7:37 PM
    Edited by: user11278087 on Mar 9, 2012 7:41 PM
    Edited by: user11278087 on Mar 9, 2012 7:42 PM

    Hi,
    Thank you for your replay.
    Disk group was mounting previously without doing any modification in spfile after executing the following commands.
    srvctl add database -d dbrac -o /u01/app/oracle/product/11.2.0/dbhome_1 -a "extdg,nordg"
    srvctl add instance -d dbrac -i dbrac2 -n vmsol2
    srvctl add instance -d dbrac -i dbrac1 -n vmsol1
    But suddenly this issue happened.
    I do not have metalink access. Could you please help me?
    Regards,
    Mat.

  • Create Diskgroup Fails

    Count not mount the diskgroup.
    SQL> alter diskgroup stag_data mount;
    alter diskgroup stag_data mount
    ERROR at line 1:
    ORA-15032: not all alterations performed
    ORA-15063: ASM discovered an insufficient number of disks for diskgroup
    "STAG_DATA"
    Could not add the disk, since it is not mounted. (Catch 22)
    SQL> ALTER DISKGROUP stag_data add disk '/dev/rdsk/c1txxxxxxxxs6';
    ALTER DISKGROUP stag_data add disk '/dev/rdsk/c1txxxxxxxxs6'
    ERROR at line 1:
    ORA-15032: not all alterations performed
    ORA-15001: diskgroup "STAG_DATA" does not exist or is not mounted
    SQL> show parameter asm_diskstring
    NAME TYPE VALUE
    asm_diskstring string
    SQL> select name, state, type, total_mb, free_mb from v$asm_diskgroup;
    NAME STATE TYPE TOTAL_MB FREE_MB
    STAG_DATA DISMOUNTED 0 0
    STAG_FLASH MOUNTED EXTERN 102093 102000
    Oracle Metalink solution was to
    1) Drop and recreate the raw devices.
    Note: Recreate the raw devices is required in order to remove completely the ASM metadata.
    dd if=/dev/zero of=/dev/rdsk/<device_name> bs=1024 count=100
    dd if=/dev/zero of=/dev/rdsk/<device_name> bs=8192 count=2560
    But this did not fix this issue. Any Thoughts
    Thanks
    -Prasad

    How do i remove invalid disk names and where is this info stored in the ASM.
    export ORACLE_SID=+ASM1
    SQL> startup
    ASM instance started
    Total System Global Area 130023424 bytes
    Fixed Size 2028368 bytes
    Variable Size 102829232 bytes
    ASM Cache 25165824 bytes
    ORA-15032: not all alterations performed
    ORA-15024: discovered duplicately numbered ASM disk 0
    SQL> select name, state, type, total_mb, free_mb from v$asm_diskgroup;
    NAME STATE TYPE TOTAL_MB FREE_MB
    STAG_DATA DISMOUNTED 0 0
    STAG_FLASH MOUNTED EXTERN 102093 101759
    SQL> alter system set asm_diskstring='/dev/rdsk/c1t500*';
    System altered.
    SQL> alter diskgroup stag_data mount;
    Diskgroup altered.
    SQL> select name, state, type, total_mb, free_mb from v$asm_diskgroup;
    NAME STATE TYPE TOTAL_MB FREE_MB
    STAG_DATA MOUNTED EXTERN 95625 92541
    STAG_FLASH MOUNTED EXTERN 102093 101759
    Now i want to drop the invalid member of the diskgroup. But could not drop it.
    SQL> ALTER DISKGROUP STAG_DATA drop DISK '/dev/rdsk/c0t50060E800043A501d247s6';
    ALTER DISKGROUP STAG_DATA drop DISK '/dev/rdsk/c0t50060E800043A501d247s6'
    ERROR at line 1:
    ORA-01948: identifier's name length (35) exceeds maximum (30)
    SQL> ALTER DISKGROUP STAG_DATA drop DISK 'c0t50060E800043A501d247s6';
    ALTER DISKGROUP STAG_DATA drop DISK 'c0t50060E800043A501d247s6'
    ERROR at line 1:
    ORA-15032: not all alterations performed
    ORA-15054: disk "C0T50060E800043A501D247S6" does not exist in diskgroup
    "STAG_DATA"
    SQL> shutdown
    ASM diskgroups dismounted
    ASM instance shutdown
    SQL> startup
    ASM instance started
    Total System Global Area 130023424 bytes
    Fixed Size 2028368 bytes
    Variable Size 102829232 bytes
    ASM Cache 25165824 bytes
    ORA-15032: not all alterations performed
    ORA-15024: discovered duplicately numbered ASM disk 0
    Diskgroup is invalid on every reboot. Where is this info stored in ASM
    Thanks
    -Prasad

  • Unable to Create ASM Diskgroup ORA-15020 and ORA-15018

    Hello Team,
    Unable to create ASM diskgroup with following error:
    SQL> create diskgroup data_asm1 external redundancy disk '/dev/sdf*';
    create diskgroup data_asm1 external redundancy disk '/dev/sdf*'
    ERROR at line 1:
    ORA-15018: diskgroup cannot be created
    ORA-15020: discovered duplicate ASM disk "DATA_ASM1_0000"
    ASM Diskstring
    SQL> show parameter asm_diskstring
    NAME                                 TYPE        VALUE
    asm_diskstring                       string      /dev/oracleasm/disks/DISK*, /dev/sd*
    Please let me know how to i solve this issue
    Regards,

    Hi Tobi,
    I checked the status of the res GRID.dg ... it was offline on second node. Logged on second node and checked the status of it viz:v$asm_diskgroup, it was dismount. I mounted it and then try to add the newly added diskgroup(+GRID) with OCR and viola it worked....
    ========================================================
    ora.GRID.dg
                   ONLINE  ONLINE       rac3                                       
                   OFFLINE OFFLINE      rac4                                       
    SQL> select group_number,name,state,type from v$asm_diskgroup;
    GROUP_NUMBER NAME                           STATE       TYPE
               1 DATA                           MOUNTED     EXTERN
               0 GRID                           DISMOUNTED
    SQL> alter diskgroup grid mount;
    Diskgroup altered.
    SQL>  select group_number,name,state,type from v$asm_diskgroup;
    GROUP_NUMBER NAME                           STATE       TYPE
               1 DATA                           MOUNTED     EXTERN
               2 GRID                           MOUNTED     EXTERN
    ==============================================
    ora.GRID.dg
                   ONLINE  ONLINE       rac3                                       
                   ONLINE  ONLINE       rac4                                       
    ===============================================
    [root@rac3 bin]# ./ocrcheck
    Status of Oracle Cluster Registry is as follows :
             Version                  :          3
             Total space (kbytes)     :     262120
             Used space (kbytes)      :       2804
             Available space (kbytes) :     259316
             ID                       :   48011651
             Device/File Name         :      +DATA
                                        Device/File integrity check succeeded
             Device/File Name         :      +grid
                                        Device/File integrity check succeeded
                                        Device/File not configured
                                        Device/File not configured
                                        Device/File not configured
             Cluster registry integrity check succeeded
             Logical corruption check succeeded
    ==========================================================================================
    ASMCMD> lsdg
    State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
    MOUNTED  EXTERN  N         512   4096  1048576     20472    16263                0           16263              0             N  DATA/
    MOUNTED  EXTERN  N         512   4096  1048576      5114     4751                0            4751              0             N  GRID/
    ======================================================================================================
    Thank you very much, appreciated..
    Thank you Aritra .
    Guys you rock.
    Regards,

  • How to drop the last diskgroup in ASM 11g?

    Hello,
    I installed an ASM instance and created a DATA diskgroup using the Oracle installer. I would like to drop the diskgroup and re-create it using partitions instead of disk devices in order to use ASMLib.
    I have not found a way to get rid of the DATA diskgroup yet. How do I get rid of it?
    [+ASM@asm]$ sqlplus / as sysasm
    SQL*Plus: Release 11.2.0.1.0 Production on Fri Sep 3 20:12:05 2010
    Copyright (c) 1982, 2009, Oracle.  All rights reserved.
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
    With the Automatic Storage Management option
    SQL> DROP DISKGROUP data INCLUDING CONTENTS;
    DROP DISKGROUP data INCLUDING CONTENTS
    ERROR at line 1:
    ORA-15039: diskgroup not dropped
    ORA-15027: active use of diskgroup "DATA" precludes its dismountI used the following for data_0000 - data_0005, thinking it would be a good idea to drop each disk before dropping the diskgroup - which worked, although I had to use force for data_0000;
    SQL> alter diskgroup data drop disk data_0006;
    ... etc...
    SQL> alter diskgroup data drop disk data_0000 force;It all looks a bit weired now:
    SQL> select name, header_status, path from v$asm_disk;
                        MEMBER        /dev/sda
    _DROPPED_0000_DATA  UNKNOWN
    DATA_0001   MEMBER        /dev/sdb
    DATA_0002   MEMBER        /dev/sdc
    DATA_0003   MEMBER        /dev/sdd
    DATA_0004  MEMBER         /dev/sde
    DATA_0005   MEMBER        /dev/sdf
    SQL> alter diskgroup data dismount force;
    Diskgroup altered.
    SQL> alter diskgroup data mount;
    Diskgroup altered.
    SQL>

    Thanks for the reply.
    Unfortunately I cannot use EM since I do not have any database installed that uses the ASM instance yet, and to my knowledge ASM alone does not provide a web console.
    Anyway, I finally just manged to drop the diskgroup. Normally a diskgroup needs to be mountedto be dropped, but the force clause can be used for a diskgroup that is dismounted. So I tried it with the diskgroup dismounted... and it worked.
    SQL> alter diskgroup data dismount force;
    SQL> drop diskgroup data force including contents;
    Diskgroup dropped.
    Thanks!
    Edited by: Markus Waldorf on Sep 3, 2010 6:13 PM

  • Oracel 11gR1 RAC Cluster issue

    We have 2-node Oracle 11gR2 RAC on HP-UX 11.31 environment. It was running lase 2 month without any issue.
    We got some netconfig issue, and node-1 got rebooted today. after the reboot cluster didn't not start on node-1, database is running on node-2.
    grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4529: Cluster Synchronization Services is online
    CRS-4534: Cannot communicate with Event Manager
    grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ crs_stat -t
    CRS-0184: Cannot communicate with the CRS daemon.
    grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ ocrcheck
    PROT-602: Failed to retrieve data from the cluster registry
    PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=301, loc=kgfokge
    AMDU-00301: Unable to open file tmp-AMIPOCR01.ocr
    AMDU-00204: Disk N0002 is in currently mounted diskgroup AMIPOCR01
    AMDU-00201: Disk N0002: '/dev/rdisk/ora_OCR
    ] [8]
    grid@hublhp4:/app/oracle/grid/product/11.2.0.1/log/hublhp4/crsd$ olsnodes -n
    hublhp4 1
    hublhp5 2
    any idea please.
    Edited by: ManoRangasamy on Jul 5, 2011 6:38 PM

    Hi,
    Please post the alertlog ASM from node 1, crsd.log and ocssd.log from node 1
    It might be because node 1 can't see asm disk or permission accidentally changed when the node rebooted
    Cheers

  • Ocrcheck success on node1 but failed on node2.(11gR2 for Windows)

    Hi all , I have installed the Oracle Grid Infrastructure with ASM on windows 2008 x64, Everything is done without any error, but ocrcheck failed on nodde2. detail is below:
    checking commnds on node 1:
    *>crsctl check crs*
    CRS-4638: Oracle High Availability Services is online
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    *>srvctl status asm -a*
    ASM on rac1,rac2 is running
    ASM enable
    *>crs_stat -t*
    Name Type Target State Host
    ora.DATA.dg ora....up.type ONLINE ONLINE rac1
    ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
    ora....N1.lsnr ora....er.type ONLINE ONLINE rac2
    ora.asm ora.asm.type ONLINE ONLINE rac1
    ora.eons ora.eons.type ONLINE ONLINE rac1
    ora.gsd ora.gsd.type OFFLINE OFFLINE
    ora....network ora....rk.type ONLINE ONLINE rac1
    ora.oc4j ora.oc4j.type OFFLINE OFFLINE
    ora.ons ora.ons.type ONLINE ONLINE rac1
    ora....SM1.asm application ONLINE ONLINE rac1
    ora....C1.lsnr application ONLINE ONLINE rac1
    ora.rac1.gsd application OFFLINE OFFLINE
    ora.rac1.ons application ONLINE ONLINE rac1
    ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
    ora....SM2.asm application ONLINE ONLINE rac2
    ora....C2.lsnr application ONLINE ONLINE rac2
    ora.rac2.gsd application OFFLINE OFFLINE
    ora.rac2.ons application ONLINE ONLINE rac2
    ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
    ora.scan1.vip ora....ip.type ONLINE ONLINE rac2
    *>ocrcheck*
    Status of Oracle Cluster Registry is as follows :
    Version : 3
    Total space (kbytes) : 262120
    Used space (kbytes) : 2364
    Available space (kbytes) : 259756
    ID : 257699632
    Device/File Name : +DATA
    Device/File integrity check succeeded
    Device/File not configured
    Device/File not configured
    Device/File not configured
    Device/File not configured
    Cluster registry integrity check succeeded
    Logical corruption check succeeded
    checking commnds on node 2:
    *>crsctl check crs*
    CRS-4638: Oracle High Availability Services is online
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online
    *>srvctl status asm -a*
    ASM on rac1,rac2 is running
    ASM enable
    *>crs_stat -t*
    Name Type Target State Host
    ora.DATA.dg ora....up.type ONLINE ONLINE rac1
    ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
    ora....N1.lsnr ora....er.type ONLINE ONLINE rac2
    ora.asm ora.asm.type ONLINE ONLINE rac1
    ora.eons ora.eons.type ONLINE ONLINE rac1
    ora.gsd ora.gsd.type OFFLINE OFFLINE
    ora....network ora....rk.type ONLINE ONLINE rac1
    ora.oc4j ora.oc4j.type OFFLINE OFFLINE
    ora.ons ora.ons.type ONLINE ONLINE rac1
    ora....SM1.asm application ONLINE ONLINE rac1
    ora....C1.lsnr application ONLINE ONLINE rac1
    ora.rac1.gsd application OFFLINE OFFLINE
    ora.rac1.ons application ONLINE ONLINE rac1
    ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
    ora....SM2.asm application ONLINE ONLINE rac2
    ora....C2.lsnr application ONLINE ONLINE rac2
    ora.rac2.gsd application OFFLINE OFFLINE
    ora.rac2.ons application ONLINE ONLINE rac2
    ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
    ora.scan1.vip ora....ip.type ONLINE ONLINE rac2
    *>ocrcheck*
    PROT-602: Failed to retrieve data from the cluster registry
    PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=
    kgfolclcpi1, dep=204, loc=kgfokge
    AMDU-00204: Disk N0002 is in currently mounted diskgroup DATA
    AMDU-00201: Disk N0002: '\\.\ORCLDISKDATA1'
    ] [8]
    also, asmca can run on node1 but can not work on node 2.
    Can anyone help me to resolve this issue?
    Regards.
    Edited by: user8306020 on 2010-7-4 下午11:10

    Thank you for your replay.. I tried to run "cluvfy comp ocr -n all -verbose" on both node, and get the response like below:
    Node1:
    *>cluvfy comp ocr -n all -verbose*
    Verifying OCR integrity
    Checking OCR integrity...
    Checking the absence of a non-clustered configuration...
    All nodes free of non-clustered, local-only configurations
    ASM Running check passed. ASM is running on all cluster nodes
    Disk group for ocr location "+DATA" available on all the nodes
    Checking size of the OCR location "+DATA" ...
    rac2:Size check for OCR location "+DATA" successful...
    rac1:Size check for OCR location "+DATA" successful...
    WARNING:
    This check does not verify the integrity of the OCR contents. Execute 'ocrcheck'
    as a privileged user to verify the contents of OCR.
    OCR integrity check passed
    Verification of OCR integrity was successful.
    =========================================================================
    Node2:
    *>cluvfy comp ocr -n all -verbose*
    Verifying OCR integrity
    Checking OCR integrity...
    Checking the absence of a non-clustered configuration...
    All nodes free of non-clustered, local-only configurations
    ASM Running check passed. ASM is running on all cluster nodes
    Disk group for ocr location "+DATA" available on all the nodes
    Checking size of the OCR location "+DATA" ...
    rac2:Size check for OCR location "+DATA" successful...
    rac1:Size check for OCR location "+DATA" successful...
    WARNING:
    This check does not verify the integrity of the OCR contents. Execute 'ocrcheck'
    as a privileged user to verify the contents of OCR.
    OCR integrity check passed
    Verification of OCR integrity was successful.
    But ocrcheck failed on node2 again. the log file in \app\11.2.0\grid\log\rac2\client\orccheck_4844.log:
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2010-07-05 15:52:44.985: [OCRCHECK][4728]ocrcheck starts...
    2010-07-05 15:52:45.141: [    GPnP][4728]clsgpnp_Init: [at clsgpnp0.c:406] gpnp tracelevel 1, component tracelevel 0
    2010-07-05 15:52:45.141: [    GPnP][4728]clsgpnp_Init: [at clsgpnp0.c:536] 'E:\app\11.2.0\grid' in effect as GPnP home base.
    2010-07-05 15:52:45.157: [    GPnP][4728]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : E:\app\11.2.0\grid\gpnp\rac2\wallets\peer\
    [   CLWAL][4728]clsw_Initialize: OLR initlevel [70000]
    2010-07-05 15:52:45.173: [    GPnP][4728]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2 providers - fatal if all fail)
    2010-07-05 15:52:45.188: [    GPnP][4728]clsgpnp_getCK: [at clsgpnp0.c:1967] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1 of 2 (LSKP-FSW(1))
    2010-07-05 15:52:45.188: [    GPnP][4728]clsgpnp_getCK: [at clsgpnp0.c:1984] Got gpnp security keys (wallet).>
    2010-07-05 15:52:45.188: [    GPnP][4728]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2 providers - fatal if all fail)
    2010-07-05 15:52:45.188: [    GPnP][4728]clsgpnp_getCK: [at clsgpnp0.c:1967] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1 of 2 (LSKP-FSW(1))
    2010-07-05 15:52:45.188: [    GPnP][4728]clsgpnp_getCK: [at clsgpnp0.c:1984] Got gpnp security keys (wallet).>
    2010-07-05 15:52:45.188: [    GPnP][4728]clsgpnp_Init: [at clsgpnp0.c:839] GPnP client pid=4844, tl=1, f=3
    2010-07-05 15:53:04.221: [  OCRASM][4728]proprasmo: Failed to open file in dirty mode
    2010-07-05 15:53:04.221: [  OCRASM][4728]proprasmo: Error in open/create file in dg [DATA]
    [  OCRASM][4728]SLOS : SLOS: cat=8, opn=kgfolclcpi1, dep=204, loc=kgfokge
    AMDU-00204: Disk N0002 is in currently mounted diskgroup DATA
    AMDU-00201: Disk N0002: '\\.\ORCLDISKDATA1'
    2010-07-05 15:53:04.314: [  OCRASM][4728]proprasmo: kgfoCheckMount returned [7]
    2010-07-05 15:53:04.314: [  OCRASM][4728]proprasmo: The ASM instance is down
    2010-07-05 15:53:04.361: [  OCRRAW][4728]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
    2010-07-05 15:53:04.361: [  OCRRAW][4728]proprioo: No OCR/OLR devices are usable
    2010-07-05 15:53:04.361: [  OCRASM][4728]proprasmcl: asmhandle is NULL
    2010-07-05 15:53:04.361: [  OCRRAW][4728]proprinit: Could not open raw device
    2010-07-05 15:53:04.361: [  OCRASM][4728]proprasmcl: asmhandle is NULL
    2010-07-05 15:53:04.361: [ default][4728]a_init:7!: Backend init unsuccessful : [26]
    2010-07-05 15:53:04.361: [OCRCHECK][4728]Failed to access OCR repository: [PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep=204, loc=kgfokge
    AMDU-00204: Disk N0002 is in currently mounted diskgroup DATA
    AMDU-00201: Disk N0002: '\\.\ORCLDISKDATA1'
    ] [8]]
    2010-07-05 15:53:04.361: [OCRCHECK][4728]Failed to initialize ocrchek2
    2010-07-05 15:53:04.361: [OCRCHECK][4728]Exiting [status=failed]...

  • ASM instances on 2 node Oracle RAC 10g r2  on Red Hat 4 u1

    Hi all
    I'm experiencing a problem in configuring diskgroups under +ASM instances on a two node Oracle RAC.
    I followed the official guide and also official documents from metalink site, but i'm stuck with the visibility of asm disks.
    I created fake disks on nfs with Netapp certified storage binding them to block device with the usual trick "losetup /dev/loopX /nfs/disk1 " ,
    run "oracleasm createdisk DISKX /dev/loopX" on one node and
    "oracleasm scandisks" on the other one.
    With "oracleasm listdisks" i can see the disks at OS level in both nodes , but , when i try to create and mount diskgroup in the ASM instances , on the instance on which i create the diskgroup all is well, but the other one doesn't see the disks at all, and diskgroup mount fails with :
    ERROR: no PST quorum in group 1: required 2, found 0
    Tue Sep 20 16:22:32 2005
    NOTE: cache dismounting group 1/0x6F88595E (DG1)
    NOTE: dbwr not being msg'd to dismount
    ERROR: diskgroup DG1 was not mounted
    any help would be appreciated
    thanks a lot.
    Antonello

    I'm having this same problem. Did you ever find a solution?

  • OCR and voting disks on ASM, problems in case of fail-over instances

    Hi everybody
    in case at your site you :
    - have an 11.2 fail-over cluster using Grid Infrastructure (CRS, OCR, voting disks),
    where you have yourself created additional CRS resources to handle single-node db instances,
    their listener, their disks and so on (which are started only on one node at a time,
    can fail from that node and restart to another);
    - have put OCR and voting disks into an ASM diskgroup (as strongly suggested by Oracle);
    then you might have problems (as we had) because you might:
    - reach max number of diskgroups handled by an ASM instance (63 only, above which you get ORA-15068);
    - experiment delays (especially in case of multipath), find fake CRS resources, etc.
    whenever you dismount disks from one node and mount to another;
    So (if both conditions are true) you might be interested in this story,
    then please keep reading on for the boring details.
    One step backward (I'll try to keep it simple).
    Oracle Grid Infrastructure is mainly used by RAC db instances,
    which means that any db you create usually has one instance started on each node,
    and all instances access read / write the same disks from each node.
    So, ASM instance on each node will mount diskgroups in Shared Mode,
    because the same diskgroups are mounted also by other ASM instances on the other nodes.
    ASM instances have a spfile parameter CLUSTER_DATABASE=true (and this parameter implies
    that every diskgroup is mounted in Shared Mode, among other things).
    In this context, it is quite obvious that Oracle strongly recommends to put OCR and voting disks
    inside ASM: this (usually called CRS_DATA) will become diskgroup number 1
    and ASM instances will mount it before CRS starts.
    Then, additional diskgroup will be added by users, for DATA, REDO, FRA etc of each RAC db,
    and will be mounted later when a RAC db instance starts on the specific node.
    In case of fail-over cluster, where instances are not RAC type and there is
    only one instance running (on one of the nodes) at any time for each db, it is different.
    All diskgroups of db instances don't need to be mounted in Shared Mode,
    because they are used by one instance only at a time
    (on the contrary, they should be mounted in Exclusive Mode).
    Yet, if you follow Oracle advice and put OCR and voting inside ASM, then:
    - at installation OUI will start ASM instance on each node with CLUSTER_DATABASE=true;
    - the first diskgroup, which contains OCR and votings, will be mounted Shared Mode;
    - all other diskgroups, used by each db instance, will be mounted Shared Mode, too,
    even if you'll take care that they'll be mounted by one ASM instance at a time.
    At our site, for our three-nodes cluster, this fact has two consequences.
    One conseguence is that we hit ORA-15068 limit (max 63 diskgroups) earlier than expected:
    - none ot the instances on this cluster are Production (only Test, Dev, etc);
    - we planned to have usually 10 instances on each node, each of them with 3 diskgroups (DATA, REDO, FRA),
    so 30 diskgroups each node, for a total of 90 diskgroups (30 instances) on the cluster;
    - in case one node failed, surviving two should get resources of the failing node,
    in the worst case: one node with 60 diskgroups (20 instances), the other one with 30 diskgroups (10 instances)
    - in case two nodes failed, the only node survived should not be able to mount additional diskgroups
    (because of limit of max 63 diskgroup mounted by an ASM instance), so all other would remain unmounted
    and their db instances stopped (they are not Production instances);
    But it didn't worked, since ASM has parameter CLUSTER_DATABASE=true, so you cannot mount 90 diskgroups,
    you can mount 62 globally (once a diskgroup is mounted on one node, it is given a number between 2 and 63,
    and other diskgroups mounted on other nodes cannot reuse that number).
    So as a matter of fact we can mount only 21 diskgroups (about 7 instances) on each node.
    The second conseguence is that, every time our CRS handmade scripts dismount diskgroups
    from one node and mount it to another, there are delays in the range of seconds (especially with multipath).
    Also we found inside CRS log that, whenever we mounted diskgroups (on one node only), then
    behind the scenes were created on the fly additional fake resources
    of type ora*.dg, maybe to accomodate the fact that on other nodes those diskgroups were left unmounted
    (once again, instances are single-node here, and not RAC type).
    That's all.
    Did anyone go into similar problems?
    We opened a SR to Oracle asking about what options do we have here, and we are disappointed by their answer.
    Regards
    Oscar

    Hi Klaas-Jan
    - best practises require that also online redolog files are in a separate diskgroup, in case of ASM logical corruption (we are a little bit paranoid): in case DATA dg gets corrupted, you can restore Full backup plus Archived RedoLog plus Online Redolog (otherwise you will stop at the latest Archived).
    So we have 3 diskgroups for each db instance: DATA, REDO, FRA.
    - in case of fail-over cluster (active-passive), Oracle provide some templates of CRS scripts (in $CRS_HOME/crs/crs/public) that you edit and change at your will, also you might create additionale scripts in case of additional resources you might need (Oracle Agents, backups agent, file systems, monitoring tools, etc)
    About our problem, the only solution is to move OCR and voting disks from ASM and change pfile af all ASM instance (parameter CLUSTER_DATABASE from true to false ).
    Oracle aswers were a litlle bit odd:
    - first they told us to use Grid Standalone (without CRS, OCR, voting at all), but we told them that we needed a Fail-over solution
    - then they told us to use RAC Single Node, which actually has some better features, in csae of planned fail-over it might be able to migreate
    client sessions without causing a reconnect (for SELECTs only, not in case of a running transaction), but we already have a few fail-over cluster, we cannot change them all
    So we plan to move OCR and voting disks into block devices (we think that the other solution, which needs a Shared File System, will take longer).
    Thanks Marko for pointing us to OCFS2 pros / cons.
    We asked Oracle a confirmation that it supported, they said yes but it is discouraged (and also, doesn't work with OUI nor ASMCA).
    Anyway that's the simplest approach, this is a non-Prod cluster, we'll start here and if everthing is fine, after a while we'll do it also on Prod ones.
    - Note 605828.1, paragraph 5, Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0, 11.2.0) on RHEL5/OL5
    - Note 428681.1: OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE)
    -"Grid Infrastructure Install on Linux", paragraph 3.1.6, Table 3-2
    Oscar

  • Unable to bring up ASM on 2nd node of a 2-node Cluster

    Having a very wierd problem on a 2-node cluster. I can only bring up on ASM instance at a time. If i bring up the second, it hangs. This is what the second (hung) instance puts in the alert log:
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /ORAUTL/oraasm/product/ASM/dbs/arch
    Autotune of undo retention is turned off.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
    large_pool_size = 12582912
    instance_type = asm
    cluster_interconnects = 192.168.0.12
    cluster_database = TRUE
    instance_number = 2
    remote_login_passwordfile= EXCLUSIVE
    background_dump_dest = /ORAUTL/oraasm/admin/+ASM2/bdump
    user_dump_dest = /ORAUTL/oraasm/admin/+ASM2/udump
    core_dump_dest = /ORAUTL/oraasm/admin/+ASM2/cdump
    pga_aggregate_target = 0
    Cluster communication is configured to use the following interface(s) for this instance
    192.168.0.12
    Fri Nov 21 21:10:48 2008
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=5428
    DIAG started with pid=3, OS id=5430
    PSP0 started with pid=4, OS id=5432
    LMON started with pid=5, OS id=5434
    LMD0 started with pid=6, OS id=5436
    LMS0 started with pid=7, OS id=5438
    MMAN started with pid=8, OS id=5442
    DBW0 started with pid=9, OS id=5444
    LGWR started with pid=10, OS id=5446
    CKPT started with pid=11, OS id=5448
    SMON started with pid=12, OS id=5458
    RBAL started with pid=13, OS id=5475
    GMON started with pid=14, OS id=5487
    Fri Nov 21 21:10:49 2008
    lmon registered with NM - instance id 2 (internal mem no 1)
    Fri Nov 21 21:10:49 2008
    Reconfiguration started (old inc 0, new inc 2)
    ASM instance
    List of nodes:
    0 1
    Global Resource Directory frozen
    Communication channels reestablished
    After this it hangs. i've checked everything. CRS is fine.
    I suspect its the kernel revision. This is a cluster of two v890's. Kernel rev is 127127-11. Anyone seen this issue ?
    thanks

    Responses in-line:
    Have you got any issue reported from Lock Monitor's (LMON) ? (those messages are in the alert.log are summaries of the reconfiguration event.
    No issues that I have seen. I see trc files on both nodes for lmon, but neither contain errors.Do you have any post issues on the date that issue began (something with Reconfiguration started) ?
    This is a new build. Its going to be a DR environment (Dataguard Physical Standby), so we've never managed to get ASM up yet.Do you have any other errors on the second node on the date the issue appears (some ORA-27041 or other messages) errors?
    No errors at all.What is the result of a crs_stat -t ?
    HA Resource Target State
    ora.vzdfwsdbp01.LISTENER_VZDFWSDBP01.lsnr ONLINE ONLINE on vzdfwsdbp01
    ora.vzdfwsdbp01.gsd ONLINE ONLINE on vzdfwsdbp01
    ora.vzdfwsdbp01.ons ONLINE ONLINE on vzdfwsdbp01
    ora.vzdfwsdbp01.vip ONLINE ONLINE on vzdfwsdbp01
    ora.vzdfwsdbp02.LISTENER_VZDFWSDBP02.lsnr ONLINE ONLINE on vzdfwsdbp02
    ora.vzdfwsdbp02.gsd ONLINE ONLINE on vzdfwsdbp02
    ora.vzdfwsdbp02.ons ONLINE ONLINE on vzdfwsdbp02
    ora.vzdfwsdbp02.vip ONLINE ONLINE on vzdfwsdbp02
    ASM isn't registered with CRS/OCR yet. I did add it at one time, but it didnt seem to make any difference.What is the release of your installation 10.2.0.4? Otherwise control if you can upgrade CRS, ASM and your RDBMS to that release.
    CRS, ASM and Oracle will be 10.2.0.3Can't go to 10.2.0.4 yet as primary site is at 10.2.0.3 on a live system.
    Can you please tell us what is the OS / Hardware in use?
    Solaris 10, Sun v890$ uname -a
    SunOS dbp02 5.10 Generic_127127-11 sun4u sparc SUNW,Sun-Fire-V890
    What is the result of that on the second node:
    even a startup nomount hangs on second node.connect sqlplus / as sysdba;
    startup nomount
    desc v$asmdiskgroup;
    select name, mount from v$diskgroup;
    In the case that no group is mounted do
    alter database mount diskgroup 'your diskgroupname';
    What is the result of that?
    thanks
    -toby

Maybe you are looking for