Voting disk problem

I had a problem with my votingdisks:
First CSS would not start. CSS log said votingdisks were missing:
C:\oracle\product\10.2.0>crsctl query css votedisk
0. 0 \\.\votedsk1
1. 0
2. 0
located 3 votedisk(s).
I stopped the cluster and could add my voting disks:
Result:
C:\oracle\product\10.2.0>crsctl query css votedisk
0. 0 \\.\votedsk1
1. 0
2. 0
3. 0 \\.\votedsk3
4. 0 \\.\votedsk2
located 5 votedisk(s).
How can I get rid off the "empty" values?

I ran into the samething today and figured out how to get rid of them at least on a Win 2003 SP2R2 cluster. Mine were showing 0s for the path as well so after a lot of trial and error I finally just tried:
crsctl delete css votedisk 0
And it worked... What happens is, when you have to do a -force to create extra votedisks the cluster has to be stopped on all nodes or this will happen. We were doing a totally fresh install so we played with it a little to determine this is what happened. When we stopped everything before adding votedisks using -force and cleaning up the bad ones, everything went fine.

Similar Messages

OCR and voting disks on ASM, problems in case of fail-over instances

Hi everybody
in case at your site you :
- have an 11.2 fail-over cluster using Grid Infrastructure (CRS, OCR, voting disks),
where you have yourself created additional CRS resources to handle single-node db instances,
their listener, their disks and so on (which are started only on one node at a time,
can fail from that node and restart to another);
- have put OCR and voting disks into an ASM diskgroup (as strongly suggested by Oracle);
then you might have problems (as we had) because you might:
- reach max number of diskgroups handled by an ASM instance (63 only, above which you get ORA-15068);
- experiment delays (especially in case of multipath), find fake CRS resources, etc.
whenever you dismount disks from one node and mount to another;
So (if both conditions are true) you might be interested in this story,
then please keep reading on for the boring details.
One step backward (I'll try to keep it simple).
Oracle Grid Infrastructure is mainly used by RAC db instances,
which means that any db you create usually has one instance started on each node,
and all instances access read / write the same disks from each node.
So, ASM instance on each node will mount diskgroups in Shared Mode,
because the same diskgroups are mounted also by other ASM instances on the other nodes.
ASM instances have a spfile parameter CLUSTER_DATABASE=true (and this parameter implies
that every diskgroup is mounted in Shared Mode, among other things).
In this context, it is quite obvious that Oracle strongly recommends to put OCR and voting disks
inside ASM: this (usually called CRS_DATA) will become diskgroup number 1
and ASM instances will mount it before CRS starts.
Then, additional diskgroup will be added by users, for DATA, REDO, FRA etc of each RAC db,
and will be mounted later when a RAC db instance starts on the specific node.
In case of fail-over cluster, where instances are not RAC type and there is
only one instance running (on one of the nodes) at any time for each db, it is different.
All diskgroups of db instances don't need to be mounted in Shared Mode,
because they are used by one instance only at a time
(on the contrary, they should be mounted in Exclusive Mode).
Yet, if you follow Oracle advice and put OCR and voting inside ASM, then:
- at installation OUI will start ASM instance on each node with CLUSTER_DATABASE=true;
- the first diskgroup, which contains OCR and votings, will be mounted Shared Mode;
- all other diskgroups, used by each db instance, will be mounted Shared Mode, too,
even if you'll take care that they'll be mounted by one ASM instance at a time.
At our site, for our three-nodes cluster, this fact has two consequences.
One conseguence is that we hit ORA-15068 limit (max 63 diskgroups) earlier than expected:
- none ot the instances on this cluster are Production (only Test, Dev, etc);
- we planned to have usually 10 instances on each node, each of them with 3 diskgroups (DATA, REDO, FRA),
so 30 diskgroups each node, for a total of 90 diskgroups (30 instances) on the cluster;
- in case one node failed, surviving two should get resources of the failing node,
in the worst case: one node with 60 diskgroups (20 instances), the other one with 30 diskgroups (10 instances)
- in case two nodes failed, the only node survived should not be able to mount additional diskgroups
(because of limit of max 63 diskgroup mounted by an ASM instance), so all other would remain unmounted
and their db instances stopped (they are not Production instances);
But it didn't worked, since ASM has parameter CLUSTER_DATABASE=true, so you cannot mount 90 diskgroups,
you can mount 62 globally (once a diskgroup is mounted on one node, it is given a number between 2 and 63,
and other diskgroups mounted on other nodes cannot reuse that number).
So as a matter of fact we can mount only 21 diskgroups (about 7 instances) on each node.
The second conseguence is that, every time our CRS handmade scripts dismount diskgroups
from one node and mount it to another, there are delays in the range of seconds (especially with multipath).
Also we found inside CRS log that, whenever we mounted diskgroups (on one node only), then
behind the scenes were created on the fly additional fake resources
of type ora*.dg, maybe to accomodate the fact that on other nodes those diskgroups were left unmounted
(once again, instances are single-node here, and not RAC type).
That's all.
Did anyone go into similar problems?
We opened a SR to Oracle asking about what options do we have here, and we are disappointed by their answer.
Regards
Oscar

Hi Klaas-Jan
- best practises require that also online redolog files are in a separate diskgroup, in case of ASM logical corruption (we are a little bit paranoid): in case DATA dg gets corrupted, you can restore Full backup plus Archived RedoLog plus Online Redolog (otherwise you will stop at the latest Archived).
So we have 3 diskgroups for each db instance: DATA, REDO, FRA.
- in case of fail-over cluster (active-passive), Oracle provide some templates of CRS scripts (in $CRS_HOME/crs/crs/public) that you edit and change at your will, also you might create additionale scripts in case of additional resources you might need (Oracle Agents, backups agent, file systems, monitoring tools, etc)
About our problem, the only solution is to move OCR and voting disks from ASM and change pfile af all ASM instance (parameter CLUSTER_DATABASE from true to false ).
Oracle aswers were a litlle bit odd:
- first they told us to use Grid Standalone (without CRS, OCR, voting at all), but we told them that we needed a Fail-over solution
- then they told us to use RAC Single Node, which actually has some better features, in csae of planned fail-over it might be able to migreate
client sessions without causing a reconnect (for SELECTs only, not in case of a running transaction), but we already have a few fail-over cluster, we cannot change them all
So we plan to move OCR and voting disks into block devices (we think that the other solution, which needs a Shared File System, will take longer).
Thanks Marko for pointing us to OCFS2 pros / cons.
We asked Oracle a confirmation that it supported, they said yes but it is discouraged (and also, doesn't work with OUI nor ASMCA).
Anyway that's the simplest approach, this is a non-Prod cluster, we'll start here and if everthing is fine, after a while we'll do it also on Prod ones.
- Note 605828.1, paragraph 5, Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0, 11.2.0) on RHEL5/OL5
- Note 428681.1: OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE)
-"Grid Infrastructure Install on Linux", paragraph 3.1.6, Table 3-2
Oscar

Root.sh hangs at formatting voting disk on OEL32 11gR2 RAC with ocfs2

Hi,
Am trying to bring up Oracle11gR2 RAC on Enterprise Linux x86 (32bit) version 5.6. I am using Ocfs2 1.4 as my cluster file share. Everything went fine till the root.sh and it hangs with a message "now formatting voting disk <vdsk path>
The logs are mentioned below.
Checked the alert log:
{quote}
cssd(9506)]CRS-1601:CSSD Reconfiguration complete. Active nodes are oel32rac1 .
2011-08-04 15:58:55.356
[ctssd(9552)]CRS-2407:The new Cluster Time Synchronization Service reference node is host oel32rac1.
2011-08-04 15:58:55.917
[ctssd(9552)]CRS-2401:The Cluster Time Synchronization Service started on host oel32rac1.
2011-08-04 15:58:56.213
[client(9567)]CRS-1006:The OCR location /u02/storage/ocr is inaccessible. Details in /u01/app/11.2.0/grid/log/oel32rac1/client/ocrconfig_9567.log.
2011-08-04 15:58:56.365
[client(9567)]CRS-1001:The OCR was formatted using version 3.
2011-08-04 15:58:59.977
[crsd(9579)]CRS-1012:The OCR service started on node oel32rac1.
{quote}
crsctl.log:
{quote}
2011-08-04 15:59:00.246: [ CRSCTL][3046184656]crsctl_vformat: obtain cssmode 1
2011-08-04 15:59:00.247: [ CRSCTL][3046184656]crsctl_vformat: obtain VFListSZ 0
2011-08-04 15:59:00.258: [ CRSCTL][3046184656]crsctl_vformat: Fails to obtain backuped Lease from CSSD with error code 16
2011-08-04 15:59:01.857: [ CRSCTL][3046184656]crsctl_vformat: to do clsscfg fmt with lease sz 0
2011-08-04 15:59:01.910: [   SKGFD][3046184656]NOTE: No asm libraries found in the system
2011-08-04 15:59:01.910: [    CLSF][3046184656]Allocated CLSF context
2011-08-04 15:59:01.910: [   SKGFD][3046184656]Discovery with str:/u02/storage/vdsk:
2011-08-04 15:59:01.910: [   SKGFD][3046184656]UFS discovery with :/u02/storage/vdsk:
2011-08-04 15:59:01.910: [   SKGFD][3046184656]Fetching UFS disk :/u02/storage/vdsk:
2011-08-04 15:59:01.911: [   SKGFD][3046184656]OSS discovery with :/u02/storage/vdsk:
2011-08-04 15:59:01.911: [   SKGFD][3046184656]Handle 0xa6c19f8 from lib :UFS:: for disk :/u02/storage/vdsk:
2011-08-04 17:10:37.522: [   SKGFD][3046184656]WARNING:io_getevents timed out 618 sec
2011-08-04 17:10:37.526: [   SKGFD][3046184656]WARNING:io_getevents timed out 618 sec
{quote}
ocrconfig log:
{quote}
2011-08-04 15:58:56.214: [ OCRRAW][3046991552]ibctx: Failed to read the whole bootblock. Assumes invalid format.
2011-08-04 15:58:56.214: [ OCRRAW][3046991552]proprinit:problem reading the bootblock or superbloc 22
2011-08-04 15:58:56.214: [ OCROSD][3046991552]utread:3: Problem reading buffer 8532000 buflen 4096 retval 0 phy_offset 102400 retry 0
2011-08-04 15:58:56.214: [ OCROSD][3046991552]utread:3: Problem reading buffer 8532000 buflen 4096 retval 0 phy_offset 102400 retry 1
2011-08-04 15:58:56.214: [ OCROSD][3046991552]utread:3: Problem reading buffer 8532000 buflen 4096 retval 0 phy_offset 102400 retry 2
2011-08-04 15:58:56.214: [ OCROSD][3046991552]utread:3: Problem reading buffer 8532000 buflen 4096 retval 0 phy_offset 102400 retry 3
2011-08-04 15:58:56.214: [ OCROSD][3046991552]utread:3: Problem reading buffer 8532000 buflen 4096 retval 0 phy_offset 102400 retry 4
2011-08-04 15:58:56.214: [ OCROSD][3046991552]utread:3: Problem reading buffer 8532000 buflen 4096 retval 0 phy_offset 102400 retry 5
2011-08-04 15:58:56.214: [ OCRRAW][3046991552]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
2011-08-04 15:58:56.365: [ OCRRAW][3046991552]iniconfig:No 92 configuration
2011-08-04 15:58:56.365: [ OCRAPI][3046991552]a_init:6a: Backend init successful
2011-08-04 15:58:56.390: [ OCRCONF][3046991552]Initialized DATABASE keys
2011-08-04 15:58:56.564: [ OCRCONF][3046991552]csetskgfrblock0: output from clsmft: [clsfmt: successfully initialized file /u02/storage/ocr
2011-08-04 15:58:56.577: [ OCRCONF][3046991552]Successfully set skgfr block 0
2011-08-04 15:58:56.578: [ OCRCONF][3046991552]Exiting [status=success]...
{quote}
ocssd.log:
{quote}
2011-08-04 15:59:00.140: [    CSSD][2963602320]clssgmFreeRPCIndex: freeing rpc 23
2011-08-04 15:59:00.228: [    CSSD][2996054928]clssgmExecuteClientRequest: CONFIG recvd from proc 6 (0xb35f7438)
2011-08-04 15:59:00.228: [    CSSD][2996054928]clssgmConfig: type(1)
2011-08-04 15:59:00.234: [    CSSD][2996054928]clssgmExecuteClientRequest: VOTEDISKQUERY recvd from proc 6 (0xb35f7438)
2011-08-04 15:59:00.247: [    CSSD][2996054928]clssgmExecuteClientRequest: CONFIG recvd from proc 6 (0xb35f7438)
2011-08-04 15:59:00.247: [    CSSD][2996054928]clssgmConfig: type(1)
2011-08-04 15:59:03.039: [    CSSD][2942622608]clssnmSendingThread: sending status msg to all nodes
2011-08-04 15:59:03.039: [    CSSD][2942622608]clssnmSendingThread: sent 4 status msgs to all nodes
2011-08-04 15:59:07.047: [    CSSD][2942622608]clssnmSendingThread: sending status msg to all nodes
2011-08-04 15:59:07.047: [    CSSD][2942622608]clssnmSendingThread: sent 4 status msgs to all nodes
2011-08-04 15:59:11.057: [    CSSD][2942622608]clssnmSendingThread: sending status msg to all nodes
2011-08-04 15:59:11.057: [    CSSD][2942622608]clssnmSendingThread: sent 4 status msgs to all nodes
2011-08-04 15:59:16.068: [    CSSD][2942622608]clssnmSendingThread: sending status msg to all nodes
2011-08-04 15:59:16.068: [    CSSD][2942622608]clssnmSendingThread: sent 5 status msgs to all nodes
2011-08-04 15:59:21.079: [    CSSD][2942622608]clssnmSendingThread: sending status msg to all nodes
2011-08-04 15:59:21.079: [    CSSD][2942622608]clssnmSendingThread: sent 5 status msgs to all nodes
{quote}
Any help here is appreciated.
Regards
Amith R
Edited by: Mithzz on Aug 4, 2011 4:58 AM

Did an lsof on vdisk and it showed
>
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
crsctl.bi 9589 root 26u REG 8,17 21004288 102980 /u02/storage/vdsk
[root@oel32rac1 ~]# ps -ef |grep crsctl
root 9589 7583 0 15:58 pts/1 00:00:00 [crsctl.bin] <defunct>
>
Could this be a permission issue ?
--Amith

OCR and Voting Disk partition

Dear all
I uninstall clusterware. And i want to install again that but when i made partition in windows 2003 then CVU fail the stage command.
I did the following things..
1) diskpart --->select Disk 1----> clean -->exit (both the nodes)
2) diskpart -->automount enable--->exit ---> restart (both the nodes)
3) Then i make four partition in one node. And see after restart node 2 have that partition with drive letter. I remove all the drive letter.
But CVU stage command failed. And if i run clusterware then it can get partition which has not previous OCR and voting. I think previous OCR and Voting disk could not clean.
Could u pls tell me how can i get solution.
Moinul

Check Metalink for this problem area ! Raise TAR

Recovery scenario - Voting disk does not match with the cluster guid

Hi all,
Think of you can not start your guest VMs just because it has a corrupted system.img root image. And assume it contains 5 physical disk( which are all created by the RAC template) hence ASM on them.
What is the simplest recovery scneario of the guest vms (RAC)?
Can it be a feasible scenario for recover of the availablity? (Assume both of the RAC system images are corrupted and we prefer not a system level recovery rather than backup / restore)
1. Create 2 RAC instances using the same networking and hostname details as the ones that are corrupted. - Use 5 different new disks.
2 Shutdown the newly created instances. Drop the disks from the newly created instances using VM manager.
3. Add the old disks whose system image is failing to be recoverd but ASM disks are still in use (from the newly created instances using VM manager.) to the newly created instances.
4. Open the newly created instances
Can we expect the ASM and CRS could be initialized and be opened without a problem?
When I try this scenario I get the folllowing error from the cssd/crsd .
- Cluster guid 9112ddc0824fefd5ff2b7f9f7be8f048 found in voting disk does not match with the cluster guid a3eec66a2854ff0bffe784260856f92a obtained from the GPnP profile.
- Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity.
What could be the simplest way of recovery of a virtual machine that has healthy ASM disks but corrupted system image?
Thank you

Hi,
you have a similar problem, when trying to clone databases with 11.2.
The problem is that a cluster is uniquely identified, and this information is hold in the OCR and the Voting disks. So exactly these 2 are not to be cloned.
To achieve what you want, simply setup your system in that way, that you have a separate diskgroup for OCR and Voting (and ASM spfile), which is not to be restored in this case of szeanrio.
Only all database files in ASM will then be exchanged later.
Then what you want can be achieved.
However I am not sure that the RAC templates have the option to install OCR and Voting into a separated diskgroup.
Regards
Sebastian

Why do we need voting Disk or Voting File

Hi all
What is the need of Voting file or disk??? i know it maintains the node information n all. But if nodes can ping all other nodes what is the need of voting disk.
Can anyone please explain.
PS : I read the
Oracle® Database
Oracle Clusterware and Oracle Real Application Clusters
Administration and Deployment Guide
10g Release 2 (10.2)
But could not find how does it actually come into picture.
Thanks in anticipation

Lets take the case of a 2 node RAC - instance 'A' and instance 'B';
Lets assume that instances A & instance B is unable to communicate with each other. Now, this might be due the fact that the other instance is actually down oR because there is some problem with the interconnect communication between the instances.
A split-brain occurs when the instances cannot speak to each other due to interconnect communication problem and each instance happens to think that it's the only surviving instance and starts updating the database. This obviously is problematic, isn't it?
To avoid split brain, both instances are required to update the voting disk periodically.
Now consider the earlier case from the perspecive of one of the nodes:
case 1) instance B is actually down:- Here, looking from instance A's perspective; there is no communication over the interconnect with instance B and also instance B is not updating the voting disk ('coz it's down). Instance A can safely assume that instance B is down and it can go on providing the services and updating the database.
Case 2) the problem is with interconnect communication:- Here, looking from instance A's perspective; there is no communication over the interconnect with instance B. However it can see that instance B is updating the voting disk; which means instance B is not actually down. This state is true from instance B's perspective; it can see that instance A is updating the voting disk but it cannot speak to instance A over the interconnect. At this point, both instances rush and try to lock down the voting disk and whoever gets the lock ejects the other one thus avoiding split-brain syndrome.
That's the theory behind using a voting disk.
When the instances of a RAC boots up, one of the instances will be assigned the master. In the case of a split-brain the master gets the lock on the voting disk and survives. In reality the instance with the lower node number survives, even if the problem is with the interconnect network card on that node :)
Cheers.

Voting disk issue in Windows 2003 server

I installed oracle RAC 10g with ASM on windows server 2003 R2. I was able to successfully install clusterware as well as ASM with database. Prior to the install, I ran diskpart and then set automount enable. I was able to configure the ocr, ocrmirror and votedisk without any problem. After completion of the installation and then restarting both the nodes, looking at the Disk Management from the Computer Management, my first node is showing the partition for votedisk as raw disk but on the second node, I am seeing drive letters configured for the votedisks. I ran crsctl query css votedisk on both the nodes and both the nodes are pointing to the same location, \\.\votedsk1, \\.\votedsk2 and \\.\votedsk3.
Now my question is why am I seeing the drive letter(s) (G:) for the drives that were configured to votedsk from the second node and not seeing the drive letters from the first node?
Can I change the drive letter that were configured for voting disks from the second node?
I did take a backup of the votedsk using ocopy, but can this be a valid backup if I clear the drive letter on second node and then restore from the backup? I appreciate any help to correct this problem. Thanks

I can perform delete and add another voting disk, But what about the drive letter that is showing on the second node and not on the first? Do I need to remove the drive letter showing on node 2 while the voting disk is still showing up prior to performing the delete and add of another voting disk? Thanks for your help.
Edited by: user8769472 on May 25, 2010 3:43 PM

OCR and voting disk location

Hi all,
I'm testing to install an Oracle RAC database (11g) on linux RHE 5 with a shared storage DELL Equallogic SCSI SAN.
I will setup a 2 nodes and i will use ASM for the database storage. I'm new in RAC database and i'm checking some docs and find that for the installation should have 3 volumes.
1- volumes for OCR and voting disk
2- volume for Database file (ASM)
3- volume for flash Recovery area and archive log file (ASM)
My question is which file system i will use for OCR and voting disk volume as i know for Datafiles and flash recovery area i will use ASM, but for voting disk i cannot use ASM because the system will access first OCR before starting ASm instance.
If i will use other file system how can i do ?
Thank you.

Hi
Thank you all for your help, actually i'm again facing problem in installing ASMlib, i cannot find ASMlib package for my OS even Raw device service is also not present.
My OS is CentOS 6 and Oracle 11g R2 RAC then installation type.
[root@Etopup-srv01 ~]# cat /etc/redhat-release
CentOS release 6.4 (Final)
Would you please help me what can i do to configure my ASM disk group ?
Thank you.

Regarding Voting disk recovery scenarios

Hi,
For years i have read about RAC and Voting disk and it is said that each node should access more than half of the voting disks but never got a chance to work on the below scnerios which i have mentioned, if some one has practical done the below scenarios or have good knowledge do let me know.
1) If i have 5 voting disks and out of which 2 got corrupted or deleted will the cluster keep working properly? If i boot my system or restart the cluster will it still work fine or not?
2) The above scenario with 3 voting disk deleted or got corrupted what will happen?
3) If i have 2 OCR and i got deleted or corrupted will the system run fine?

Aman,
During startup the clusterware requires the majority of votes to start the CSS Daemon.
The majority is counted on how many was configured, not how many remain alive.
Below test using 3 Votedisk (11.2.0.3)
alert.log
[cssd(26233)]CRS-1705:Found 1 configured voting files but 2 voting files are required, terminating to ensure data integrity; details at (:CSSN                     M00021:) in /u01/app/11.2.0/grid/log/node11g02/cssd/ocssd.log
ocssd.log
2014-11-14 05:20:05.126: [    CSSD][3021551360]clssnmvDiskVerify: Successful discovery of 1 disks
2014-11-14 05:20:05.126: [    CSSD][3021551360]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2014-11-14 05:20:05.126: [    CSSD][3021551360]clssnmCompleteVFDiscovery: Completing voting file discovery
2014-11-14 05:20:05.126: [    CSSD][3021551360]clssnmvDiskStateChange: state from discovered to pending disk /dev/asm-ocr-vote01
2014-11-14 05:20:05.126: [    CSSD][3021551360]clssnmvDiskStateChange: state from pending to configured disk /dev/asm-ocr-vote01
2014-11-14 05:20:05.126: [    CSSD][3021551360]clssnmvVerifyCommittedConfigVFs: Insufficient voting files found, found 1 of 3 configured, needed 2 voting files
With 2 voting files online, no problem, clusterware start with warning on logs.
Ps. If was configured a diskgroup (OCR_VOTE) with high redundancy (5 asmdisk each in your onw failgroup), even diskgroup can hold 3 asmdisk failure. The diskgroup goes down and if there is only one OCR (no mirror) on that Diskgroup the whole clusterware goes down.

OCFS2 , ASM and Raw Device for Voting Disk and OCR

Dear all ,
My manager assigned me to establish a 11gRAC , 3 node on OEL. He likes to locate ASM with datafiles , at the same time, 2 OCR vol on OCFS2 and RawDevice respectively , 2 voting disks on OCFS2 and RawDevice respectively also.
This really annoys me .
Oh all experts, could anybody give me some advices ? Can those stuff be co-existed ?

Billy Verreynne wrote:
MccLok wrote:
My manager assigned me to establish a 11gRAC , 3 node on OEL. He likes to locate ASM with datafiles , at the same time, 2 OCR vol on OCFS2 and RawDevice respectively , 2 voting disks on OCFS2 and RawDevice respectively also. Not a great idea to introduce additional s/w layers between the CRS software and its OCR and voting disks. If ocfs2 is for example used as a shared file system for these devices (files), then the ocfs2 cluster s/w needs to be up and running, the shared devices mounted, before CRS can successfully start. Thus any problem with the ocfs2 stack, will impact on CRS. Why? What does ocfs2 buy you in this regard? What are the benefits that justifies the increased complexity and the dependency of CRS on another s/w layer?
Fewer moving parts means less complexity, less stuff that needs to be configured and can go wrong.In fact, some users had perception that " Pay mroe so that Get more".

3rd voting disk on nfs share

Hey, as described in Oracle Clusterware 11g Release 2 (11.2) – Using standard NFS to support a third voting file for extended cluster configurations
I added the nfs file to my OCR diskgroup in ASM as quorum device.
/crs/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
1. ONLINE 0eb6aaa85e5c4fd6bf9d0481cfd7d517 (ORCL:VOTE1) [OCR]
2. ONLINE 5a66449fbada4f34bf0c3be4574f03bc (ORCL:VOTE2) [OCR]
3. ONLINE 76ad295ebc054f07bfea124fb08da432 (ORCL:VOTE3) [OCR]
Located 3 voting disk(s).
asmca is showing also ORCL:VOTE3 and /voting_disk/vote3_nfs
Is my system now correctly configured, or do I still need to change my votedisk configuration ?
crsctl replace..... ?
Chris

Hey Levi, I setup a test enviroment for my problem.
I added an ocrmirror to my OCR diskgroup (containing 4 disk, 22)
My ocrmirror is diskgroup DATA (22 disk with 2 failgroups) - the adding of the ocrmirror was successful. ocrcheck showed both diskgroups available
Then I removed 2 disk from storage 1 - so OCR diskgroup had only 2 disks left. Right away the logfiles show errors. OCR inaccesible. +DATA can´t be used due to the error below.
So what I am doing wrong, while adding the ocrmirror ? Which command to use, for replacing +OCR diskgroup ?
[client(27318)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /crs/log/rac1/client/ocrcheck_27318.log.
2012-01-06 22:52:57.393
[client(27318)]CRS-1011:OCR cannot determine that the OCR content contains the latest updates. Details in /crs/log/rac1/client/ocrcheck_27318.log.
2012-01-06 22:53:33.859
2012-01-06 23:00:17.272: [ OCRRAW][3674175232]proprior: Header check from OCR device 0 offset 6574080 failed (26).
2012-01-06 23:00:17.272: [ OCRRAW][3674175232]proprior: Retrying buffer read from another mirror for disk group [+OCR] for block at offset [6574080]
2012-01-06 23:00:17.273: [ OCRASM][3674175232]proprasmres: Total 0 mirrors detected
2012-01-06 23:00:17.273: [ OCRASM][3674175232]proprasmres: Only 1 mirror found in this disk group.
2012-01-06 23:00:17.273: [ OCRASM][3674175232]proprasmres: Need to invoke checkdg. Mirror #0 has an invalid buffer.
2012-01-06 23:00:17.300: [ OCRASM][3674175232]ASM Error Stack : ORA-27091: unable to queue I/O
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-06512: at line 4
OCRCHECK
racle Database 11g Clusterware Release 11.2.0.3.0 - Production Copyright 1996, 2011 Oracle. All rights reserved.
2012-01-06 22:55:25.572: [OCRCHECK][903825152]ocrcheck starts...
2012-01-06 22:55:25.640: [ OCRASM][903825152]proprasmo: kgfoCheckMount return [6]. Cannot proceed with dirty open.
2012-01-06 22:55:25.640: [ OCRASM][903825152]proprasmo: Error in open/create file in dg [OCR]
[ OCRASM][903825152]SLOS : SLOS: cat=6, opn=kgfo, dep=0, loc=kgfoCkMt03
2012-01-06 22:55:25.640: [ OCRASM][903825152]ASM Error Stack :
2012-01-06 22:55:25.667: [ OCRASM][903825152]proprasmo: kgfoCheckMount returned [6]
2012-01-06 22:55:25.667: [ OCRASM][903825152]proprasmo: The ASM disk group OCR is not found or not mounted
2012-01-06 22:55:25.668: [ OCRRAW][903825152]proprioo: Failed to open [+OCR]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2012-01-06 22:55:25.707: [ OCRRAW][903825152]proprioini: disk 1 (+DATA) does not have enough votes (1,2)
2012-01-06 22:55:25.707: [ OCRRAW][903825152]proprioo: Not enought quorum to open the disks (26)
2012-01-06 22:55:25.707: [ OCRRAW][903825152]proprinit: Could not open raw device
2012-01-06 22:55:25.707: [ OCRASM][903825152]proprasmcl: asmhandle is NULL
2012-01-06 22:55:25.709: [ default][903825152]a_init:7!: Backend init unsuccessful : [26]
2012-01-06 22:55:25.709: [OCRCHECK][903825152]initreboot: Failed to initialize OCR in REBOOT level. Retval:[26] Error:[PROC-26: Error while accessing the physical storage

2 out of 3 voting disk gets corrupted

O/S - Oracle Linux
DB Version - Oracle 11.2.0,3
Three node RAC
Voting disks is 3 in Normal redundancy out of which 2 disks gets corrupted.
I know the process of recovering the voting disk but I am not able to stop the cluster services and start the same in exclusive mode. I am always getting "Unable to communicate with CRS services" . I rebotted 3 nodes but the same issue persists.
Any advice will be appreciated.

Is ASM used for OCR and voting disk storage?
This is the recommendation and the default - creating a normal redundancy diskgroup with an additional quorum disk (i.e. 3 disks in total).
There is an automated backup of the OCR. No backup needed for voting files. So it is not a disaster loosing the cluster's OCR and/or voting disks.
Assuming ASM and a diskgroup is used.
Anything else on the diskgroup?
I recently had a similar problem. I used the diskgroup only for OCR and voting disks. Lost 2 disks. I shutdown the RAC. Started CRS in exclusive mode on 1 node. Launched sqlplus and dropped the diskgroup and recreated it with 3 disks (and a quorum). Restored the OCR backup. Confirmed the voting disks config (no need to change it as the underlying diskgroup did not have a name change). Restarted that node. Forced a rebalance of the diskgroup. Then started the remaining cluster nodes.
There are Oracle Support Notes that describe these steps in detail. It should be relatively easy finding the note for fixing a 11gr2 RAC's OCR/voting ASM diskgroup.
If you have database data on it too - problems. If the remaining working disk is the quorum disk, then both mirror disks are toast. If not, you can try to force the disks online again - assuming disks are okay and broken path was fixed. Else add 2 new disks to replace the 2 broken disks and rebalance. Never done this myself though with a quorum disk..

Why Voting Disk & OCR not on ASM?

What could be the reason not to have Voting disk and OCR on ASM.
Is it that, we require Voting Disk and OCR before ASM? Any idea?

user9198889 wrote:
but this doesnt seem to be a problem with 11.2 anymore! oracle 11g r2 recomends storing ocr and voting disks on asm? why is does?
how does it work ?
i cant get my head around thisOn 11gR2 we can keep OCR and VOTE in ASM Diskgroup.
see start process
http://download.oracle.com/docs/cd/E11882_01/rac.112/e10717/intro.htm#CEGDBFIE
10g -> 11gR1, we create ASM diskgroup from database software
11gR2, we create ASM diskgroup from cluster software.
recomends storing ocr and voting disks on asm? why is does? 11gR2 not support raw devices, support ASM and CFS

Keeping voting disk in OCFS or CFS

Is there any problem keeping voting disk in OCFS or CFS?
Thanks,
Mayilselvan.S

There is no issues in putting voting disks there.
Oracle statement -
Oracle
recommends that you use Automatic Storage Management (ASM) and Oracle Managed Files
(OMF), or a cluster file system such as Oracle Cluster File System (OCFS). If you use ASM or a
cluster file system, you can also utilize OMF and other Oracle Database 10g storage features.
Oracle Clusterware requires that the OCR files and voting disk files be shared. Note that ASM
cannot be used to store these files as the clusterware components are started before the ASM or
RAC instances. These files could map to raw devices or exist on an OCFS volume.

Defining OCR / Voting disks

I followed this guide
http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnOEL5UsingVMwareServer2.php#install_grid_infrastructure
to install "clusterware" on my SLES11 maschines.
Question:
In 10g I were able to select which disks a for voting and which are for ocr.
Does it make sense to create different ASM Diskgroups for OCR and Voting, and where can I select which diskgroup should be used ?

zekeriya, you are only right for Version 10. Things changed with Oracle 11. Now you cannot store vote and ocr on raw devices, only in ASM or a cluster filesystem. I know it is a chicken-and-egg problem and I have not quite understood how they do it but that's how things work now.
You can store voting disks and the OCR on Oracle Automatic Storage Management (Oracle ASM), or a certified cluster file system.See this link for more details
http://download.oracle.com/docs/cd/E11882_01/rac.112/e10717/votocr.htm#CHDJDFIC
And yes, it'd recommend to use a special diskgroup just for these.
Bjoern

Voting disk problem

Similar Messages

Maybe you are looking for