CRS restarting - OCR inaccessible 10.2.0.4

Hi all,
i'm facing this problem and i don't understand why node2 and node1 "sometimes" (1 or 2 days a week) gives this type of error and crs restarts.
The ocr files are in an OCFS2 filesystem on a dedicated LUN (mpath):
/dev/mapper/mpath2 /ocr_voting ocfs2 _netdev,datavolume,nointr 0 0
I'm on itanium platform:
Linux 2.6.18-194.3.1.el5 #1 SMP Sun May 2 04:23:11 EDT 2010 ia64 ia64 ia64 GNU/Linux
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
CRS LOG:
[crsd(4430)]CRS-1006:The OCR location /ocr_voting/OCRfile is inaccessible. Details in /u01/app/oracle/product/10.2.0/crs/log/srvorcl02/crsd/crsd.log.
2011-08-05 18:43:32.894
CRSD.LOG
2011-08-05 18:43:29.449: [ OCRRAW][551547456]rtnode:3: invalid tnode 475
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprsetvali: problem reading the tnode
2011-08-05 18:43:29.449: [ OCRRAW][551547456]rrecover:1:problem syncing mirrors [26]
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprseterror: Error in accessing physical storage [26] Marking context invalid.
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc: backend_ctx->prop_ctx_tag=PROPCTXT
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc: backend_ctx->prop_valid=0
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc: backend_ctx->prop_boot_mode=1
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc: begin dumping backenctx->prop_ctx
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_sctx=[0xc1fb0]
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_sctx=[0xc1fb0]
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_native[0]=12
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_native_state[0]=5
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_native_numpage[0]=65536
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_filename[0]=/ocr_voting/OCRfile
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_native[1] is NULL
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_native_state[1]=0
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_native_numpage[1]=0
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_filename[1]=
2011-08-05 18:43:29.449: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_read_device=0
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->numpage=65530
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->totalsize=268435456
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->num_free=0
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_ocrid=440645862
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->version[0]=2
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->version[1]=0
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->bootbuf=[0x107cc0]
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->bootbuf[0]=0x1
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->bootbuf[1]=0x82
2011-08-05 18:43:29.450: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->bootbuf[2]=0x28
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_num_gid=3
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_ocrctx=[0xfd738]
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_os_min_block_size=512
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc_proprhandle: propb_ctx->proprhandle_islocal_only=0
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc: end dumping backenctx->propb_ctx
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc: backend_ctx->metactx=[0xca450]
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc: backend_ctx->prop_sctx=[0xc1fb0]
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc: backend_ctx->prop_sltsmx=[0x0]
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc: backend_ctx->prop_sclsctx=[0xe69f8]
2011-08-05 18:43:29.864: [ OCRRAW][551547456]proprdc: backend_ctx->prop_ctx_ocrctx=[0xfd738]
[ OCRAPI][551547456]procr_ctx_set_invalid_no_abort: ctx set to invalid
[ OCRAPI][551547456]procr_ctx_set_invalid: aborting...
2011-08-05 18:43:32.471: [ default][49043984][ENTER]0
Oracle Database 10g CRS Release 10.2.0.4.0 Production Copyright 1996, 2004, Oracle. All rights reserved
2011-08-05 18:43:32.471: [ default][49043984]0CRS Daemon Starting
2011-08-05 18:43:32.472: [ CRSMAIN][49043984]0Checking the OCR device
2011-08-05 18:43:32.474: [ CRSMAIN][49043984]0Connecting to the CSS Daemon
2011-08-05 18:43:32.860: [ CLSVER][49043984]0Active Version from OCR:10.2.0.4.0
2011-08-05 18:43:32.861: [ CLSVER][49043984]0Active Version and Software Version are same
2011-08-05 18:43:32.861: [ CRSMAIN][49043984]0Initializing OCR
2011-08-05 18:43:32.869: [ OCRRAW][49043984]proprioo: for disk 0 (/ocr_voting/OCRfile), id match (1), my id set (1513150135,1028247821) total id sets (1), 1st set (1
513150135,1028247821), 2nd set (0,0) my votes (2), total votes (2)
2011-08-05 18:43:32.904: [ CRSD][49043984]0ENV Logging level for Module: allcomp 0
2011-08-05 18:43:32.912: [ CRSD][49043984]0ENV Logging level for Module: default 0
2011-08-05 18:43:32.930: [ CRSD][49043984]0ENV Logging level for Module: COMMCRS 0
2011-08-05 18:43:32.950: [ CRSD][49043984]0ENV Logging level for Module: COMMNS 0
2011-08-05 18:43:32.973: [ CRSD][49043984]0ENV Logging level for Module: CRSUI 0
2011-08-05 18:43:33.001: [ CRSD][49043984]0ENV Logging level for Module: CRSCOMM 0
2011-08-05 18:43:33.014: [ CRSD][49043984]0ENV Logging level for Module: CRSRTI 0
2011-08-05 18:43:33.027: [ CRSD][49043984]0ENV Logging level for Module: CRSMAIN 0
2011-08-05 18:43:33.039: [ CRSD][49043984]0ENV Logging level for Module: CRSPLACE 0
2011-08-05 18:43:33.053: [ CRSD][49043984]0ENV Logging level for Module: CRSAPP 0
2011-08-05 18:43:33.064: [ CRSD][49043984]0ENV Logging level for Module: CRSRES 0
2011-08-05 18:43:33.073: [ CRSD][49043984]0ENV Logging level for Module: CRSOCR 0
2011-08-05 18:43:33.076: [ CRSD][49043984]0ENV Logging level for Module: CRSTIMER 0
2011-08-05 18:43:33.085: [ CRSD][49043984]0ENV Logging level for Module: CRSEVT 0
2011-08-05 18:43:33.087: [ CRSD][49043984]0ENV Logging level for Module: CRSD 0
2011-08-05 18:43:33.096: [ CRSD][49043984]0ENV Logging level for Module: CLUCLS 0
2011-08-05 18:43:33.099: [ CRSD][49043984]0ENV Logging level for Module: CLSVER 0
2011-08-05 18:43:33.108: [ CRSD][49043984]0ENV Logging level for Module: OCRRAW 0
2011-08-05 18:43:33.111: [ CRSD][49043984]0ENV Logging level for Module: OCROSD 0
2011-08-05 18:43:32.904: [ CRSD][49043984]0ENV Logging level for Module: allcomp 0
2011-08-05 18:43:32.912: [ CRSD][49043984]0ENV Logging level for Module: default 0
2011-08-05 18:43:32.930: [ CRSD][49043984]0ENV Logging level for Module: COMMCRS 0
2011-08-05 18:43:32.950: [ CRSD][49043984]0ENV Logging level for Module: COMMNS 0
2011-08-05 18:43:32.973: [ CRSD][49043984]0ENV Logging level for Module: CRSUI 0
2011-08-05 18:43:33.001: [ CRSD][49043984]0ENV Logging level for Module: CRSCOMM 0
2011-08-05 18:43:33.014: [ CRSD][49043984]0ENV Logging level for Module: CRSRTI 0
2011-08-05 18:43:33.027: [ CRSD][49043984]0ENV Logging level for Module: CRSMAIN 0
2011-08-05 18:43:33.039: [ CRSD][49043984]0ENV Logging level for Module: CRSPLACE 0
2011-08-05 18:43:33.053: [ CRSD][49043984]0ENV Logging level for Module: CRSAPP 0
2011-08-05 18:43:33.064: [ CRSD][49043984]0ENV Logging level for Module: CRSRES 0
2011-08-05 18:43:33.073: [ CRSD][49043984]0ENV Logging level for Module: CRSOCR 0
2011-08-05 18:43:33.076: [ CRSD][49043984]0ENV Logging level for Module: CRSTIMER 0
2011-08-05 18:43:33.085: [ CRSD][49043984]0ENV Logging level for Module: CRSEVT 0
2011-08-05 18:43:33.087: [ CRSD][49043984]0ENV Logging level for Module: CRSD 0
2011-08-05 18:43:33.096: [ CRSD][49043984]0ENV Logging level for Module: CLUCLS 0
2011-08-05 18:43:33.099: [ CRSD][49043984]0ENV Logging level for Module: CLSVER 0
2011-08-05 18:43:33.108: [ CRSD][49043984]0ENV Logging level for Module: OCRRAW 0
2011-08-05 18:43:33.111: [ CRSD][49043984]0ENV Logging level for Module: OCROSD 0
2011-08-05 18:43:33.120: [ CRSD][49043984]0ENV Logging level for Module: CSSCLNT 0
2011-08-05 18:43:33.123: [ CRSD][49043984]0ENV Logging level for Module: OCRAPI 0
2011-08-05 18:43:33.132: [ CRSD][49043984]0ENV Logging level for Module: OCRUTL 0
2011-08-05 18:43:33.135: [ CRSD][49043984]0ENV Logging level for Module: OCRMSG 0
2011-08-05 18:43:33.144: [ CRSD][49043984]0ENV Logging level for Module: OCRCLI 0
2011-08-05 18:43:33.147: [ CRSD][49043984]0ENV Logging level for Module: OCRCAC 0
2011-08-05 18:43:33.156: [ CRSD][49043984]0ENV Logging level for Module: OCRSRV 0
2011-08-05 18:43:33.159: [ CRSD][49043984]0ENV Logging level for Module: OCRMAS 0
2011-08-05 18:43:33.159: [ CRSMAIN][49043984]0Filename is /u01/app/oracle/product/10.2.0/crs/crs/init/srvorcl02.pid
[ clsdmt][450884160]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=srvorcl02DBG_CRSD))
2011-08-05 18:43:33.201: [ CRSMAIN][49043984]0Using Authorizer location: /u01/app/oracle/product/10.2.0/crs/crs/auth/
2011-08-05 18:43:33.230: [ CRSMAIN][49043984]0Initializing RTI
2011-08-05 18:43:33.256: [CRSTIMER][471855680]0Timer Thread Starting.
2011-08-05 18:43:33.259: [ CRSRES][49043984]0Parameter SECURITY = 1, running in USER Mode
2011-08-05 18:43:33.259: [ CRSMAIN][49043984]0Initializing EVMMgr
2011-08-05 18:43:33.814: [ CRSMAIN][49043984]0CRSD locked during state recovery, please wait.
Resources ONLINE
2011-08-05 18:43:54.782: [ CRSMAIN][49043984]0CRSD recovered, unlocked.
2011-08-05 18:43:54.783: [ CRSMAIN][49043984]0QS socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
2011-08-05 18:43:54.783: [ CRSMAIN][49043984]0QS socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
2011-08-05 18:43:54.802: [ CRSMAIN][49043984]0CRSD UI socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET))
2011-08-05 18:43:54.816: [ CRSMAIN][49043984]0E2E socket on: (ADDRESS=(PROTOCOL=tcp)(HOST=orapriv2)(PORT=49896))
2011-08-05 18:43:54.816: [ CRSMAIN][49043984]0Starting Threads
2011-08-05 18:43:54.816: [ CRSMAIN][49043984]0CRS Daemon Started.
2011-08-05 18:43:54.817: [ CRSMAIN][526381632]0Starting runCommandServer for (UI = 1, E2E = 0). 0
2011-08-05 18:43:54.817: [ CRSMAIN][528478784]0Starting runCommandServer for (UI = 1, E2E = 0). 1
OCRCHECK:
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 262120
Used space (kbytes) : 7856
Available space (kbytes) : 254264
ID : 440645862
Device/File Name : /ocr_voting/OCRfile
Device/File integrity check succeeded
Device/File not configured
Cluster registry integrity check succeeded
Verified /etc/hosts of the 2 nodes , are ok.
Jumbo frames, on the interconnected interfaces are not enabled.
[oracle@srvorcl01 ~]$ crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.4.0]
checked ocr_file permissions:
[oracle@srvorcl02 ~]$ ls -la /ocr_voting
total 18504
drwxr-xr-x 4 oracle oinstall 3896 Nov 19 2010 .
drwxr-xr-x 32 root root 4096 May 11 20:11 ..
-rw-r--r-- 1 oracle oinstall 10240000 Jul 1 2010 CSSfile
drwxr-xr-x 2 oracle oinstall 3896 Nov 19 2010 dbs
drwxr-xr-x 2 oracle oinstall 3896 Jun 30 2010 lost+found
-rw-r----- 1 root oinstall 8765440 Nov 15 2010 OCRfile
[oracle@srvorcl02 ~]$
I'm looking for some suggestion and where to focus attention.
Thanks in advance
Roby

i'm facing this problem and i don't understand why node2 and node1 "sometimes" (1 or 2 days a week) gives this type of error and crs restartsHow did you come to know this ?
Was there any network delay ? Did you check that?
I think it may be because of the netwrok traffic and can you please post /etc/ocfs.conf and also /var/log/messages .. Did you find any thing odd in the /var/log/messages file?
GoodLuck!

Similar Messages

CRS 와 10G REAL APPLICATION CLUSTERS

제품 : ORACLE SERVER
작성날짜 : 2004-11-30
CRS 와 10G REAL APPLICATION CLUSTERS
===================================
PURPOSE
이 문서는, 10g Real Application Cluster의 CRS (Cluster Ready Services)에 대한 추가적인
정보를 제공하는 것을 목적으로 한다.
Explanation
1. CRS 와 10g REAL APPLICATION CLUSTERS
CRS (Cluster Ready Services)는 10g Real Application Cluster의 새로운 기능으로,
모든 플랫폼에 대해 표준화된 클러스터 인터페이스를 제공 해 주고, 이전 버전에서는 없었던
새로운 고가용 서비스를 제공해 준다.
2. CRS 핵심 기능
CRS와 10g RAC를 설치하기 전에, CRS및 10g RAC에 대해 사전에 알아 두어야 할 사항이 있다 :
- 10g RAC를 설치를 위해서는 CRS는 사전에 설치 되고 실행되어야 한다.
- CRS는 하드웨어 공급 업체에서 제공하는 클러스터 제품 (예 : Sun Cluster,
HP Serviceguard, IBM HACMP, TruCluster, Veritas Cluster, Fujitsu Primecluster,
기타 ...) 위에서 실행 될 수도 있고, 하드웨어 공급 업체에서 제공하는 클러스터 제품
없이도 실행될 수 있다. 하드웨어 업체에서 공급하는 클러스터 제품은 9i RAC까지는 반드시
필요했지만, 10g RAC에서는 선택 사항이다.
- CRS HOME 과 ORACLE_HOME 은 반드시 다른 디렉토리에 설치 되어야 한다.
- CRS를 설치하기 전에, voting 파일 또는 OCR (Oracle Configuration Repository) 파일을
설치 할 수 있는 공유된 디렉토리 또는 디바이스가 셋업되어야 한다. voting file은 최소 20MB 정도
크기이며, OCR 파일은, 최소 100MB 크기가 되어야 한다.
- CRS 및 RAC를 설치하기 위해서는 다음과 같은 네트워크 인터페이스가 구성되어야 한다 :
- Public Interface
- Private Interface
- Virtual (Public) Interface
관련된 추가적인 정보는 <Bulletin No: 22345> 참조.
- CRS 설치 후 root.sh를 실행시키면 CRS 서비스를 구동시킨다. 만약 CRS가 정상적으로
구동되지 않는다면, Note 240001.1 참조.
- RAC 노드당 1개의 CRS 데모만 실행 가능.
- 유닉스 시스템의 경우, CRS 서비스는 /etc/inittab 상의 ‘respawn’ entry로 등록되어 있다.
- 네트워크 split이 있을 경우 (노드간 통신 두절) data corruption을 방지 하기 위해 하나
또는 그 이상의 노드에서 리부팅이 발생할 수 있다.
- CRS 서비스를 구동시키는 올바른 방법은 장비를 부팅시키는 것이다.
- 서비스를 중단시키는 올바른 방법은, 장비를 shutdown 시키거나, "init.crs stop" 명령을
실행시키는 것이다.
- CRS 데몬을 kill 시키는 것은 올바른 방법이 아니며, 오직 설치된 CRS를 제거한 경우에만
적용해도 되는 방법이다. (Bulletin No: 22343 참조) 이것은 , 플래그 파일에 불일치가 발생 할
수 있기 때문이다.
- 시스템 유지 보수를 위해서는, OS를 단일 사용자 모드 (single user mode)로 전환한다.
서비스 스택이 구동되면, ps -ef 명령으로 관련된 데몬 프로세스를 확인 할 수 있다 :
[rac1]/u01/home/beta> ps -ef | grep crs
oracle 1363 999 0 11:23:21 ? 0:00 /u01/crs_home/bin/evmlogger.bin -o /u01
oracle 999 1 0 11:21:39 ? 0:01 /u01/crs_home/bin/evmd.bin
root 1003 1 0 11:21:39 ? 0:01 /u01/crs_home/bin/crsd.bin
oracle 1002 1 0 11:21:39 ? 0:01 /u01/crs_home/bin/ocssd.bin
3. CRS DAEMON 기능
다음은 각각의 CRS 데몬 프로세스에 대한 간략한 설명이다 :
CRSD:
- HA 작업을 위한 엔진
- '애플리케이션 자원'관리
- '애플리케이션 자원'을 구동, 정지, fail over 처리
- 애플리케이션 자원 구동/정지/점검 하기 위한 별도의 'actions'을 spawn
- OCR(Oracle Configuration Repository)의 구성 프로파일 관리
- OCR의 현재 알려진 상태를 저장
- root 권한으로 실행
- 장애 발생시 자동으로 재 구동됨
OCSSD:
- OCSSD는 RAC의 일부로, ASM과 함께 단일 인스턴스를 구성함
- 노드 멤버쉽에 대한 액세스를 제공
- 그룹 서비스 제공
- 기본적인 클러스터 lock 기능 제공
- 하드웨어 공급 업체에서 제공하는 클러스터 소프트웨어가 설치되어 있을 경우, 통합을 실시
- 하드웨어 공급 업체에서 제공하는 클러스터 소프트웨어 없이도 실행 가능
- 오라클 계정으로 실행
- 장애로 인한 종료시 시스템 리부팅됨
--- 리부팅은 split brain 현상 발생시, 데이터 corruption 방지를 목적으로 함.
EVMD:
- 특정한 사건 발생 시 이벤트 생성
- 자식 프로세스로 evmlogger를 spawn 시킴
- Evmlogger는 필요시 자식 프로세스를 spawn 시킴
- callout directory를 스캔하고 callout을 호출
- 오라클 계정으로 실행.
- 장애로 인한 종료시 자동으로 재 구동됨
4. CRS 로그 디렉토리
CRS 문제의 원인을 추적할 때는, CRS 홈 디렉토리 아래 디렉토리를 살펴보는 것이
중요하다.
$ORA_CRS_HOME/crs/log - 이 디렉토리는, CRS 자원들에 대한 트레이스를 포함하며,
CRS에 의해 식별된 가입(joining), 탈퇴(leaving), 재구동(restarting), 재배치(relocating)와
관련된 정보들이 기록된다.
$ORA_CRS_HOME/crs/init - crsd.bin 데몬과 관련된 모든 core dump가 기록된다.
$ORA_CRS_HOME/css/log - css 로그는, 재구성(reconfiguration)이나, 성공하지
못한 체크인 (missed checkin), 클라이언트의 css listener로 부터 발생한 연결(connect) 및
연결해제(disconnect)와 관련된 모든 액션을 기록한다. 때에 따라서는 로거에서는 (auth.crit)
유형의 메시지를 남기는데 이것은 오라클에 의해 리부팅이 발생할 때 남는다. 이 정보는
리부팅이 정확히 언제 발생했는지를 확인하는데 사용될 수 있다.
$ORA_CRS_HOME/css/init - 기본적으로는 ocssd로 부터의 core dump 파일을 저장하며, 프로세스의
종료가 심각한 문제로 간주되는 css 데몬의 pid 정보 또한 기록된다. css의 비정상 재 구동이 발생할
경우, core 파일은, core.<pid> 형태로 기록된다.
$ORA_CRS_HOME/evm/log - evn과 evmlogger 데몬의 로그 파일이 기록된다. CRS 또는 CSS 관련 디렉토리
처럼 디버깅 용도로 자주 사용되지는 않는다.
$ORA_CRS_HOME/evm/init - EVM의 pid와 lock 파일이 저장된다. EVM으로 부터 발생한 core 파일 또한
이 디렉토리에 저장된다. 디버깅을 위해서는 Note 1812.1 참조.
$ORA_CRS_HOME/srvm/log - OCR을 위한 로그 파일.
5. CRS 자원의 상태
RAC를 설치하고, RAC root.sh을 실행시키면 VIPCA (Virtual IP Configuration Assistant)가
구동된다. crs_stat 명령을 이용하여 모든 CRS 자원을 확인할 수 있다. 예 :
cd $ORA_CRS_HOME/bin
./crs_stat
NAME=ora.rac1.gsd
TYPE=application
TARGET=ONLINE
STATE=ONLINE
NAME=ora.rac1.oem
TYPE=application
TARGET=ONLINE
STATE=ONLINE
NAME=ora.rac1.ons
TYPE=application
TARGET=ONLINE
STATE=ONLINE
NAME=ora.rac1.vip
TYPE=application
TARGET=ONLINE
STATE=ONLINE
NAME=ora.rac2.gsd
TYPE=application
TARGET=ONLINE
STATE=ONLINE
NAME=ora.rac2.oem
TYPE=application
TARGET=ONLINE
STATE=ONLINE
NAME=ora.rac2.ons
TYPE=application
TARGET=ONLINE
STATE=ONLINE
NAME=ora.rac2.vip
TYPE=application
TARGET=ONLINE
STATE=ONLINE
CRS 자원을 읽기 쉬운 형태로 확인하기 위한 스크립트 또한 사용할 수 있다.
다음은 shell script의 예이다 :
--------------------------- Begin Shell Script -------------------------------
#!/usr/bin/ksh
# Sample 10g CRS resource status query script
# Description:
# - Returns formatted version of crs_stat -t, in tabular
# format, with the complete rsc names and filtering keywords
# - The argument, $RSC_KEY, is optional and if passed to the script, will
# limit the output to HA resources whose names match $RSC_KEY.
# Requirements:
# - $ORA_CRS_HOME should be set in your environment
RSC_KEY=$1
QSTAT=-u
AWK=/usr/xpg4/bin/awk # if not available use /usr/bin/awk
# Table header:echo ""
$AWK \
'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State";
printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}'
# Table body:
$ORA_CRS_HOME/bin/crs_stat $QSTAT | $AWK \
'BEGIN { FS="="; state = 0; }
$1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};
state == 0 {next;}
$1~/TARGET/ && state == 1 {apptarget = $2; state=2;}
$1~/STATE/ && state == 2 {appstate = $2; state=3;}
state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'
--------------------------- End Shell Script -------------------------------
실행 결과 예시 :
[opcbsol1]/u01/home/usupport> ./crsstat
HA Resource Target State
ora.V10SN.V10SN1.inst ONLINE ONLINE on opcbsol1
ora.V10SN.V10SN2.inst ONLINE ONLINE on opcbsol2
ora.V10SN.db ONLINE ONLINE on opcbsol2
ora.opcbsol1.ASM1.asm ONLINE ONLINE on opcbsol1
ora.opcbsol1.LISTENER_OPCBSOL1.lsnr ONLINE ONLINE on opcbsol1
ora.opcbsol1.gsd ONLINE ONLINE on opcbsol1
ora.opcbsol1.ons ONLINE ONLINE on opcbsol1
ora.opcbsol1.vip ONLINE ONLINE on opcbsol1
ora.opcbsol2.ASM2.asm ONLINE ONLINE on opcbsol2
ora.opcbsol2.LISTENER_OPCBSOL2.lsnr ONLINE ONLINE on opcbsol2
ora.opcbsol2.gsd ONLINE ONLINE on opcbsol2
ora.opcbsol2.ons ONLINE ONLINE on opcbsol2
ora.opcbsol2.vip ONLINE ONLINE on opcbsol2
6. CRS 자원 관리
자원을 관리하기 위해서는, srvctl 명령을 사용한다. 다음은 명령어 문법 예제이다.
1) CRS 자원 상태
srvctl status database -d <database-name> [-f] [-v] [-S <level>]
srvctl status instance -d <database-name> -i <instance-name> >[,<instance-name-list>]
[-f] [-v] [-S <level>]
srvctl status service -d <database-name> -s <service-name>[,<service-name-list>]
[-f] [-v] [-S <level>]
srvctl status nodeapps [-n <node-name>]
srvctl status asm -n <node_name>
예제:
데이터베이스의 상태, 모든 인스턴스와 모든 서비스
srvctl status database -d ORACLE -v
이름이 부여된 인스턴스의 상태와 현재 서비스
srvctl status instance -d ORACLE -i RAC01, RAC02 -v
이름이 부여된 서비스의 상태.
srvctl status service -d ORACLE -s ERP -v
데이터베이스 애플리케이션을 지원하는 모든 노드의 상태.
srvctl status node
2) CRS 자원의 구동
srvctl start database -d <database-name> [-o < start-options>]
[-c <connect-string> | -q]
srvctl start instance -d <database-name> -i <instance-name>
[,<instance-name-list>] [-o <start-options>] [-c <connect-string> | -q]
srvctl start service -d <database-name> [-s <service-name>[,<service-name-list>]]
[-i <instance-name>] [-o <start-options>] [-c <connect-string> | -q]
srvctl start nodeapps -n <node-name>
srvctl start asm -n <node_name> [-i <asm_inst_name>] [-o <start_options>]
예제:
데이터베이스를 모든 활성화된 인스턴스와 함께 구동.
srvctl start database -d ORACLE
이름이 부여된 인스턴스의 구동.
srvctl start instance -d ORACLE -i RAC03, RAC04
이름이 부여된 서비스의 구동. 연관된 인스턴스는 필요시 구동됨.
srvctl start service -d ORACLE -s CRM
이름이 부여된 인스턴스의 서비스의 구동.
srvctl start service -d ORACLE -s CRM -i RAC04
노드 애플리케이션의 구동.
srvctl start nodeapps -n myclust-4
3) CRS 자원의 정지
srvctl stop database -d <database-name> [-o <stop-options>]
[-c <connect-string> | -q]
srvctl stop instance -d <database-name> -i <instance-name> [,<instance-name-list>]
[-o <stop-options>][-c <connect-string> | -q]
srvctl stop service -d <database-name> [-s <service-name>[,<service-name-list>]]
[-i <instance-name>][-c <connect-string> | -q] [-f]
srvctl stop nodeapps -n <node-name>
srvctl stop asm -n <node_name> [-i <asm_inst_name>] [-o <start_options>]
예제:
데이터베이스, 모든 인스턴스, 모든 서비스를 정지 시킴.
srvctl stop database -d ORACLE
이름이 부여된 인스턴스를 정지 시킴. 그 전에 우선 존재하는 모든 서비스를 재배치 함.
srvctl stop instance -d ORACLE -i RAC03,RAC04
서비스를 정지시킴.
srvctl stop service -d ORACLE -s CRM
이름이 부여된 인스턴스의 서비스를 정지 시킴.
srvctl stop service -d ORACLE -s CRM -i RAC04
노드 애플리케이션을 정지 시킴. 인스턴스와 서비스 역시 정지됨.
srvctl stop nodeapps -n myclust-4
4) CRS 자원의 추가
srvctl add database -d <name> -o <oracle_home> [-m <domain_name>] [-p <spfile>]
[-A <name|ip>/netmask] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY}]
[-s <start_options>] [-n <db_name>]
srvctl add instance -d <name> -i <inst_name> -n <node_name>
srvctl add service -d <name> -s <service_name> -r <preferred_list>
[-a <available_list>] [-P <TAF_policy>] [-u]
srvctl add nodeapps -n <node_name> -o <oracle_home>
[-A <name|ip>/netmask[/if1[|if2|...]]]
srvctl add asm -n <node_name> -i <asm_inst_name> -o <oracle_home>
OPTIONS:
-A vip range, node, and database, address specification. The format of
address string is:
[<logical host name>]/<VIP address>/<net mask>[/<host interface1[ |
host interface2 |..]>] [,] [<logical host name>]/<VIP address>/<net mask>
[<host interface1[ | host interface2 |..]>]
-a for services, list of available instances, this list cannot include
preferred instances
-m domain name with the format “us.mydomain.com”
-n node name that will support one or more instances
-o $ORACLE_HOME to locate Oracle binaries
-P for services, TAF preconnect policy - NONE, PRECONNECT
-r for services, list of preferred instances, this list cannot include
available instances.
-s spfile name
-u updates the preferred or available list for the service to support the
specified instance. Only one instance may be specified with the -u
switch. Instances that already support the service should not be
included.
예제:
새로운 노드의 추가.
srvctl add nodeapps -n myclust-1 -o $ORACLE_HOME ?A
139.184.201.1/255.255.255.0/hme0
새로운 데이터 베이스의 추가.
srvctl add database -d ORACLE -o $ORACLE_HOME
이미 존제하는 데이터베이스에 이름이 부여된 인스턴스 추가.
srvctl add instance -d ORACLE -i RAC01 -n myclust-1
srvctl add instance -d ORACLE -i RAC02 -n myclust-2
srvctl add instance -d ORACLE -i RAC03 -n myclust-3
서비스를 이미 존재하는 데이터베이스에 추가하며, 선호되는 인스턴스를 지정 (-r)하고,
가용한 인스턴스를 지정함(-a). 가용 인스턴스에 대해서는 기본 failover를 사용함.
srvctl add service -d ORACLE -s STD_BATCH -r RAC01,RAC02 -a RAC03,RAC04
이미 존재하는 데이터베이스에 선호되는 인스턴스를 list 1, 가용한 인스턴스를 list 2에
추가함. 가용 인스턴스에 대해서는 사전연결 (preconnect) 방식을 사용함.
srvctl add service -d ORACLE -s STD_BATCH -r RAC01,RAC02 -a RAC03,RAC04 -P PRECONNECT
5) CRS RE자원의 제거
srvctl remove database -d <database-name>
srvctl remove instance -d <database-name> [-i <instance-name>]
srvctl remove service -d <database-name> -s <service-name> [-i <instance-name>]
srvctl remove nodeapps -n <node-name>
예제:
데이터베이스에 대한 애플리케이션의 제거.
srvctl remove database -d ORACLE
이미 존재하는 데이터베이스의 이름이 부여된 인스턴스에 대한 애플리케이션 제거.
srvctl remove instance -d ORACLE -i RAC03
srvctl remove instance -d ORACLE -i RAC04
서비스 제거.
srvctl remove service -d ORACLE -s STD_BATCH
인스턴스로부터 서비스 제거.
srvctl remove service -d ORACLE -s STD_BATCH -i RAC03,RAC04
노드로 부터 모든 노드 애플리케이션 제거.
srvctl remove nodeapps -n myclust-4
6) CRS 자원의 변경
srvctl modify database -d <name> [-n <db_name] [-o <ohome>] [-m <domain>]
[-p <spfile>] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY}]
[-s <start_options>]
srvctl modify instance -d <database-name> -i <instance-name> -n <node-name>
srvctl modify instance -d <name> -i <inst_name> {-s <asm_inst_name> | -r}
srvctl modify service -d <database-name> -s <service_name> -i <instance-name>
-t <instance-name> [-f]
srvctl modify service -d <database-name> -s <service_name> -i <instance-name>
-r [-f]
srvctl modify nodeapps -n <node-name> [-A <address-description> ] [-x]
OPTIONS:
-i <instance-name> -t <instance-name> the instance name (-i) is replaced by the
instance name (-t)
-i <instance-name> -r the named instance is modified to be a preferred instance
-A address-list for VIP application, at node level
-s <asm_inst_name> add or remove ASM dependency
예제:
인스턴스가 다른 노드에서 실행되도록 변경.
srvctl moinstance -d ORACLE -n myclust-4
서비스가 다른 노드에서 실행되도록 변경.
srvctl modify service -d ORACLE -s HOT_BATCH -i RAC01 -t RAC02
인스턴스가 서비스의 선호되는 인스턴스가 되도록 변경.
srvctl modify service -d ORACLE -s HOT_BATCH -i RAC02 ?r
7) SERVICE의 재 배치
srvctl relocate service -d <database-name> -s <service-name> [-i <instance-name >]-t<instance-name > [-f]
예제:
서비스를 한 인스턴스에서 다른 인스턴스로 재 배치
srvctl relocate service -d ORACLE -s CRM -i RAC04 -t RAC01
8) CRS 자원을 활성화 (자원은 이 기능을 사용할 당시 실행중이거나, 정지된 상태일 수 있음)
srvctl enable database -d <database-name>
srvctl enable instance -d <database-name> -i <instance-name> [,<instance-name-list>]
srvctl enable service -d <database-name> -s <service-name>] [, <service-name-list>] [-i <instance-name>]
예제:
데이터베이스를 활성화.
srvctl enable database -d ORACLE
이름이 부여된 인스턴스의 활성화.
srvctl enable instance -d ORACLE -i RAC01, RAC02
서비스의 활성화.
srvctl enable service -d ORACLE -s ERP,CRM
이름이 부여된 인스턴스에서 서비스의 활성화.
srvctl enable service -d ORACLE -s CRM -i RAC03
9) CRS 자원의 비활성화 (자원은 이 기능을 사용할 당시 정지된 상태 이어야만 함)
srvctl disable database -d <database-name>
srvctl disable instance -d <database-name> -i <instance-name> [,<instance-name-list>]
srvctl disable service -d <database-name> -s <service-name>] [,<service-name-list>] [-i <instance-name>]
예제:
데이터베이스를 전역(global) 비활성화 시킴.
srvctl disable database -d ORACLE
이름이 부여된 인스턴스의 비활성화.
srvctl disable instance -d ORACLE -i RAC01, RAC02
서비스를 전역(global) 비활성화.
srvctl disable service -d ORACLE -s ERP,CRM
이름이 부여된 인스턴스상의 서비스를 비 활성화.
srvctl disable service -d ORACLE -s CRM -i RAC03,RAC04
추가 정보는 Oracle10g Real Application Clusters Administrator's Guide - Appendix B 참조
Example
Reference Documents
<Note:259301.1> CRS and 10g Real Application Clusters

To download the registered logos by Oracle for OCP, OCA, OCE, & OCM Certifications.
You have to request to Oracle Exam Support Team to provide the link and the credentials (User Name/Password) to download the same.
Mail to the following E-Mail: [email protected]
Note 1: To get the response on time, you may need to provide your prometric information details i.e. Exam Passed Date, Prometric ID, Full Name, and Corresponding Address.
Note 2: You will never have to disclose your credentials (User Name/Password) to anybody, as Oracle is going to give this for you especially.
Regards,
Sabdar Syed,
http://sabdarsyed.blogspot.com

Failure at final check of Oracle CRS stack.AIX oracle 10g RAC with GPFS

HI ，I install oracle 10R2 RAC using GPFS，os is AIX 6.1,but when I installed CRS，at executing the second root.sh ,I am in trouble ，the error information as follow :
Failure at final check of Oracle CRS stack.
10
I look up the log file the information :
The OCR location /data_gpfs/CRS/ocr_disk2 is inaccessible. Details in /orac
leapp/product/10.2.0/crs/log/p615a/client/ocrconfig_6881286.log.
the ocrconfig_6881286.log.information:
2014-01-24 01:32:20.361: [ OCRCONF][1]ocrconfig starts...
2014-01-24 01:32:20.389: [ OCRCONF][1]Upgrading OCR data
2014-01-24 01:32:20.391: [ OCROSD][1]utread:3: problem reading buffer 100f21d0 buflen 512 retval 0 phy_offset 102400 retry 0
2014-01-24 01:32:20.391: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2014-01-24 01:32:20.391: [ OCROSD][1]utread:3: problem reading buffer 100f21d0 buflen 512 retval 0 phy_offset 102400 retry 0
2014-01-24 01:32:20.391: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2014-01-24 01:32:20.391: [ OCROSD][1]utread:3: problem reading buffer ffffb1d0 buflen 4096 retval 0 phy_offset 102400 retry 0
2014-01-24 01:32:20.392: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2014-01-24 01:32:20.392: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2014-01-24 01:32:20.392: [ OCROSD][1]utread:3: problem reading buffer ffffb1d0 buflen 4096 retval 0 phy_offset 102400 retry 0
2014-01-24 01:32:20.392: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2014-01-24 01:32:20.392: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2014-01-24 01:32:20.392: [ OCRRAW][1]proprioini: both disks are not OCR formatted
2014-01-24 01:32:20.392: [ OCRRAW][1]proprinit: Could not open raw device
2014-01-24 01:32:20.392: [ default][1]a_init:7!: Backend init unsuccessful : [26]
2014-01-24 01:32:20.393: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
2014-01-24 01:32:20.393: [ OCRAPI][1]a_init:7!: Backend init unsuccessful : [33]
propriogid:1: INVALID FORMAT
2014-01-24 01:32:20.516: [ OCRRAW][1]propriowv: Vote information on disk 0 [/data_gpfs/CRS/ocr_disk1] is adjusted from [0/0] to [1/2]
2014-01-24 01:32:20.527: [ OCRRAW][1]propriowv: Vote information on disk 1 [/data_gpfs/CRS/ocr_disk2] is adjusted from [0/0] to [1/2]
2014-01-24 01:32:20.960: [ OCRRAW][1]propriniconfig:No 92 configuration
2014-01-24 01:32:20.960: [ OCRAPI][1]a_init:6a: Backend init successful
2014-01-24 01:32:21.191: [ OCRCONF][1]Initialized DATABASE keys in OCR
2014-01-24 01:32:21.349: [ OCRCONF][1]Successfully set skgfr block 0
2014-01-24 01:32:21.351: [ OCRCONF][1]Exiting [status=success]...
I dont know what cause this error,i am really trouble,who can help me !!!

HI ，I install oracle 10R2 RAC using GPFS，os is AIX 6.1,but when I installed CRS，at executing the second root.sh ,I am in trouble ，the error information as follow :
Failure at final check of Oracle CRS stack.
10
I look up the log file the information :
The OCR location /data_gpfs/CRS/ocr_disk2 is inaccessible. Details in /orac
leapp/product/10.2.0/crs/log/p615a/client/ocrconfig_6881286.log.
the ocrconfig_6881286.log.information:
2014-01-24 01:32:20.361: [ OCRCONF][1]ocrconfig starts...
2014-01-24 01:32:20.389: [ OCRCONF][1]Upgrading OCR data
2014-01-24 01:32:20.391: [ OCROSD][1]utread:3: problem reading buffer 100f21d0 buflen 512 retval 0 phy_offset 102400 retry 0
2014-01-24 01:32:20.391: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2014-01-24 01:32:20.391: [ OCROSD][1]utread:3: problem reading buffer 100f21d0 buflen 512 retval 0 phy_offset 102400 retry 0
2014-01-24 01:32:20.391: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2014-01-24 01:32:20.391: [ OCROSD][1]utread:3: problem reading buffer ffffb1d0 buflen 4096 retval 0 phy_offset 102400 retry 0
2014-01-24 01:32:20.392: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2014-01-24 01:32:20.392: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2014-01-24 01:32:20.392: [ OCROSD][1]utread:3: problem reading buffer ffffb1d0 buflen 4096 retval 0 phy_offset 102400 retry 0
2014-01-24 01:32:20.392: [ OCROSD][1]utread:4: problem reading the buffer errno 2 errstring No such file or directory
2014-01-24 01:32:20.392: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2014-01-24 01:32:20.392: [ OCRRAW][1]proprioini: both disks are not OCR formatted
2014-01-24 01:32:20.392: [ OCRRAW][1]proprinit: Could not open raw device
2014-01-24 01:32:20.392: [ default][1]a_init:7!: Backend init unsuccessful : [26]
2014-01-24 01:32:20.393: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
2014-01-24 01:32:20.393: [ OCRAPI][1]a_init:7!: Backend init unsuccessful : [33]
propriogid:1: INVALID FORMAT
2014-01-24 01:32:20.516: [ OCRRAW][1]propriowv: Vote information on disk 0 [/data_gpfs/CRS/ocr_disk1] is adjusted from [0/0] to [1/2]
2014-01-24 01:32:20.527: [ OCRRAW][1]propriowv: Vote information on disk 1 [/data_gpfs/CRS/ocr_disk2] is adjusted from [0/0] to [1/2]
2014-01-24 01:32:20.960: [ OCRRAW][1]propriniconfig:No 92 configuration
2014-01-24 01:32:20.960: [ OCRAPI][1]a_init:6a: Backend init successful
2014-01-24 01:32:21.191: [ OCRCONF][1]Initialized DATABASE keys in OCR
2014-01-24 01:32:21.349: [ OCRCONF][1]Successfully set skgfr block 0
2014-01-24 01:32:21.351: [ OCRCONF][1]Exiting [status=success]...
I dont know what cause this error,i am really trouble,who can help me !!!

Oracle 10g CRS autorecovery from network failures - Solaris with IPMP

Hi all,
Just wondering if anyone has experience with a setup similar to mine. Let me first apologise for the lengthy introduction that follows >.<
A quick run-down of my implementation: Sun SPARC Solaris 10, Oracle CRS, ASM and RAC database patched to version 10.2.0.4 respectively, no third-party cluster software used for a 2-node cluster. Additionally, the SAN storage is attached directly with fiber cable to both servers, and the CRS files (OCR, voting disks) are always visible to the servers, there is no switch/hub between the server and the storage. There is IPMP configured for both the public and interconnect network devices. When performing the usual failover tests for IPMP, both the OS logs and the CRS logs show a failure detected, and a failover to the surviving network interface (on both the public and the private network devices).
For the private interconnect, when both of the network devices are disabled (by manually disconnecting the network cables), this results in the 2nd node rebooting, and the CRS process starting, but unable to synchronize with the 1st node (which is running fine the whole time). Further, when I look at the CRS logs, it is able to correctly identify all the OCR files and voting disks. When the network connectivity is restored, both the OS and CRS logs reflect this connection has been repaired. However, the CRS logs at this point still state that node 1 (which is running fine) is down, and the 2nd node attempts to join the cluster as the master node. When I manually run the 'crsctl stop crs' and 'crsctl start crs' commands, this results in a message stating that the node is going to be rebooted to ensure cluster integrity, and the 2nd node reboots, starts the CRS daemons again at startup, and joins the cluster normally.
For the public network, when the 2nd node is manually disconnected, the VIP is seen to not failover, and any attempts to connect to this node via the VIP result in a timeout. When connectivity is restored, as expected the OS and CRS logs acknowledge the recovery, and the VIP for node 2 automatically fails over, but the listener goes down as well. Using the 'srvctl start listener' command brings it up again, and everything is fine. During this whole process, the database instance runs fine on both nodes.
From the case studies above, I can see that the network failures are detected by the Oracle Clusterware, and a simple command run once this failure is repaired restores full functionality to the RAC database. However, is there anyway to automate this recovery, for the 2 cases stated above, so that there is no need for manual intervention by the DBAs? I was able to test case 2 (public network) with the Oracle document 805969.1 (VIP does not relocate back to the original node after public network problem is resolved), is there a similar workaround for the interconnect?
Any and all pointers would be appreciated, and again, sorry for the lengthy post.
Edited by: NS Selvam on 16-Dec-2009 20:36
changed some minor typos

hi
i ve given the shell script.i just need to run that i usually get the op like
[root@rac-1 Desktop]# sh iscsi-corntab.sh
Logging in to [iface: default, target: iqn.2010-02-23.de.sayantan-chakraborty:storage.disk1.amiens.sys1.xyz, portal: 192.168.181.10,3260]
Login to [iface: default, target: iqn.2010-02-23.de.sayantan-chakraborty:storage.disk1.amiens.sys1.xyz, portal: 192.168.181.10,3260]: successfulthe script contains :
iscsiadm -m node -T iqn.2010-02-23.de.sayantan-chakraborty:storage.disk1.amiens.sys1.xyz -p 192.168.181.10 -l
iscsiadm -m node -T iqn.2010-02-23.de.sayantan-chakraborty:storage.disk1.amiens.sys1.xyz -p 192.168.181.10 --op update -n node.startup -v automatic
(cd /dev/disk/by-path; ls -l *sayantan-chakraborty* | awk '{FS=" "; print $9 " " $10 " " $11}')
[root@rac-1 Desktop]# (cd /dev/disk/by-path; ls -l *sayantan-chakraborty* | awk '{FS=" "; print $9 " " $10 " " $11}')
ip-192.168.181.10:3260-iscsi-iqn.2010-02-23.de.sayantan-chakraborty:storage.disk1.amiens.sys1.xyz-lun-1 -> ../../sdc
[root@rac-1 Desktop]# can you post the oput of ls /dev/iscsi ??you may get like this:
[root@rac-1 Desktop]# ls /dev/iscsi
xyz
[root@rac-1 Desktop]#

Can we have separate 10gR2 RAC CRS and 11gR2 RAC CRS homes on same Unix s

We'd like to test the upgrade of our 10gR2 RAC databases to 11gR2 RAC.
We have six databases on the same server and want to install a separate 11g RAC stack (11g CRS, 11g ASM, 11g RDBMS) and upgrade each 10g database from 10g RAC stack on same server (10g CRS, 10g ASM, 10g RDBMS) . Is it ok to have 10g CRS home and 11g CRS home same server? If not, how best to upgrade without buying more hardware?
Thanks in advance,
Lou

In more detail, the main issue/question is whether there can be 2 Oracle CRS homes on the same server i.e. separate 10gR2 and 11gR2 CRS homes residing on the same server for the purposes of (gradually) upgrading all Oracle RDBMS homes from 10g RAC to 11g RAC.
Can there be? What would be the implications for the OCR and voting disk? In our case 10g CRS has OCR and Voting Disk on raw devices. In 11gR2, these would need to be migrated/upgraded/stored in ASM, right?
We have 6 10g rdbms homes in addition to the one 10g CRS home and one 10g ASM home. To upgrade can we
- Install 11gR2 CRS in a separate home (this would now include ASM as well)
- Install 11gR2 RDBMS home
- Upgrade one of the 6 10g databases while leaving the others untouched as 10g databases in the separate 10g CRS, 10g ASM stack?
Any ideas, pointers greatly appreciated.
Thanks,
Lou

Help needed : How to move OCR and Voting Disk

Hi all experts...
I am Mahesh. Working in DXB as DBA. We have serveral RAC installed clients. Among those one major client wants their OCR and Voting Disk to be moved from existing SAN to a new SAN which will be configured. Only OCR and Voting Disks needs to be moved.
Details:
=====
Oracle Clusterware,ASM and DB version 10.2.0.3.0
Windows 2003 Server Enterprise Edition
2-Node RAC
OCR and Voting disks are on RAW devices (not on CFS or NFS)
I have read the clusterware administration document and also Metalink Note 428681.1 for the same.
But it does not contain information how to move it to another new RAW in a new SAN device. (we need to keep it in raw device itself)
How to identify the new disk path from windows operating system ? Then only we can move the OCR and VD right ???
Help me for the same please...
Thanks & regards,
Mahesh.

Hi Mashes,
Could you list out the steps you did after you create the link on the new SAN using GUIObjectManager.exe?
I tried on my VM and also in customer's test environment but I only successful using the following steps
1. shutdown asm service, database service, listener service, CRS, EVM and CSS
2. Object Manager Service is kept running
3. from both nodes I issued the following command
ocrconfig -repair \\.\OCRPRIMARY
ocrconfig -repair \\.\OCRMIRROR
4. according to metalink id 428681.1, I should do the following
ocrconfig -overwrite
However I was not successful because that command returns an error message (PROT-1: Failed to initialize ocr config).
Therefore, I ran the following command
ocrconfig -restore D:\oracle\1020\CRS\cdata\crs\backup00.ocr
5. after that command, ocrcheck giving me the result and the ocr primary and mirror have been moved to the new partition.
6. start the whole services starting from CSS, EVM, CRS, ASM, Listerner and Database service.
Although what I experienced seems a bit weird and not the same as what was written in metalink, the RAC is running fine until now.
What I would like to know is your experience on this. May be you can share it here.
Thank you,
Adhika

How to determine raw disks configured for OCR/voting disk, ASM spfile

I have a two-node Oracle 10gR2 RAC configuration using raw disks. Basically, raw disks are used for CRS's OCR/voting disks, ASM's ASM spfile and disk groups.
Is there a better way to figure out what raw disks are configured in Oracle using methods other than those shown below:
- To find out votedisk:
# crsctl query css votedisk
0. 0 /dev/ora_crs_vote1
1. 0 /dev/ora_crs_vote2
2. 0 /dev/ora_crs_vote3
- To find out OCR:
# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 525836
Used space (kbytes) : 3856
Available space (kbytes) : 521980
ID : 1603037329
Device/File Name : /dev/ora_ocr_file1
Device/File integrity check succeeded
Device/File Name : /dev/ora_ocr_file2
Device/File integrity check succeeded
Cluster registry integrity check succeeded
- Is there a way to figure out what disk device is used for ASM spfile?
- To find out raw disks configured for disk groups:
while connected to the ASM instance in sqlplus:
SQL> select name,path from v$asm_disk where name like 'DG%';
NAME
PATH
DG_DC_ASM_0000
/dev/rhdiskpower13
DG_DC_ASM_0001
/dev/rhdiskpower14
DG_DC_ASM_0002
/dev/rhdiskpower15
NAME
PATH
DG_DC_ASM_0003
/dev/rhdiskpower22

http://docs.oracle.com/cd/B19306_01/install.102/b14203/storage.htm#BABFFBBA
and
Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 [ID 465001.1]
Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 [ID 564580.1]

OCR/Votting disk change location

Hi,
I need to move my 2 nodes RAC (Solaris 10 / Oracle 10g (10.2.0.3)) to a new data center.
My OCR and Votting disks are on a SAN (x) and on the new datacenter I want to put them on a new SAN (y).
How can I perform this server movement without losing the current RAC/Clusterware configuration?
I was planning to do the following:
- Backup current databases
- Backup spfile
- Backup Diskgroups/ASM configuration
- Delete databases
- Delete Diskgroups/ASM
- Shutdown CRS (crsctl stop crs)
- Backup OCR disks (ocrconfig -export export_filename)
- Backup Votting disks (dd if=voting_disk_name of=backup_file_name)
- Turn off the servers
- Move them to the new Datacenter
- Attch the new SAN
- Restore Votting disks (dd if=backup_file_name of=voting_disk_name)
- Restore OCR disks ( ocrconfig -import export_filename)
- Then recreate Diskgroups/ASM
- Restore databases
Is this the right procedure/commands to accomplish it without losing the RAC configuration?
I am trying to avoid rebuilding the whole RAC from scratch.
Thanks,
AShum

I forgot to mention that there is no connection between the old and new datacenter. If there would, I would just present the new SAN space from the new datacenter to the old one, so I can just add the new OCR/Votting disk to the current ones (mirror) and then remove the old disks.
Same for the ASM and databases, but unfortunally, there is no connection between this two datacenter.
It is like moving the servers from one state to another.
Thanks,

CRS Features

Hi Experts,
Need some help in understanding the following concepts:
CRS Deamon is responsible for maintaing the high availability of the resources in the RAC envi. What resources is it maintaing are they ASM,DATABASE,LISTENER,GSD,GCS,ONS please help me in understanding this feature.
CSS is responsible for group services, what exactly are group services?
Cluster interconnects: IS this used only for interinstance communication or is it also used for files transfer for exmaple lets say for transfering the installtion files from one node to another?
Please advice
Thanks for your help in advance.

Thanks for the help sir,
I am using this guide as my reference but not able to get the indept meaning of the core functionality offered by crs in terms on Resources and Services and CSS in terms of Group service.
Does CRS restarts any failed resources such as asm,db,listeners,nodeapps etc...
Please help me to clarify the fundamental concept..
-Thanks

3rd voting disk on nfs share

Hey, as described in Oracle Clusterware 11g Release 2 (11.2) – Using standard NFS to support a third voting file for extended cluster configurations
I added the nfs file to my OCR diskgroup in ASM as quorum device.
/crs/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
1. ONLINE 0eb6aaa85e5c4fd6bf9d0481cfd7d517 (ORCL:VOTE1) [OCR]
2. ONLINE 5a66449fbada4f34bf0c3be4574f03bc (ORCL:VOTE2) [OCR]
3. ONLINE 76ad295ebc054f07bfea124fb08da432 (ORCL:VOTE3) [OCR]
Located 3 voting disk(s).
asmca is showing also ORCL:VOTE3 and /voting_disk/vote3_nfs
Is my system now correctly configured, or do I still need to change my votedisk configuration ?
crsctl replace..... ?
Chris

Hey Levi, I setup a test enviroment for my problem.
I added an ocrmirror to my OCR diskgroup (containing 4 disk, 22)
My ocrmirror is diskgroup DATA (22 disk with 2 failgroups) - the adding of the ocrmirror was successful. ocrcheck showed both diskgroups available
Then I removed 2 disk from storage 1 - so OCR diskgroup had only 2 disks left. Right away the logfiles show errors. OCR inaccesible. +DATA can´t be used due to the error below.
So what I am doing wrong, while adding the ocrmirror ? Which command to use, for replacing +OCR diskgroup ?
[client(27318)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /crs/log/rac1/client/ocrcheck_27318.log.
2012-01-06 22:52:57.393
[client(27318)]CRS-1011:OCR cannot determine that the OCR content contains the latest updates. Details in /crs/log/rac1/client/ocrcheck_27318.log.
2012-01-06 22:53:33.859
2012-01-06 23:00:17.272: [ OCRRAW][3674175232]proprior: Header check from OCR device 0 offset 6574080 failed (26).
2012-01-06 23:00:17.272: [ OCRRAW][3674175232]proprior: Retrying buffer read from another mirror for disk group [+OCR] for block at offset [6574080]
2012-01-06 23:00:17.273: [ OCRASM][3674175232]proprasmres: Total 0 mirrors detected
2012-01-06 23:00:17.273: [ OCRASM][3674175232]proprasmres: Only 1 mirror found in this disk group.
2012-01-06 23:00:17.273: [ OCRASM][3674175232]proprasmres: Need to invoke checkdg. Mirror #0 has an invalid buffer.
2012-01-06 23:00:17.300: [ OCRASM][3674175232]ASM Error Stack : ORA-27091: unable to queue I/O
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-06512: at line 4
OCRCHECK
racle Database 11g Clusterware Release 11.2.0.3.0 - Production Copyright 1996, 2011 Oracle. All rights reserved.
2012-01-06 22:55:25.572: [OCRCHECK][903825152]ocrcheck starts...
2012-01-06 22:55:25.640: [ OCRASM][903825152]proprasmo: kgfoCheckMount return [6]. Cannot proceed with dirty open.
2012-01-06 22:55:25.640: [ OCRASM][903825152]proprasmo: Error in open/create file in dg [OCR]
[ OCRASM][903825152]SLOS : SLOS: cat=6, opn=kgfo, dep=0, loc=kgfoCkMt03
2012-01-06 22:55:25.640: [ OCRASM][903825152]ASM Error Stack :
2012-01-06 22:55:25.667: [ OCRASM][903825152]proprasmo: kgfoCheckMount returned [6]
2012-01-06 22:55:25.667: [ OCRASM][903825152]proprasmo: The ASM disk group OCR is not found or not mounted
2012-01-06 22:55:25.668: [ OCRRAW][903825152]proprioo: Failed to open [+OCR]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2012-01-06 22:55:25.707: [ OCRRAW][903825152]proprioini: disk 1 (+DATA) does not have enough votes (1,2)
2012-01-06 22:55:25.707: [ OCRRAW][903825152]proprioo: Not enought quorum to open the disks (26)
2012-01-06 22:55:25.707: [ OCRRAW][903825152]proprinit: Could not open raw device
2012-01-06 22:55:25.707: [ OCRASM][903825152]proprasmcl: asmhandle is NULL
2012-01-06 22:55:25.709: [ default][903825152]a_init:7!: Backend init unsuccessful : [26]
2012-01-06 22:55:25.709: [OCRCHECK][903825152]initreboot: Failed to initialize OCR in REBOOT level. Retval:[26] Error:[PROC-26: Error while accessing the physical storage

Uninstall & Reinstall RAC (10GR2) on Solaris 10 SPARC

Hi,
I'm trying to install 2 node RAC. I got below errors while executing "root.sh" at the end of installation on both nodes.
[client(19083)]CRS-1011:OCR cannot determine that the OCR content contains the latest updates.
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
2011-06-09 18:37:58.698: [ OCRCONF][1]ocrconfig starts...
2011-06-09 18:37:58.701: [ OCRCONF][1]Upgrading OCR data
2011-06-09 18:37:59.363: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2011-06-09 18:37:59.363: [ OCRRAW][1]proprioini: disk 0 (/dev/rdsk/emcpower5e) doesn't have enough votes (1,2)
2011-06-09 18:37:59.376: [ OCRRAW][1]proprinit: Could not open raw device
2011-06-09 18:37:59.376: [ default][1]a_init:7!: Backend init unsuccessful : [26]
2011-06-09 18:37:59.377: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
2011-06-09 18:37:59.377: [ OCRAPI][1]a_init:7!: Backend init unsuccessful : [33]
2011-06-09 18:37:59.377: [ OCRCONF][1]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
2011-06-09 18:37:59.393: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2011-06-09 18:37:59.393: [ OCRRAW][1]proprioini: disk 0 (/dev/rdsk/emcpower5e) doesn't have enough votes (1,2)
2011-06-09 18:37:59.394: [ OCRRAW][1]proprinit: Could not open raw device
2011-06-09 18:37:59.394: [ default][1]a_init:7!: Backend init unsuccessful : [26]
2011-06-09 18:37:59.409: [ OCRRAW][1]propriogid:1: INVALID FORMAT
2011-06-09 18:37:59.409: [ OCRRAW][1]proprioini: disk 0 (/dev/rdsk/emcpower5e) doesn't have enough votes (1,2)
2011-06-09 18:37:59.410: [ OCRRAW][1]proprinit: Could not open raw device
2011-06-09 18:37:59.410: [ OCRAPI][1]a_init:6b!: Backend init unsuccessful : [26]
2011-06-09 18:37:59.410: [ OCRCONF][1]Failed to initialized OCR context. error:[PROC-26: Error while accessing the physical storage]
2011-06-09 18:37:59.411: [ OCRCONF][1]Exiting [status=failed]...
The permissions for OCR disks on both the nodes were "root:dba" and the disks was accessible across the nodes.
I'm not sure why we got that error.
To over come it, I'm going as below:
- rootdelete.sh on both nodes
- rootdeinstall on node 1
- uninstall using OUI
- remove /var/opt/oracle/*
- remove oracle_home
- remove oraInventory
- format the disks
- reboot the nodes.
Just wanted to know if there's something else that should be done.
thanks

I strongly advise against uninstalling Clusterware without first making a serious attempt to troubleshoot what went wrong and trying to fix the errors. If you unistall and retry without knowing what to do differently, you will have hardly any choice but to repeat the same steps that led to the errors in root.sh
>
The permissions for OCR disks on both the nodes were "root:dba" and the disks was accessible across the nodes.
>
That could be your error right there. The installation instructions say to put the voting disk in the dba group, but to put the cluster registry in the oinstall group. Quoting from page 4-7 of the Clusterware and RAC installation guide:
http://download.oracle.com/docs/cd/B19306_01/install.102/b14205.pdf
>
– The voting disk is a partition that Oracle Clusterware uses to verify cluster
node membership and status.
The voting disk must be owned by the oracle user, must be in the dba group,
and must have permissions set to 644. Provide at least 20 MB disk space for
the voting disk.
– The Oracle Cluster Registry (OCR) contains cluster and database configuration
information for the RAC database and for Oracle Clusterware, including the
node list, and other information about cluster configuration and profiles.
The OCR disk must be owned by root, must be in the oinstall group, and
must have permissions set to 640. Provide at least 100 MB disk space for the
OCR.

Oracle RAC on Solarais Configuration Issue

Hi,
We are trying to install Oracle RAC 10g R2 on Solaris 10.
Following are the products
1=> Soalris 10 OS
2=> Sun Cluster 3.1
3=> Veritas Volume Manager
4=> Veritas Cluster File System
Can i deploy Oracle RAC using the above listed Software.
Here we have not purchase any of the Cluster Volume Manager.
Is the CVM required to install Oracle RAC.
Is there any alternative wherein we can install Oracle RAC without using CVM like for example using the RAW device where the VM will manage storage from a single node rather than buying the CVM that will allow to manage the storage from multiple nodes.
Also i would like to highlight is suggestion for using ASM is rule out hence ASM would not be used.
Can anyone suggest me some solution to the above problem.

Well, my impression is that you don't really know what your requirements are and you are trying to fit the technologies somehow. Worse yet, there is no clear understanding how those technologies fit together and as long as you explain what your vendor is saying you (which vendor?) they don't have clear idea either.
I would also suggest to review your decision to use Sun Network Data Replicator for DR site. Consider Oracle Physical Standby database instead - it's more flexible solution and doesn't limit your choice of storage stack.
1) Do Oracle RAC 10gR2 requires any Veritas Cluster Volume Manager
Words "any" and "Veritas" contradict in your question. But the answer is no. RAC require shared storage which can generally be one of those:
- raw devices with or without some kind of cluster volume manager
- cluster file system
- NAS storage (NFS mounted)
- ASM with raw devices for CRS files (OCR and voting disks)
2) Does Oracle RAC 10g R2 require any Vertias Cluster File System or it can sit on normal Vertias File System
Again, Oracle database files MUST reside on shared storage and non-clustered file system is not an option.
3) Is there any solution where we can use the Sun Cluster with Veritas Componets to configure RAC.
If I recall correctly, Sun Cluster includes in it's license some of component of Veritas storage stack. So you might be all set. You should turn to documentation on that stage and see exactly what you have licensed and if your stack allows shared storage. You might want to have a look at http://www.sun.com/software/whitepapers/solaris10/solaris_cluster.pdf
and Oracle Certification Matrix on Metalink.

RAC general

1. Using shared SPfile Vs separate spfile for each instance. What is the best practice and any known issues with single shared spfile usage?
2. CRS restarts the database after system reboot automatically. I want to control and not have CRS restart the database. What file to be looked at and does it have to be changed on all the nodes?
Thanks for your help.

1) Best practice is to maintain shared SPFILE to manage all the instances in the cluster. DBCA automatically does that when you configure the database in RAC mode.
Not aware of any issues due to this.
2)
Refer ML note .294526.1. The same applies to your case as well. You can modify auto_start parameter with the help of this.
Setting AUTO_START=0 will disable automatic starting of the resource.
Suggest you to log an SR with Oracle support if you need assistance on this.
Thx,
Jomon

How can I find the ASM port

Hi,
When I use OEM grid control to monitor the ASM, during the configuration, I was asked to enter port, SID, please tell me from which file I can find the ASM instance port information?
Thanks,
Hank
USA 2007

hank, we have dozens of listeners on different ports and i'm not sure what it does for us. this is a legacy configuration going back to the days when our connect traffic overwhelmed the ability of the unix system to respond to one (or two or three ...) port(s). today's processors and memory are much faster and software is more efficient but there is still a point where connect traffic will overburden the port and then you need another listener. but all those listeners create a maintenance burden for me (and you) and can create a lot of confusion. bringing down one instance should never affect the availability of other instances. i think chandra meant bringing down the listener rather than instance. in 10gr2 i never turn down any listeners at any time for any maintenance. crs restarts all the listeners on any node after an unplanned outage and i never touch them. you can control connections to instances using ha services rather than listener availability. hth, mike

Questions of RAC setup

Questions are:
1. How many voting disks do we need per cluster (considering we have a redundant Fibre-channel connections)?
2. How many OCR disks do we need and what is the recommended access (OCFS2 or RAW)?
3. How do we calculate the bandwidth needed for the cluster interconnect? We currently have configured 2 x 1Gbit Ethernet ports for this.
The installation is on "Oracle Enterprise Linux 5.2 x86_64" and Oracle RAC 10GR2 on IBM x3850 M2 Servers and IBM N-Series storage connected via redundant Fibre-channel switches. Using ASM for disk handling.

I believe there are two issues here.
The first is that raw devices will not be supported in future Linux kernels. Raw device is a logical (character device) layer for the actual physical (block) device. In Linux for example, this will be +/dev/raw/raw1+ and it will be the logical raw device for physical device +/dev/sdf+.
With 10.2 CRS, you no longer need to use a raw device - you can run CRS's OCR and voting disks directly on the physical device (as the kernel now support the O_DIRECT flag for I/O on physical devices too and a raw device interface is not needed). (see Metalink note 401132.1)
Thus raw devices (as in the logical device required to be created on a physical device) are already on the way out.
The second issue is that we need a means of allocating shared storage for the OCR and voting devices at CRS startup time. Which means we have two basic choices. We either allocate a device directly (via raw for example), or we install some 3rd party (non-CRS) software that creates a sharable device for us - such as gfs, ocfs, and so on.
The problem with additional software - it increases the number of moving parts for CRS and is another software layer that can fail. Which leaves something like using raw devices a better option (less complex).
But this choice is forced because there is no "+native method+" we can use. This has changed with 11.2 and later, as ASM now can provide that functionality for us.
However, if you are still on pre 11.2 CRS and RAC/ASM, what are your choices then? Using a cluster file system or raw is pretty much the only choices available. And in that respect, raw is the one that will cause less problems and issues and is the preferred option (from personal experience).

CRS restarting - OCR inaccessible 10.2.0.4

Similar Messages

Maybe you are looking for