CRS10g patchset을 적용한 이후 Veritas SF Oracle RAC 관련 MODULE

Question

Problem Description
다음은 SF(Storage Foundation) Oracle RAC를 구성하기 위하여 Oracle CRS
PATCHSET 등을 적용하기 전에 SF Oracle RAC Veritas libraries 를 ORACLE_HOME에 설치해야 한다는 내용입니다.
또한, ORACLE CRS 패치 Version 을 올리면서 Veritas 쪽 관련하여 필요한
Veritas skgxp module 등이 설치 과정에 빠져 있거나 Veritas library file들이 overwrite되어 에러를 만나는 경우를 볼 수 있습니다.
이런 관점에서 Veritas storage foundation 을 설치하고 oracle CRS stack을
설치 또는 갱신하는 과정에서 만날 수 있는 에러에 대해 원인 및 해결방안을 알아봅니다.
에러 증상
다음은 CRS 10.2.0.3.0 Patchset이 설치되면서 Veritas library가 overwrite 되면서 발생하는 에러임.
Veritas clusterware 5.0 not recognized by Oracle due to the fact that Veritas libraries over written with crs 10.2.0.3.0 patchset installation.
The cssd.log shows:
[ CSSD]2007-11-08 03:28:02.603 [5] >TRACE: clssnm_skgxnmon: skgxn init failed,
rc 1 [ CSSD]2007-11-08 03:28:02.603 [1] >TRACE: clssnm_skgxnonline: Using vacuous skgxn monitor [ CSSD]2007-11-08 03:28:02.604 [1] >TRACE:
clssnmNMInitialize: misscount set to (30), impending reconfig threshold set to
(26)
The cssd.log should show this
[CSSD]2007-09-20 14:14:06.008 [5] >TRACE: clssnm_skgxninit: initialized
skgxn version (2/0/Veritas Cluster Server MM <<== USING VERITAS SKGXN LIBRARY
Changes
CRS patchset 의 installation 이후에 Veritas library file들이 overwrite되는 문제 발생.
Explanation
Environment :
Oracle Server - Enterprise Edition - Version: 10.2.0.3.0
OS : Solaris Operating System (SPARC 64-bit)
이 예제는 sun os에 근거함.
OS에 따라 해결책이 약간 다를 수는 있습니다.
원인
새로운 CRS patch 관련 변경이 있은 이후에 Veritas SFRAC 관련 module이 인식되지 못하는 문제 야기됨.
Installing and Configuring SF Oracle RAC Software
Veritas 에서 제공하는 Storage/Clusterware 관련한 절차는 이 문서에서 생략합니다.
Install CD 안의 installsfrac 스크립트 수행 session 에서 RAC software installation을 하는 과정이 들어갑니다.
Veritas가 제공하는 SF Oracle RAC 5.0 installation 문서를 보면 다음과 같은 내용이 포함되어 있습니다.
아래 내용에는 SF Oracle RAC component 들을 설치하는 과정만 기술합니다.
# cd /cdrom/storage_foundation_for_oracle_rac
# .installsfrac -configure
Note: Do not run root.sh yet, but return to the installsfrac session from step 15.
19 In the installsfrac session, press Return. The installsfrac utility now verifies
the database software installation, copies the SF Oracle RAC libraries to $ORACLE_HOME, and relinks Oracle on each node in the cluster.
다음은 위와 같은 installsfrac 스크립트를 수행한 이후에 Installer가 5.0 SF Oracle RAC library file들을 ORACLE_HOME으로 Copy하는 과정이고, Oracle library file들을 relink합니다.
자세한 절차는 다음의 site에서 step 별로 확인을 할 수 있습니다.
http://ftp.support.veritas.com/pub/support/products/DBE_Advanced_Cluster_for_Oracle_RAC/288502.pdf
[ 참조 ]
Performing Post-upgrade tasks for SF Oracle RAC 5.0 MP1.
To Relink Oracle 10g R1 or R2 using the installer.
installsfrac를 invoke하기 위해 다음과 같이 stage가 있다고 가정하고 invoke를 합니다.
# cd /opt/VRTS/install
# ./installsfrac -configure
Oracle environment information verification 과정입니다.
Oracle Unix User : oracle
Oracle Unix Group : oinstall
Oracle Clusterware (CRS) Home: /app/oracle/orahome
Oracle Release: 10.2
Oracle Patch Level: 0.1
Oracle Base: /app/oracle
Oracle Home: /app/oracle/orahome
Is this information correct: [y,n,q] (y)
Verifying binaries in /app/oracle/orahome on galaxy ...ok
Verifying binaries in /app/oracle/orahome on nebula ...ok
Copying SFRAC libskgxn on galaxy ......................ok
Copying SFRAC libskgxn on nebula ......................ok
Copying SFRAC ODM library on galaxy ...................ok
Copying SFRAC ODM library on nebula ...................ok
Copying SFRAC libskgxp on galaxy ......................ok
Copying SFRAC libskgxp on nebula ......................ok
Relinking Oracle on galaxy ............................ok
Relinking Oracle on nebula ............................ok
Oracle Relinking is now complete.
Solution Description
Article
http://seer.entsupport.symantec.com/docs/288502.htm
다음은 CRS patchset 적용 후, 문제 해결 위해 Oracle 10g 를 manual 하게 relink하는 과정입니다.
방법1. Relinking Oracle 10g (Using the command line)
[ 10g R1 ]
For Oracle 10gR1, enter one set of the following commands
a. For 32bit oracle:
# cp /opt/VRTSvcs/rac/lib/libskgxn2_32.so
/opt/ORCLcluster/rac/lib/libskgxn2.so
$ cp /opt/VRTSvcs/rac/lib/libskgxp10_ver23_32.so
$ORACLE_HOME/lib32/libskgxp.so
$ ln -s /usr/lib/libodm.so libodm10.so
b. For 64bit oracle:
# cp /opt/VRTSvcs/rac/lib/libskgxn2_64.so
/opt/ORCLcluster/rac/lib/libskgxn2.so
$ cp /opt/VRTSvcs/rac/lib/libskgxp10_ver23_64.so
$ORACLE_HOME/lib32/libskgxp.so
$ ln -s /usr/lib/amd64/libodm.so libodm10.so
[ 10g R2 ]
For 10gR2, enter one set of the following commands:
a. For 32bit oracle:
# cp /opt/VRTSvcs/rac/lib/libskgxn2_32.so
/opt/ORCLcluster/rac/lib/libskgxn2.so
$ cp /opt/VRTSvcs/rac/lib/libskgxp10_ver25_32.so
$ORACLE_HOME/lib32/libskgxp.so
b. For 64bit oracle:
# cp /opt/VRTSvcs/rac/lib/libskgxn2_64.so
/opt/ORCLcluster/rac/lib/libskgxn2.so
$ cp /opt/VRTSvcs/rac/lib/libskgxp10_ver25_64.so
$ORACLE_HOME/lib32/libskgxp.so
방법2. Relinking Oracle10g (Installer)
위 Article에서 Relinking Oracle 10g after upgrading SF Oracle RAC 부분을 참조하여 해결하는 것도 가능합니다.
여기서 2개의 node name이 galaxy 와 nebula 라고 가정합니다.
1. Invoke installsfrac once again:
# cd /opt/VRTS/install
#./installsfrac -configure
2. Enter the system names when prompted:
Enter the system names separated by spaces on which to configure
SFRAC: galaxy nebula
3. Navigate to the "Install and Relink Oracle" menu.
a. Select the appropriate Oracle 10g version (3):
1) Oracle 10gR1
2) Oracle 10gR2
b. Select "Relink Oracle" (3) from the menu:
1) Install Oracle Clusterware (CRS)
2) Install Oracle RDBMS server
3) Relink Oracle
b) [Go to previous menu]
c. From the menu displayed, enter the required information. For example:
Enter Oracle UNIX user name: (oracle) oracle
Enter Oracle UNIX group name: [b] (oinstall) oinstall
Enter Oracle base directory: [b] /app/oracle
Enter absolute path of CRS Home directory: [b] /app/crshome
Enter absolute path of Database Home directory: [b] /app/oracle/orahome
Enter Oracle Bits (64/32) [b] (64) 64
d. Confirm your responses in the verification screen. The installer copies the SF 5.0 Oracle RAC libraries to /opt/ORCLcluster, where it expects libskgxn.
Oracle environment information verification
Oracle Unix User: oracle
Oracle Unix Group: oinstall
Oracle Clusterware (CRS) Home: /app/crshome
Oracle Release: 10.2
Oracle Bits: 64
Oracle Base: /app/oracle
Oracle Home: /app/oracle/orahome
Is this information correct? [y,n,q] (y)
galaxy
Copying /opt/VRTSvcs/rac/lib/libskgxn2_64.so
/opt/ORCLcluster/lib/libskgxn2.so ........... success
nebula
Copying /opt/VRTSvcs/rac/lib/libskgxn2_64.so
/opt/ORCLcluster/lib/libskgxn2.so .............. success
galaxy
Copying /opt/VRTSvcs/rac/lib/libskgxp10_ver25_64.so to
/app/oracle/orahome/lib/libskgxp10.so ........... success
Removing /oracle/10g/lib/libodm10.so ............ success
Linking /opt/VRTSodm/lib/amd64/libodm.so /app/oracle/orahome/
lib/libodm10.so ... success
Setting permissions oracle:oinstall /app/oracle/orahome/lib/
libskgxp10.so ... success
nebula
Copying /opt/VRTSvcs/rac/lib/libskgxp10_ver25_64.so to
/app/oracle/orahome/lib/libskgxp10.so ........... success
Removing /oracle/10g/lib/libodm10.so ............ success
Linking /opt/VRTSodm/lib/amd64/libodm.so /app/oracle/orahome/
lib/libodm10.so ... success
Setting permissions oracle:oinstall /app/oracle/orahome/lib/
libskgxp10.so ... success
e. Enter "q" at the next prompt to leave the installer now that CRS setup
tasks are complete.
4. Bring the CSSD resource online. Enter:
# hares -online cssd -sys galaxy
# hares -online cssd -sys nebula
5. Confirm that CRS in online. Enter:
$CRS_HOME/bin/crs_stat -t
6. Bring online the oracle resources configured under VCS. If they're directly controlled by CRS, you may run the CRS commands to start the instance.
[ 참고 ]
CRS patch 적용 이후에 HP server와 Veritas 환경에서 CRS stack 이 올라오지
않는 경우를 추가 설명해 봅니다.
참고로, Veritas 쪽에서 Node status를 check해 주는 tool을 제공하고 있고,
CRS의 init.cssd 에서 그 tool을 사용하도록 init.cssd 에 patch가 되어야 한
다는 내용의 문서가 있습니다.
그 작업을 위한 patch를 역시 Veritas에서 제공하고 있습니다.
CRS patchset 이나 CRS 를 위한 cumulative patch를 하기 전에 SFRAC 를 위해
init.cssd에 patch를 적용해야 합니다.
Symptoms
1. postrootpatch.sh hangs when apply a patch[set]
2. prerootpatch.sh hangs when rollback a patch[set]
자세한 사항은 다음의 문서에서 안내하고 있습니다.
http://seer.entsupport.symantec.com/docs/281875.htm
Late Breaking News (LBN) - Updates to the Release Notes for Veritas Storage
Foundation (tm) and High Availability Solutions 5.0 and 5.0 Maintenance
Pack 1
on HP-UX 11iv2 and cross references to product documentation
환경
Veritas on HP server
이 patch 는 HP-UX 11i 에만 해당함.
예방책
Before you run the root.sh script, you need to add the init.cssd.patch.
a. Open another window on the system where you are running the installer
b. Log in as superuser
c. Change to the directory where the patch is to be copied:
For Oracle 10gR1:
# cd $CRS_HOME/css/admin
# cp /opt/VRTSvcs/rac/patch /init.cssd-10gR1.patch .
For Oracle 10gR2:
# cd $CRS_HOME/css/admin
# cp /opt/VRTSvcs/rac/patch /init.cssd-10gR2.patch .
d. Run the following command to install the patch:
For Oracle 10gR1:
# patch < init.cssd-10gR1.patch init.cssd
For Oracle 10gR2:
# patch < init.cssd-10gR2.patch init.cssd
e. Run the root.sh script. For example:
# cd $CRS_HOME
# ./root.sh
This starts the CRS daemons on the node where you enter the command.
References
http://ftp.support.veritas.com/pub/support/products/DBE_Advanced_Cluster_for_Oracle_RAC/288502.pdf
http://ftp.support.veritas.com/pub/support/products/DBE_Advanced_Cluster_for_Oracle_RAC/283979.pdf
http://seer.entsupport.symantec.com/docs/281875.htm
<Note:467753.1> Title : Veritas clusterware 5.0 not recognized by Oracle
due to the fact that Veritas libraries over written with crs 10.2.0.3 patchset installation

Answer 1