12c Grid Infrastructure 2 node instance redhat linux 6.3

I have a two node 12c rac database.  I am frequently seeing one node get shutdown for no apparent reason.
The alert log says
Error: Controlfile sequence number in file header is different from the one in memory
Please check that the correct mount options are used if controlfile is located on NFS
One node is up and the other node is down.  If I startup the node that is down, then the other node will come down leaving only one instance.

Are your controlfiles located on NFS? If so I'd check the mount options on the partition they are located to make sure they are correct.
Otherwise, I'm assuming you are also getting the ORA-235 error in the alert log? Your controlfiles may be corrupted and it would be worth looking at the article: Production Database Is Not Coming Up due to ORA-235 (Doc ID 1266376.1)
Are you using EMC Data Mover? If so this may apply to your case: Database corruption due to lost write while using EMC DATAMOVER (Doc ID 1905184.1)
Regards,
Oscar

Similar Messages

  • Too many LISTENER processes (11gR2 Grid Infrastructure (2 nodes))

    Greetings -
    I get an alert that says that my LISTENER is down (EM Alert Details). It suggests a corrective action
    to stop the current running LISTENER and start it again using the listener parameter (listener.ora) from
    the database (RDBMS) home. There are two (2) LISTENER processes running (ps -ef | grep LISTENER) .
    One process is running from the grid home and the other from the database home however only
    the one running from the grid home has a listener parameter file (listener.ora) there are no others.
    To confuse me even further the documentation says that I should not set the 'local_listener' parameter
    and that the Oracle Agent will maintain this setting.
    Any suggestions ?
    Thanks in advance.
    Brian

    Thanks Rodrigo,
    That was VERY helpful.
    I disabled the old listener and then
    I was able to create a new listener with the correct settings.
    I can see the instances on each node using this
    expression.
    lsnrctl status NEW_LISTENER
    However this shows on the first node that +ASM1, DM3, DM31
    are available on the second node I see only +ASM2 but nothing else.
    I was expecting DM3, DM32 .
    Now when I first created the new listener I read that the 'local_listener'
    setting need NOT be set. When I attemptied to unset that parameter then
    I lost the local instances. I reset the local_listener as before
    and the local instance on node one returned. Not on node two however.
    My question: Do I need to set the 'local_listener' on each instance to the same
    value or to different values to reflect which node they are configured for ?
    I cannot see it now but I remember there is a section that references the
    first node 'node_one-vip' .
    Should the second node's local_listener section be 'node_two-vip' ?
    Now if it need not be set this then something else has to occur to make it work.
    Sorry for not being more brief,
    Brian

  • How to install oracle 11gR2 Grid Infrastructure for standalond server

    Hi Experts,
    we do not want to install ASM. under this condition, does any expert install oracle 11gR2 Grid Infrastructure for standalone server in Linux 5 at file system?
    Please share tip and experiences of installed oracle 11gR2 as single instance include install oracle 11gR2 Grid Infrastructure for standalone server in Linux 5. no one node RAC configuration.
    If we do not install 11gR2 Grid Infrastructure for standalone server . which function will be lost? restat server only?
    Thanks
    Jim

    Grid Infrastructure automatically uses Oracle Restart as default if you're installing Standalone. You can turn this off using crsctl ('disable has', I believe).
    According to the inference in the install docs (http://download.oracle.com/docs/cd/E11882_01/install.112/e16763/oraclerestart.htm#CIHCBGCC), you can install 11gR2 without Grid Infrastructure, but you lose ASM and Restart.

  • 11.2 Standalone Grid Infrastructure installation to Clustered Grid Insall

    Quick question:
    Currently we have a 11.2.0.1 standalone grid infrastructure installation with ASM instance running on server a1. And there are few databases running under the +ASM instance on a1.  We are planning to make a1, a2 as a 2-node real application cluster.   Is there a way to convert the standalone grid infrastructure installation on server a1 to a clustered grid infrastructure installation along with a2 with out de-installing the existing standalone grid infrastructure on a1 and re-installing freshly?
    Thanks

    Hi Boochi,
    Thanks for responding. May be I am not clear enough in my post. Database conversion will be the next step, my question is about the actual grid infrastructure installation...Currently a1 server has been configured to work as standalone grid infrastructure, but our requirement is to convert to a 2-node RAC with server a2. Generally, I would go by de-installing the existing standalone grid infrastructure running on a1 and re-install the clustered-grid infrastructure with nodes a1 and a2. I am wondering if there is a way without de-installing the existing standalone grid infrastructure??

  • Configuring NTP for Redhat Linux on Vmware

    Hi All,
    I've configured two nodes of Redhat linux on VMWARE.How i can configure NTP server to synchorize time on both nodes.Any help will be highly appreciated.

    run a small cron job every 6 hours to run ntpdate off the domain controller as root
    the windows domain controller is usually running a timesync daemon and it should be set to sync off a few national time sites.

  • Install Grid Infrastructure

    Try to install Grid Infrastructure (ASM) 11gR2 on SUSE Linux 11. Installation went through well untill the very last steps. The message indicated:
    Configure Oracle Grid Infrastructure for Standalone Server   -- failed
    *Automatic Storage Management Configuration Assistant failed.   --failed.*
    What could be the solution?

    The log does state: Successfully configured Oracle Grid Infrastructure for a Standalone Server
    Then it moves on to configure 'Automatic Storage Management Configuration Assistant'
    There it says:
    Starting 'Automatic Storage Management Configuration Assistant'
    Error while detecting Oracle Grid Infrastructure. ASMCA needs Oracle Grid Infrastructure to configure ASM.
    It sounds like that Oracle Grid Infrastructure for a Standalone Server was indeed successfully configured. However, for some reason the ASMCA could not find it.
    Next, I did the following:
    . oraenv
    ORACLE_SID = [oracle] ? +ASM
    ORACLE_HOME ...
    asmca
    The ASMCA GUI windows opened.
    After I enter the required info, then click on Create ASM button, this is what it displyed:
    Oracle Grid Infrastructure is not configured properly. ASMCA needs Oracle Grid Infrastructure to configure ASM.
    What could be wrong?

  • Install Grid Infrastructure 11GR2

    Hi all,
    I have just installed Oracle Grid Infrastructure 11GR2 without having all linux packages installed like UnixODBC and 2 others.
    Do you thing is it a problem ?
    Thanks.

    No, I don't have error message. Just one at the end when Oracle check installation.
    When i launch runcluvfy i have this problem :
    [grid@rac1 grid]$ ./runcluvfy.sh stage -post crsinst -n rac1,rac2 -verbose
    Exécution de post-vérifications pour configuration des services de cluster
    Vérification de l'accessibilité du noeud...
    Vérification : accessibilité du noeud "rac1"
    Noeud de destination Accessible ?
    rac2 oui
    rac1 oui
    Résultat : Succès de la vérification de l'accessibilité du noeud "rac1"
    Vérification de l'équivalence utilisateur...
    Vérification : équivalence de l'utilisateur "grid"
    Nom de noeud Commentaire
    rac2 réussi
    rac1 réussi
    Résultat : Succès de la vérification de l'équivalence de l'utilisateur "grid"
    Vérification de la cohérence des fuseaux horaires...
    Succès de la vérification de la cohérence des fuseaux horaires.
    Vérification de l'intégrité du gestionnaire de clusters...
    Vérification du démon CSS...
    Nom de noeud Statut
    rac2 en cours d'exécution
    rac1 en cours d'exécution
    Oracle Cluster Synchronization Services est en ligne.
    Succès de la vérification de l'intégrité du gestionnaire de clusters
    La vérification des attributs Udev pour Emplacements d'OCR a démarré...
    Résultat : Vérification des attributs Udev réussie pour Emplacements d'OCR
    La vérification des attributs Udev pour Emplacements des disques "votants" a démarré...
    Résultat : Vérification des attributs Udev réussie pour Emplacements des disques "votants"
    Vérifier le masque de création du fichier utilisateur par défaut
    Nom de noeud Disponible Obligatoire Commentaire
    rac2 0022 0022 réussi
    rac1 0022 0022 réussi
    Résultat : Succès de la vérification du masque de création du fichier utilisateur par défaut
    Vérification de l'intégrité du cluster...
    Nom de noeud
    rac1
    rac2
    Vérification de l'intégrité du cluster réussie
    Vérification de l'intégrité d'OCR...
    Vérification de l'absence de configuration hors cluster...
    Tous les noeuds sans configuration locale hors cluster
    Succès de la vérification de l'exécution d'ASM. ASM est en cours d'exécution sur tous les noeuds de cluster
    Vérification du fichier de configuration OCR "/etc/oracle/ocr.loc"...
    Fichier de configuration OCR "/etc/oracle/ocr.loc" vérifié
    Le groupe de disques pour l'emplacement OCR "+DATA" est disponible sur tous les noeuds
    Vérification de la taille de l'emplacement OCR "+DATA"...
    Vérification de la taille pour l'emplacement OCR "+DATA" réussie...
    Vérification de la taille pour l'emplacement OCR "+DATA" réussie...
    WARNING:
    Cette vérification ne prend pas en compte l'intégrité du contenu d'OCR. Exécutez 'ocrcheck' en tant qu'utilisateur disposant de privilèges pour vérifier le contenu d'OCR.
    Succès de la vérification de l'intégrité d'OCR
    Vérification de l'intégrité de CRS...
    Le clusterware Oracle est en bon état sur le noeud "rac2"
    Le clusterware Oracle est en bon état sur le noeud "rac1"
    Succès de la vérification de l'intégrité de CRS
    Vérification de l'existence de l'application de noeud...
    Vérification de l'existence de l'application de noeud VIP
    Nom de noeud Obligatoire Statut Commentaire
    rac2 oui en ligne réussi
    rac1 oui en ligne réussi
    Résultat : Vérification réussie.
    Vérification de l'existence de l'application de noeud ONS
    Nom de noeud Obligatoire Statut Commentaire
    rac2 non en ligne réussi
    rac1 non en ligne réussi
    Résultat : Vérification réussie.
    Vérification de l'existence de l'application de noeud GSD
    Nom de noeud Obligatoire Statut Commentaire
    rac2 non n'existe pas ignoré
    rac1 non n'existe pas ignoré
    Résultat : Vérification ignorée.
    Vérification de l'existence de l'application de noeud EONS
    Nom de noeud Obligatoire Statut Commentaire
    rac2 non en ligne réussi
    rac1 non en ligne réussi
    Résultat : Vérification réussie.
    Vérification de l'existence de l'application de noeud NETWORK
    Nom de noeud Obligatoire Statut Commentaire
    rac2 non en ligne réussi
    rac1 non en ligne réussi
    Résultat : Vérification réussie.
    Vérification du nom d'accès du client unique (SCAN, Single Client Access Name)...
    Nom d'adresse IP virtuelle SCAN Noeud En cours d'exécution ? ListenerName Port En cours d'exécution ?
    rac-cluster-lpt-scan rac1 true LISTENER 1521 true
    Vérification de la configuration de la résolution de noms pour "rac-cluster-lpt-scan"...
    ERROR:
    PRVF-4664 : Entrées de résolution de noms incohérentes détectées pour le nom SCAN "rac-cluster-lpt-scan"
    Nom SCAN Adresse IP Statut Commentaire
    rac-cluster-lpt-scan 10.176.1.181 en échec Entrée NIS
    ERROR:
    PRVF-4657 : Echec de la vérification de la configuration de la résolution de noms pour "rac-cluster-lpt-scan" (adresse IP : 10.176.1.181)
    ERROR:
    PRVF-4664 : Entrées de résolution de noms incohérentes détectées pour le nom SCAN "rac-cluster-lpt-scan"
    Echec de la vérification de la configuration de l'adresse IP virtuelle SCAN et du processus d'écoute
    OCR détecté sur ASM. Exécution de vérifications de l'intégrité d'ACFS...
    Démarrage de la vérification pour contrôler si ASM est en cours d'exécution sur tous les noeuds de cluster...
    Succès de la vérification de l'exécution d'ASM. ASM est en cours d'exécution sur tous les noeuds de cluster
    Démarrage de la vérification des groupes de disques pour contrôler si au moins un groupe de disques est configuré...
    Succès de la vérification des groupes de disques. Au moins un groupe de disques a été configuré
    Succès de la vérification de l'intégrité d'ACFS de tâche
    Vérification de la configuration de disque "votant" de cluster Oracle...
    Succès de la vérification de la configuration de disque "votant" de cluster Oracle
    Vérification de la non-appartenance de l'utilisateur "grid" au groupe "root"
    Nom de noeud Statut Commentaire
    rac2 n'existe pas réussi
    rac1 n'existe pas réussi
    Résultat : L'utilisateur "grid" ne fait pas partie du groupe "root". Succès de la vérification
    Vérification de l'installation du clusterware sur tous les noeuds...
    Succès de la vérification de l'installation de clusterware
    Vérification de l'exécution de la ressource CTSS sur tous les noeuds...
    Vérification : ressource CTSS exécutée sur tous les noeuds
    Nom de noeud Statut
    rac2 réussi
    rac1 réussi
    Résultat : Succès de la vérification de la ressource CTSS
    Interrogation de CTSS pour le décalage horaire sur tous les noeuds...
    Résultat : Succès de l'interrogation de CTSS pour le décalage horaire
    Vérification de l'état de CTSS démarrée...
    Vérification : état de CTSS
    Nom de noeud Etat
    rac2 Observateur
    rac1 Observateur
    L'état de CTSS est Observateur. Permutation vers des vérifications de la synchronisation d'horloge à l'aide du protocole NTP
    Démarrage des vérifications de la synchronisation d'horloge à l'aide du protocole NTP (Network Time Protocol)...
    La vérification du fichier de configuration NTP a démarré...
    Le fichier de configuration NTP "/etc/ntp.conf" est disponible sur tous les noeuds
    Succès de la vérification du fichier de configuration NTP
    Vérification de l'activité du démon...
    Vérification : activité de "ntpd"
    Nom de noeud En cours d'exécution ?
    rac2 oui
    rac1 oui
    Résultat : Succès de la vérification de l'activité pour "ntpd"
    Recherche de l'option de rotation "-x" dans la ligne de commande du démon NTP
    Vérification : ligne de commande du démon NTP
    Nom de noeud Option de rotation définie ?
    rac2 oui
    rac1 oui
    Résultat :
    Option de rotation du démon NTP vérifiée
    Recherche de l'option de rotation "-x" dans la configuration du temps d'amorçage du démon NTP, dans le fichier "/etc/sysconfig/ntpd"
    Vérification : configuration du temps d'amorçage du démon NTP
    Nom de noeud Option de rotation définie ?
    rac2 oui
    rac1 oui
    Résultat :
    Vérification de l'option de rotation dans la configuration du temps d'amorçage du démon NTP exécutée
    La vérification du serveur d'horloge NTP commun a démarré...
    Le serveur d'horloge NTP "195.83.132.135" est commun à tous les noeuds sur lesquels il est en cours d'exécution
    Le serveur d'horloge NTP ".LOCL." est commun à tous les noeuds sur lesquels il est en cours d'exécution
    Succès de la vérification du serveur d'horloge NTP commun
    La vérification du décalage de l'horloge par rapport au serveur d'horloge NTP a démarré...
    Vérification sur les noeuds "[rac2, rac1]"...
    Vérification : décalage de l'horloge par rapport au serveur d'horloge NTP
    Serveur d'horloge : 195.83.132.135
    Limite de décalage horaire : 1000.0 ms
    Nom de noeud Décalage horaire Statut
    rac2 -2.734 réussi
    rac1 -1.906 réussi
    Les décalages horaires du serveur d'horloge "195.83.132.135" se trouvent dans les limites autorisées pour les noeuds "[rac2, rac1]".
    Serveur d'horloge : .LOCL.
    Limite de décalage horaire : 1000.0 ms
    Nom de noeud Décalage horaire Statut
    rac2 0.0 réussi
    rac1 0.0 réussi
    Les décalages horaires du serveur d'horloge ".LOCL." se trouvent dans les limites autorisées pour les noeuds "[rac2, rac1]".
    Succès de la vérification du décalage de l'horloge
    Résultat : Vérification de la synchronisation d'horloge à l'aide du protocole NTP (Network Time Protocol) réussie
    Succès de la vérification d'Oracle Cluster Time Synchronization Services
    Echec de la post-vérification de configuration des services de cluster sur tous les noeuds.
    [grid@rac1 grid]$

  • Install 11gR2 grid infrastructure

    Hi,
    I installing grid infrastructure on SINGLE INSTANCE.
    I would like to know what is the difference between the following option in the runInstaller :
    1 ) Install and configured grid infrastructure for standalone server
    2)  Install grid infrastructure sogtware onlyThanks

    Hi,
    Option 1 will install the Grid Infrastruture and also configure the ASM disks.
    Option 2 will install the Grid Infrastructure binaries only. You have the configure the ASM on your own.
    Regards

  • 11g R2 Grid Infrastructure asmca cannot detect 11g R1 ASM instance

    I am playing with 11g ASM with XEN (one node on host (or dom0) and another node on vm (or dom1)) and was trying to do a Clusterware + ASM upgrade from 11.0.6 to 11.2.0. After solving some stupid problem, (see [rootupgrade.sh failed on 11g R1 to R2 Upgrade |http://forums.oracle.com/forums/thread.jspa?threadID=1005424&tstart=0]) so that all services can be started in both nodes, I am trying to upgrade ASM.
    I am trying asmca. But it is unable to detect the 11g R1 ASM instance. When started in GUI mode, it just offer to create ASM. When started in silent mode for upgrade, it shows:
    oracle@penguin-dom0:~> asmca -silent -upgradeASM
    Enter ASMSNMP Password
    ORA-01034: ORACLE not availableAnd I can't find anything in ASM alert log
    ASM is running from 11g R1 home
    oracle@penguin-dom0:~> ps -ef|grep asm
    oracle   21811     1  0 Jan06 ?        00:00:00 asm_pmon_+ASM1
    oracle   21813     1  0 Jan06 ?        00:00:03 asm_vktm_+ASM1
    oracle   21817     1  0 Jan06 ?        00:00:02 asm_diag_+ASM1
    oracle   21819     1  0 Jan06 ?        00:00:00 asm_ping_+ASM1
    oracle   21821     1  0 Jan06 ?        00:00:00 asm_psp0_+ASM1
    oracle   21825     1  0 Jan06 ?        00:00:09 asm_dia0_+ASM1
    oracle   21827     1  0 Jan06 ?        00:00:05 asm_lmon_+ASM1
    oracle   21829     1  0 Jan06 ?        00:00:03 asm_lmd0_+ASM1
    oracle   21831     1  0 Jan06 ?        00:00:03 asm_lms0_+ASM1
    oracle   21835     1  0 Jan06 ?        00:00:00 asm_mman_+ASM1
    oracle   21837     1  0 Jan06 ?        00:00:00 asm_dbw0_+ASM1
    oracle   21839     1  0 Jan06 ?        00:00:00 asm_lgwr_+ASM1
    oracle   21841     1  0 Jan06 ?        00:00:00 asm_ckpt_+ASM1
    oracle   21843     1  0 Jan06 ?        00:00:00 asm_smon_+ASM1
    oracle   21845     1  0 Jan06 ?        00:00:01 asm_rbal_+ASM1
    oracle   21847     1  0 Jan06 ?        00:00:00 asm_gmon_+ASM1
    oracle   21856     1  0 Jan06 ?        00:00:00 asm_lck0_+ASM1
    oracle   22021     1  0 Jan06 ?        00:00:00 /u01/app/oradb/product/11.1.0/db_1/bin/racgimon daemon ora.penguin-dom0.ASM1.asm11g R1 Listener is serving to ASM
    oracle@penguin-dom0:~> lsnrctl status
    LSNRCTL for Linux: Version 11.1.0.6.0 - Production on 07-JAN-2010 00:33:44
    Copyright (c) 1991, 2007, Oracle.  All rights reserved.
    Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
    STATUS of the LISTENER
    Alias                     LISTENER_PENGUIN-DOM0
    Version                   TNSLSNR for Linux: Version 11.1.0.6.0 - Production
    Start Date                06-JAN-2010 22:29:51
    Uptime                    0 days 2 hr. 3 min. 53 sec
    Trace Level               off
    Security                  ON: Local OS Authentication
    SNMP                      OFF
    Listener Parameter File   /u01/app/oradb/product/11.1.0/db_1/network/admin/listener.ora
    Listener Log File         /u01/app/oradb/diag/tnslsnr/penguin-dom0/listener_penguin-dom0/alert/log.xml
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.201)(PORT=1521)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.101)(PORT=1521)))
    Services Summary...
    Service "+ASM" has 1 instance(s).
      Instance "+ASM1", status READY, has 1 handler(s) for this service...
    Service "+ASM_XPT" has 1 instance(s).
      Instance "+ASM1", status READY, has 1 handler(s) for this service...
    The command completed successfullyNo tnsnames.ora in 11g R1 or Grid Infrastructure home.
    Setting ORACLE_HOME to 11g R1 home and ORACLE_SID=+ASM1, I can connect to ASM instance as sysasm using R1 sqlplus
    oracle@penguin-dom0:~> export ORACLE_HOME=/u01/app/oradb/product/11.1.0/db_1
    oracle@penguin-dom0:~> export ORACLE_SID=+ASM1
    oracle@penguin-dom0:~> $ORACLE_HOME/bin/sqlplus / as sysasm
    SQL*Plus: Release 11.1.0.6.0 - Production on Thu Jan 7 00:40:15 2010
    Copyright (c) 1982, 2007, Oracle.  All rights reserved.
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
    With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,
    Data Mining and Real Application Testing options
    SQL> Tried different combinations of ORACLE_HOME (11g R1 db home, grid infrastructure home, 11g R1 clusterware home) and ORACLE_SID (+ASM, +ASM1), but asmca still cannot connect to ASM instance.
    Some documents mentioned to configure 'Oracle Restart' and the ASM instance name is +ASM. Are these necessary requirements? If so, where can I find information on Oracle Restart and how to rename the ASM instance? Besides, the documents said it is recommanded to upgrade with OUI, but I've already installed the software, can I still use OUI?
    OS: OpenSUSE 11.1, XEN DOM0 and DOM1
    Oracle: 11.0.6 upgrade to 11.2.0
    Edited by: thtsang on Jan 6, 2010 8:47 AM
    Edited by: thtsang on Jan 6, 2010 5:33 PM

    Hi Thtsang,
    sorry for not coming back to you earlier and thanks for "the helpful".
    So, are you saying that after you have removed all the environment variables, you still got the error message "ORA-01034: ORACLE not available"? I thought you got an "ASM upgrade failed", because ASM was down at the time of the upgrade, which you then corrected saying: "In fact, ASM instance was started from Grid Infrastructure home.". Overall, I need to say that I am a bit confused to be honest.
    The alert log entry sounds promising, although, one would need to see more context. For: "But when I use crsctl / srvctl to start ASM, it is still running in 11g R1 Oracle Home." So, something went wrong with the update. For: "BTW, if asmca does not work for me, can I upgrade manually?" - ASMCA is the tool that will perform an ASM upgrade. You can call it interactively, but you will have to use it.
    Support note ID 950200.1 - "ASM 11gR2: How To Upgrade An ASM Instance To Release 11gR2 (11.2.0.1.0)?" describes an ASM upgrade to 11.2. Maybe this note can help you and if it is only to analyze what stage your cluster is in right now. It sounds like, this would be first thing that we need to find out.
    Good luck. Thanks,
    Markus

  • Install a grid infrastructure for a cluster for single node Windows Server

    Hello,
    Can you suggest me how to install a Oracle Single Node RAC 11g on a Windows Server 2008 x64 bits? It's for testing purposes and I don't need two nodes. I know that Linux is better, but my company only uses Windows servers. Our former consultant was able to do it, but didn't tell us how to and that server was lost.
    I know that we have to install first Grid Infrastructure. Running the "Oracle Grid Infrastructure" installer, I could successfully install a "Grid Infrastructure for a Standalone Server". But that grid is not for a RAC database. When trying to install the Database software, it allowed to me to only install "Single instance database", not "Real Application Cluster database installation".
    Running the "Oracle Grid Infrastructure" installer, I tried to select option:
    "Install and Configure Grid Infrastructure for a Cluster" -> Typical Installation:
    SCAN Name: SRVORAC-cluster
    Validating SCAN information
    INS-40922 Invalid SCAN Name - unresolvable to IP address.
    Is it possible to install a grid infrastructure for a cluster for a single-node or one-node Windows Server ?
    Edited by: user521219 on Jan 30, 2012 12:46 PM

    HI,
    You probably have misconception about Single node RAC. This feature does not need that you have only one physical machine in a RAC, it actually means that in a RAC environment (2 or more nodes), you can run your database on only one node(single instance) and then later you can move this single instance to any other node of the RAC (so that you can do some patching on this node making your database available all the time). This also has some other benefits, see bellow.
    http://www.oracle.com/technetwork/database/clustering/overview/ug-raconenode-2009-130760.pdf
    http://docs.oracle.com/cd/E11882_01/install.112/e25666/whatsnew.htm#sthref8
    Salman

  • Problem with installation of Oracle Grid Infrastructure failed on Oracle Linux 6.4 x86_64

    Hello!
    I try install Oracle Grid Infrastructure 11.2.0.1. OS - OEL 6.4 x86_64
    But running script  "/u01/app/oracle/product/11.2.0/grid/root.sh" failed:
    [root@localhost grid]# /u01/app/oracle/product/11.2.0/grid/root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= grid
        ORACLE_HOME=  /u01/app/oracle/product/11.2.0/grid
    Enter the full pathname of the local bin directory: [/usr/local/bin]:
       Copying dbhome to /usr/local/bin ...
       Copying oraenv to /usr/local/bin ...
       Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2013-07-03 01:27:08: Checking for super user privileges
    2013-07-03 01:27:08: User has super user privileges
    2013-07-03 01:27:08: Parsing the host name
    Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'grid', privgrp 'oinstall'..
    Operation successful.
    CRS-4664: Node localhost successfully pinned.
    Adding daemon to inittab
    CRS-4124: Oracle High Availability Services startup failed.
    CRS-4000: Command Start failed, or completed with errors.
    ohasd failed to start: Inappropriate ioctl for device
    ohasd failed to start: Inappropriate ioctl for device at /u01/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 296.
    /u01/app/oracle/product/11.2.0/grid/cfgtoollogs/crsconfig/roothas.log:
    2013-07-03 01:27:08: The configuration parameter file /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params is valid
    2013-07-03 01:27:08: Checking for super user privileges
    2013-07-03 01:27:08: User has super user privileges
    2013-07-03 01:27:08: ### Printing the configuration values from files:
    2013-07-03 01:27:08:    /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
    2013-07-03 01:27:08:    /u01/app/oracle/product/11.2.0/grid/crs/install/s_crsconfig_defs
    2013-07-03 01:27:08: ASM_DISCOVERY_STRING=
    2013-07-03 01:27:08: ASM_DISKS=ORCL:ASMDATA01,ORCL:ASMDATA02,ORCL:ASMDATA03,ORCL:ASMDATA04
    2013-07-03 01:27:08: ASM_DISK_GROUP=DATA
    2013-07-03 01:27:08: ASM_REDUNDANCY=NORMAL
    2013-07-03 01:27:08: ASM_SPFILE=
    2013-07-03 01:27:08: ASM_UPGRADE=false
    2013-07-03 01:27:08: CLSCFG_MISSCOUNT=
    2013-07-03 01:27:08: CLUSTER_GUID=
    2013-07-03 01:27:08: CLUSTER_NAME=
    2013-07-03 01:27:08: CRS_NODEVIPS=
    2013-07-03 01:27:08: CRS_STORAGE_OPTION=0
    2013-07-03 01:27:08: CSS_LEASEDURATION=400
    2013-07-03 01:27:08: DIRPREFIX=
    2013-07-03 01:27:08: DISABLE_OPROCD=0
    2013-07-03 01:27:08: EMBASEJAR_NAME=oemlt.jar
    2013-07-03 01:27:08: EWTJAR_NAME=ewt3.jar
    2013-07-03 01:27:08: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
    2013-07-03 01:27:08: GNS_ADDR_LIST=
    2013-07-03 01:27:08: GNS_ALLOW_NET_LIST=
    2013-07-03 01:27:08: GNS_CONF=false
    2013-07-03 01:27:08: GNS_DENY_ITF_LIST=
    2013-07-03 01:27:08: GNS_DENY_NET_LIST=
    2013-07-03 01:27:08: GNS_DOMAIN_LIST=
    2013-07-03 01:27:08: GPNPCONFIGDIR=/u01/app/oracle/product/11.2.0/grid
    2013-07-03 01:27:08: GPNPGCONFIGDIR=/u01/app/oracle/product/11.2.0/grid
    2013-07-03 01:27:08: GPNP_PA=
    2013-07-03 01:27:08: HELPJAR_NAME=help4.jar
    2013-07-03 01:27:08: HOST_NAME_LIST=
    2013-07-03 01:27:08: ID=/etc/init.d
    2013-07-03 01:27:08: INIT=/sbin/init
    2013-07-03 01:27:08: IT=/etc/inittab
    2013-07-03 01:27:08: JEWTJAR_NAME=jewt4.jar
    2013-07-03 01:27:08: JLIBDIR=/u01/app/oracle/product/11.2.0/grid/jlib
    2013-07-03 01:27:08: JREDIR=/u01/app/oracle/product/11.2.0/grid/jdk/jre/
    2013-07-03 01:27:08: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
    2013-07-03 01:27:08: MSGFILE=/var/adm/messages
    2013-07-03 01:27:08: NETCFGJAR_NAME=netcfg.jar
    2013-07-03 01:27:08: NETWORKS=
    2013-07-03 01:27:08: NEW_HOST_NAME_LIST=
    2013-07-03 01:27:08: NEW_NODEVIPS=
    2013-07-03 01:27:08: NEW_NODE_NAME_LIST=
    2013-07-03 01:27:08: NEW_PRIVATE_NAME_LIST=
    2013-07-03 01:27:08: NODELIST=
    2013-07-03 01:27:08: NODE_NAME_LIST=
    2013-07-03 01:27:08: OCFS_CONFIG=
    2013-07-03 01:27:08: OCRCONFIG=/etc/oracle/ocr.loc
    2013-07-03 01:27:08: OCRCONFIGDIR=/etc/oracle
    2013-07-03 01:27:08: OCRID=
    2013-07-03 01:27:08: OCRLOC=ocr.loc
    2013-07-03 01:27:08: OCR_LOCATIONS=
    2013-07-03 01:27:08: OLASTGASPDIR=/etc/oracle/lastgasp
    2013-07-03 01:27:08: OLRCONFIG=/etc/oracle/olr.loc
    2013-07-03 01:27:08: OLRCONFIGDIR=/etc/oracle
    2013-07-03 01:27:08: OLRLOC=olr.loc
    2013-07-03 01:27:08: OPROCDCHECKDIR=/etc/oracle/oprocd/check
    2013-07-03 01:27:08: OPROCDDIR=/etc/oracle/oprocd
    2013-07-03 01:27:08: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
    2013-07-03 01:27:08: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
    2013-07-03 01:27:08: ORACLE_BASE=/u01/app/oracle
    2013-07-03 01:27:08: ORACLE_HOME=/u01/app/oracle/product/11.2.0/grid
    2013-07-03 01:27:08: ORACLE_OWNER=grid
    2013-07-03 01:27:08: ORA_ASM_GROUP=asmadmin
    2013-07-03 01:27:08: ORA_DBA_GROUP=oinstall
    2013-07-03 01:27:08: PRIVATE_NAME_LIST=
    2013-07-03 01:27:08: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
    2013-07-03 01:27:08: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
    2013-07-03 01:27:08: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
    2013-07-03 01:27:08: RC_KILL=K19
    2013-07-03 01:27:08: RC_KILL_OLD=K96
    2013-07-03 01:27:08: RC_START=S96
    2013-07-03 01:27:08: SCAN_NAME=
    2013-07-03 01:27:08: SCAN_PORT=0
    2013-07-03 01:27:08: SCRBASE=/etc/oracle/scls_scr
    2013-07-03 01:27:08: SHAREJAR_NAME=share.jar
    2013-07-03 01:27:08: SILENT=false
    2013-07-03 01:27:08: SO_EXT=so
    2013-07-03 01:27:08: SRVCFGLOC=srvConfig.loc
    2013-07-03 01:27:08: SRVCONFIG=/var/opt/oracle/srvConfig.loc
    2013-07-03 01:27:08: SRVCONFIGDIR=/var/opt/oracle
    2013-07-03 01:27:08: VNDR_CLUSTER=false
    2013-07-03 01:27:08: VOTING_DISKS=
    2013-07-03 01:27:08: ### Printing other configuration values ###
    2013-07-03 01:27:08: CLSCFG_EXTRA_PARMS=
    2013-07-03 01:27:08: CRSDelete=0
    2013-07-03 01:27:08: HAPatch=0
    2013-07-03 01:27:08: HAS_GROUP=oinstall
    2013-07-03 01:27:08: HAS_USER=grid
    2013-07-03 01:27:08: HOST=localhost
    2013-07-03 01:27:08: IS_SIHA=1
    2013-07-03 01:27:08: OLR_DIRECTORY=/u01/app/oracle/product/11.2.0/grid/cdata/localhost
    2013-07-03 01:27:08: OLR_LOCATION=/u01/app/oracle/product/11.2.0/grid/cdata/localhost/localhost.olr
    2013-07-03 01:27:08: ORA_CRS_HOME=/u01/app/oracle/product/11.2.0/grid
    2013-07-03 01:27:08: SUPERUSER=root
    2013-07-03 01:27:08: UPGRADE=
    2013-07-03 01:27:08: crscfg_trace=1
    2013-07-03 01:27:08: crscfg_trace_file=/u01/app/oracle/product/11.2.0/grid/cfgtoollogs/crsconfig/roothas.log
    2013-07-03 01:27:08: hahome=
    2013-07-03 01:27:08: hosts=
    2013-07-03 01:27:08: osdfile=/u01/app/oracle/product/11.2.0/grid/crs/install/s_crsconfig_defs
    2013-07-03 01:27:08: parameters_valid=1
    2013-07-03 01:27:08: paramfile=/u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
    2013-07-03 01:27:08: platform_family=unix
    2013-07-03 01:27:08: srvctl_trc_suff=0
    2013-07-03 01:27:08: user_is_superuser=1
    2013-07-03 01:27:08: ### Printing of configuration values complete ###
    2013-07-03 01:27:08: Oracle CRS stack is not configured yet
    2013-07-03 01:27:08: CRS is not yet configured. Hence, will proceed to configure CRS
    2013-07-03 01:27:10: OLR location = /u01/app/oracle/product/11.2.0/grid/cdata/localhost/localhost.olr
    2013-07-03 01:27:10: Oracle CRS Home = /u01/app/oracle/product/11.2.0/grid
    2013-07-03 01:27:10: Validating /etc/oracle/olr.loc file for OLR location /u01/app/oracle/product/11.2.0/grid/cdata/localhost/localhost.olr
    2013-07-03 01:27:10: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
    2013-07-03 01:27:10: Opening file OCRCONFIG
    2013-07-03 01:27:10: Opening file OCRCONFIG
    2013-07-03 01:27:10: create Local Only OCR...
    2013-07-03 01:27:10: create Local Only OCR on Linux...
    2013-07-03 01:27:10: OCR locations = /u01/app/oracle/product/11.2.0/grid/cdata/localhost/local.ocr
    2013-07-03 01:27:10: Validating OCR locations in /etc/oracle/ocr.loc
    2013-07-03 01:27:10: Checking for existence of /etc/oracle/ocr.loc
    2013-07-03 01:27:10: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
    2013-07-03 01:27:10: Setting ocr location /u01/app/oracle/product/11.2.0/grid/cdata/localhost/local.ocr
    2013-07-03 01:27:10: Creating or upgrading Oracle Local Registry (OLR)
    2013-07-03 01:27:10: Running as user grid: /u01/app/oracle/product/11.2.0/grid/bin/ocrconfig -local -upgrade
    2013-07-03 01:27:10:   Invoking "/u01/app/oracle/product/11.2.0/grid/bin/ocrconfig -local -upgrade" as user "grid"
    2013-07-03 01:27:11: OLR successfully created or upgraded
    2013-07-03 01:27:11: /u01/app/oracle/product/11.2.0/grid/bin/clscfg -localadd
    2013-07-03 01:27:11: Running as user grid: /u01/app/oracle/product/11.2.0/grid/bin/clscfg -localadd
    2013-07-03 01:27:11:   Invoking "/u01/app/oracle/product/11.2.0/grid/bin/clscfg -localadd" as user "grid"
    2013-07-03 01:27:11: Keys created in the OLR successfully
    2013-07-03 01:27:12: Creating local-only OCR (/u01/app/oracle/product/11.2.0/grid/bin/crsctl pin css -n localhost) ... succeeded
    2013-07-03 01:27:12: Registering ohasd
    2013-07-03 01:27:12: init file = /u01/app/oracle/product/11.2.0/grid/crs/init/init.ohasd
    2013-07-03 01:27:12: Copying file /u01/app/oracle/product/11.2.0/grid/crs/init/init.ohasd to /etc/init.d directory
    2013-07-03 01:27:12: Setting init.ohasd permission in /etc/init.d directory
    2013-07-03 01:27:12: init file = /u01/app/oracle/product/11.2.0/grid/crs/init/ohasd
    2013-07-03 01:27:12: Copying file /u01/app/oracle/product/11.2.0/grid/crs/init/ohasd to /etc/init.d directory
    2013-07-03 01:27:12: Setting ohasd permission in /etc/init.d directory
    2013-07-03 01:27:12: Removing "/etc/rc.d/rc3.d/S96ohasd"
    2013-07-03 01:27:12: Removing file /etc/rc.d/rc3.d/S96ohasd
    2013-07-03 01:27:12: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
    2013-07-03 01:27:12: Failed to remove file:
    2013-07-03 01:27:12: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
    2013-07-03 01:27:12: Removing "/etc/rc.d/rc5.d/S96ohasd"
    2013-07-03 01:27:12: Removing file /etc/rc.d/rc5.d/S96ohasd
    2013-07-03 01:27:12: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
    2013-07-03 01:27:12: Failed to remove file:
    2013-07-03 01:27:12: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
    2013-07-03 01:27:12: Removing "/etc/rc.d/rc0.d/K19ohasd"
    2013-07-03 01:27:12: Removing file /etc/rc.d/rc0.d/K19ohasd
    2013-07-03 01:27:12: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
    2013-07-03 01:27:12: Failed to remove file:
    2013-07-03 01:27:12: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-03 01:27:12: Removing "/etc/rc.d/rc1.d/K19ohasd"
    2013-07-03 01:27:12: Removing file /etc/rc.d/rc1.d/K19ohasd
    2013-07-03 01:27:12: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
    2013-07-03 01:27:12: Failed to remove file:
    2013-07-03 01:27:12: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-03 01:27:12: Removing "/etc/rc.d/rc2.d/K19ohasd"
    2013-07-03 01:27:12: Removing file /etc/rc.d/rc2.d/K19ohasd
    2013-07-03 01:27:12: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
    2013-07-03 01:27:12: Failed to remove file:
    2013-07-03 01:27:12: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-03 01:27:12: Removing "/etc/rc.d/rc4.d/K19ohasd"
    2013-07-03 01:27:12: Removing file /etc/rc.d/rc4.d/K19ohasd
    2013-07-03 01:27:12: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
    2013-07-03 01:27:12: Failed to remove file:
    2013-07-03 01:27:12: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-03 01:27:12: Removing "/etc/rc.d/rc6.d/K19ohasd"
    2013-07-03 01:27:12: Removing file /etc/rc.d/rc6.d/K19ohasd
    2013-07-03 01:27:12: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
    2013-07-03 01:27:12: Failed to remove file:
    2013-07-03 01:27:12: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
    2013-07-03 01:27:12: The file ohasd has been successfully linked to the RC directories
    2013-07-03 01:27:12: Starting ohasd
    2013-07-03 01:27:12: itab entries=
    2013-07-03 01:27:18: Created backup /etc/inittab.no_crs
    2013-07-03 01:27:18: Appending to /etc/inittab.tmp:
    2013-07-03 01:27:18: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
    2013-07-03 01:27:18: Done updating /etc/inittab.tmp
    2013-07-03 01:27:18: Saved /etc/inittab.crs
    2013-07-03 01:27:18: Installed new /etc/inittab
    2013-07-03 01:29:18: ohasd failed to start: Inappropriate ioctl for device
    2013-07-03 01:29:18: ###### Begin DIE Stack Trace ######
    2013-07-03 01:29:18:     Package         File                 Line Calling 
    2013-07-03 01:29:18:     --------------- -------------------- ---- ----------
    2013-07-03 01:29:18:  1: main            roothas.pl            166 crsconfig_lib::dietrap
    2013-07-03 01:29:18:  2: main            roothas.pl            296 main::__ANON__
    2013-07-03 01:29:18: ####### End DIE Stack Trace #######
    selinux and firewall is disabled.

    Hi,
    Refer the below metalink note :- OHASD Failed to Start: Inappropriate ioctl for device [ID 1069182.1]
    Thanks

  • Migrating Standalone Grid Infrastructure Servers to a Cluster 11.2.0.1 Linux 5.9

    Hey guys! Hope you all are doing good
    I am kinda confused migrating my 11.2.0.1 Oracle Restart installation to a RAC two node (aim is to add a second node along with migrating the existing one to the cluster pool).
    Running on OEL 5.9 x86_64
    I see that we can switch the RAC mode from 'off' to 'on' and use the existing clusterware binaries. I am trying to follow the below Oracle documentation for linux
    6.2 Migrating Standalone Grid Infrastructure Servers to a Cluster
    http://docs.oracle.com/cd/E11882_01/install.112/e41961/rem_orcl.htm#CWLIN347
    In this document, I have completed step 5 which is de-configuring Oracle Restart. All smooth.
    The problem starts here. Next step 6 says
    Prepare the server for Oracle Clusterware configuration, as described in this document. In addition, choose if you want to install Oracle Grid Infrastructure for a Cluster in the same location as Oracle Restart, or in a different location:
    You would have noticed that there's a note which says that the process uses config wizard which is available with rel 11.2.0.2 or later. Mine's 11.2.0.1.
    From the doc
    Note:
    This procedure uses Oracle Clusterware Configuration Wizard, available with release 11.2.0.2 and later.
    So how do I proceed from here? Can the existing binaries be used? If yes, how? Can't find much information. Thanks for your valuable time.
    I hope I am not missing anything..
    Warm Regards
    Abhishek

    Hello,
    Then why to they say for example here the following:
    The Oracle GSD (Global Service Daemon) process, ora.gsd, is typically offline. You must enable Oracle GSD manually if you plan to use an Oracle 9i Real Application Clusters database on the Oracle Clusterware 11g release 2 (11.2) cluster.
    ???

  • Installing Grid Infrastructure 11.2.0.2 fails on Oracle Linux 5.6 and 6.0

    Hi all,
    We are in the process of building our new enterprise environment and tried to install the following:
    Oracle Linux 6.0
    Grid Infrastructure 11.2.0.2
    ASM
    OCFS2
    Oracle RDBMS 11.2.0.2
    Installation of Grid Infrastructure failed on Oracle Linux 6.0 and 5.6 as well. Is it because it's not yet certified for these Oracle Linux versions? Where can we find compatibility matrixes or certification information to find out which versions do work together? Been trying to find them with no success.
    Thanks for any help!

    Dude wrote:
    I have recently installed Oracle 11.2.0.2 including ASM, which also installs Grid infrastructure under Oracle Linux 5.6 (default UEK kernel) without any problems. I installed the oracle-validated package prior to the Oracle installation.OEL 5.6 and 11.2..02 Grid Infrastructure installation (for x86_64) failed in our case. Have opened a SR several weeks ago for it, but no resolution. We since reverted back to OEL 5.5 for building and running this specific RAC.
    There's also errors in OEL 5.6's Infiniband s/w stack. The openibd service fails to apply errata correctly because the o/s command used has since changed command line parameters. My guess is that testing happened solely on new h/w where this errata did not have to be applied, thus were not tested. Also, this service failed to start the srp_daemon (is critical for making sure that the storage layer works each and every time and does not cause stale connections that lead to command rejections from the storage server after a reboot of a RAC node)
    I not only had to fix the openibd service to correctly apply the errata, but also write a separate srpd service that dealt with starting up and shutting down the srp_daemon.
    Not very impressed with OEL 5.6 at all and it will be unlikely that we'll use it. And it raises a lot of concern whether OEL 6 would be a robust enough o/s for running our RAC systems.

  • Can we use Grid Infrastructure 11.2 to protect a single instance database?

    Hi all,
    I would like to use Oracle Grid Infrastructure to protect a single-instance database.
    I found these docs:
    http://www.oracle.com/technetwork/products/clusterware/overview/em-db-failover-128585.pdf (for 10.2)
    http://www.oracle.com/technetwork/products/clusterware/overview/si-db-failover-11g-134623.pdf (for 11.1)
    I was unable to find a similar document for GI 11.2 . Do you know if such a document exists?
    At the moment I only know that the GI is free if used for an Oracle Product, but I am not shure whether it can be used to protext external applications or single-instance databases
    Andrea

    Hi,
    while this still works (the scripts for failover you find in the demo folder $GI_HOME7/crs/demo/coldfailover), the document does not exists, since with 11gR2 there is a supported solution called RAC ONE Node, hence rendering the manual setup obsolete.
    For more information see:
    Oracle Clusterware and Application Failover Management (Doc ID 790189.1)
    Regards
    Sebastian

  • 11g R2 RAC - Grid Infrastructure installation - "root.sh" fails on node#2

    Hi there,
    I am trying to create a two node 11g R2 RAC on OEL 5.5 (32-bit) using VMWare virtual machines. I have correctly configured both nodes. Cluster Verification utility returns on following error \[which I believe can be ignored]:
    Checking daemon liveness...
    Liveness check failed for "ntpd"
    Check failed on nodes:
    rac2,rac1
    PRVF-5415 : Check to see if NTP daemon is running failed
    Clock synchronization check using Network Time Protocol(NTP) failed
    Pre-check for cluster services setup was unsuccessful on all the nodes.
    While Grid Infrastructure installation (for a Cluster option), things go very smooth until I run "root.sh" on node# 2. orainstRoot.sh ran OK on both. "root.sh" run OK on node# 1 and ends with:
    Checking swap space: must be greater than 500 MB.   Actual 1967 MB    Passed
    The inventory pointer is located at /etc/oraInst.loc
    The inventory is located at /u01/app/oraInventory
    *'UpdateNodeList' was successful.*
    *[root@rac1 ~]#*
    "root.sh" fails on rac2 (2nd node) with following error:
    CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
    CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded
    Timed out waiting for the CRS stack to start.
    *[root@rac2 ~]#*
    I know this info may not be enough to figure out what the problem may be. Please let me know what should I look for to find the issue and fix it. Its been like almost two weeks now :-(
    Regards
    Amer

    Hi Zheng,
    ocssd.log is HUGE. So I am putting few of the last lines in the log file hoping they may give some clue:
    2011-07-04 19:49:24.007: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 2180 > margin 1500  cur_ms 36118424 lastalive 36116244
    2011-07-04 19:49:26.005: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 4150 > margin 1500 cur_ms 36120424 lastalive 36116274
    2011-07-04 19:49:26.006: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 4180 > margin 1500  cur_ms 36120424 lastalive 36116244
    2011-07-04 19:49:27.997: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:27.997: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:49:33.001: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:33.001: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:49:37.996: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:37.996: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:49:43.000: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:43.000: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:49:48.004: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:49:48.005: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:12.003: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:12.008: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1660 > margin 1500 cur_ms 36166424 lastalive 36164764
    2011-07-04 19:50:12.009: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1660 > margin 1500  cur_ms 36166424 lastalive 36164764
    2011-07-04 19:50:15.796: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 2130 > margin 1500  cur_ms 36170214 lastalive 36168084
    2011-07-04 19:50:16.996: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:16.996: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:17.826: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1540 > margin 1500 cur_ms 36172244 lastalive 36170704
    2011-07-04 19:50:17.826: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1570 > margin 1500  cur_ms 36172244 lastalive 36170674
    2011-07-04 19:50:21.999: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:21.999: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:26.011: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1740 > margin 1500 cur_ms 36180424 lastalive 36178684
    2011-07-04 19:50:26.011: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1620 > margin 1500  cur_ms 36180424 lastalive 36178804
    2011-07-04 19:50:27.004: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:27.004: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:28.002: [    CSSD][2997803920]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:DATA sched delay 1700 > margin 1500 cur_ms 36182414 lastalive 36180714
    2011-07-04 19:50:28.002: [    CSSD][2997803920]clssnmvSchedDiskThreads: KillBlockThread for voting file ORCL:DATA sched delay 1790 > margin 1500  cur_ms 36182414 lastalive 36180624
    2011-07-04 19:50:31.998: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:31.998: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    2011-07-04 19:50:37.001: [    CSSD][2901298064]clssnmSendingThread: sending status msg to all nodes
    2011-07-04 19:50:37.002: [    CSSD][2901298064]clssnmSendingThread: sent 5 status msgs to all nodes
    *<end of log file>*And the alertrac2.log contains:
    *[root@rac2 rac2]# cat alertrac2.log*
    Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
    2011-07-02 16:43:51.571
    [client(16134)]CRS-2106:The OLR location /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olr is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/client/ocrconfig_16134.log.
    2011-07-02 16:43:57.125
    [client(16134)]CRS-2101:The OLR was formatted using version 3.
    2011-07-02 16:44:43.214
    [ohasd(16188)]CRS-2112:The OLR service started on node rac2.
    2011-07-02 16:45:06.446
    [ohasd(16188)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
    2011-07-02 16:53:30.061
    [ohasd(16188)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2011-07-02 16:53:55.042
    [cssd(17674)]CRS-1713:CSSD daemon is started in exclusive mode
    2011-07-02 16:54:38.334
    [cssd(17674)]CRS-1707:Lease acquisition for node rac2 number 2 completed
    [cssd(17674)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1 and is terminating; details at (:CSSNM00006:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log
    2011-07-02 16:54:38.464
    [cssd(17674)]CRS-1603:CSSD on node rac2 shutdown by user.
    2011-07-02 16:54:39.174
    [ohasd(16188)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac2'.
    2011-07-02 16:55:43.430
    [cssd(17945)]CRS-1713:CSSD daemon is started in clustered mode
    2011-07-02 16:56:02.852
    [cssd(17945)]CRS-1707:Lease acquisition for node rac2 number 2 completed
    2011-07-02 16:56:04.061
    [cssd(17945)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
    2011-07-02 16:56:18.350
    [cssd(17945)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
    2011-07-02 16:56:29.283
    [ctssd(18020)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
    2011-07-02 16:56:29.551
    [ctssd(18020)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
    2011-07-02 16:56:29.615
    [ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 16:56:29.616
    [ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 16:56:29.641
    [ctssd(18020)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
    [client(18052)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(18056)]CRS-10001:ACFS-9322: done.
    2011-07-02 17:01:40.963
    [ohasd(16188)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ohasd/ohasd.log.
    [client(18590)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(18594)]CRS-10001:ACFS-9322: done.
    2011-07-02 17:27:46.385
    [ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 17:27:46.385
    [ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 17:46:48.717
    [crsd(22519)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:49.641
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:46:51.459
    [crsd(22553)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:51.776
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:46:53.928
    [crsd(22574)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:53.956
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:46:55.834
    [crsd(22592)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:56.273
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:46:57.762
    [crsd(22610)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:46:58.631
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:00.259
    [crsd(22628)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:00.968
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:02.513
    [crsd(22645)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:03.309
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:05.081
    [crsd(22663)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:05.770
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:07.796
    [crsd(22681)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:08.257
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:10.733
    [crsd(22699)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:11.739
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:13.547
    [crsd(22732)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 17:47:14.111
    [ohasd(16188)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 17:47:14.112
    [ohasd(16188)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    2011-07-02 17:58:18.459
    [ctssd(18020)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 17:58:18.459
    [ctssd(18020)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    [client(26883)]CRS-10001:ACFS-9200: Supported
    2011-07-02 18:13:34.627
    [ctssd(18020)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user
    2011-07-02 18:13:42.368
    [cssd(17945)]CRS-1603:CSSD on node rac2 shutdown by user.
    2011-07-02 18:15:13.877
    [client(27222)]CRS-2106:The OLR location /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olr is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/client/ocrconfig_27222.log.
    2011-07-02 18:15:14.011
    [client(27222)]CRS-2101:The OLR was formatted using version 3.
    2011-07-02 18:15:23.226
    [ohasd(27261)]CRS-2112:The OLR service started on node rac2.
    2011-07-02 18:15:23.688
    [ohasd(27261)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
    2011-07-02 18:15:24.064
    [ohasd(27261)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
    2011-07-02 18:16:29.761
    [ohasd(27261)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
    2011-07-02 18:16:30.190
    [gpnpd(28498)]CRS-2328:GPNPD started on node rac2.
    2011-07-02 18:16:41.561
    [cssd(28562)]CRS-1713:CSSD daemon is started in exclusive mode
    2011-07-02 18:16:49.111
    [cssd(28562)]CRS-1707:Lease acquisition for node rac2 number 2 completed
    2011-07-02 18:16:49.166
    [cssd(28562)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
    [cssd(28562)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1 and is terminating; details at (:CSSNM00006:) in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log
    2011-07-02 18:17:01.122
    [cssd(28562)]CRS-1603:CSSD on node rac2 shutdown by user.
    2011-07-02 18:17:06.917
    [ohasd(27261)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'rac2'.
    2011-07-02 18:17:23.602
    [mdnsd(28485)]CRS-5602:mDNS service stopping by request.
    2011-07-02 18:17:36.217
    [gpnpd(28732)]CRS-2328:GPNPD started on node rac2.
    2011-07-02 18:17:43.673
    [cssd(28794)]CRS-1713:CSSD daemon is started in clustered mode
    2011-07-02 18:17:49.826
    [cssd(28794)]CRS-1707:Lease acquisition for node rac2 number 2 completed
    2011-07-02 18:17:49.865
    [cssd(28794)]CRS-1605:CSSD voting file is online: ORCL:DATA; details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/cssd/ocssd.log.
    2011-07-02 18:18:03.049
    [cssd(28794)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
    2011-07-02 18:18:06.160
    [ctssd(28861)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
    2011-07-02 18:18:06.220
    [ctssd(28861)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
    2011-07-02 18:18:06.238
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 18:18:06.239
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 18:18:06.794
    [ctssd(28861)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
    [client(28891)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
    [client(28895)]CRS-10001:ACFS-9322: done.
    2011-07-02 18:18:33.465
    [crsd(29020)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:33.575
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:35.757
    [crsd(29051)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:36.129
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:38.596
    [crsd(29066)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:39.146
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:41.058
    [crsd(29085)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:41.435
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:44.255
    [crsd(29101)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:45.165
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:47.013
    [crsd(29121)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:47.409
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:50.071
    [crsd(29136)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:50.118
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:51.843
    [crsd(29156)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:52.373
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:54.361
    [crsd(29171)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:54.772
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:56.620
    [crsd(29202)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:57.104
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:58.997
    [crsd(29218)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/crsd/crsd.log.
    2011-07-02 18:18:59.301
    [ohasd(27261)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.
    2011-07-02 18:18:59.302
    [ohasd(27261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
    2011-07-02 18:49:58.070
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 18:49:58.070
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 19:21:33.362
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 19:21:33.362
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 19:52:05.271
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 19:52:05.271
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 20:22:53.696
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 20:22:53.696
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 20:53:43.949
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 20:53:43.949
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 21:24:32.990
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 21:24:32.990
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 21:55:21.907
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 21:55:21.908
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 22:26:45.752
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 22:26:45.752
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 22:57:54.682
    [ctssd(28861)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/grid/oracle/product/11.2.0/grid/log/rac2/ctssd/octssd.log.
    2011-07-02 22:57:54.683
    [ctssd(28861)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
    2011-07-02 23:07:28.603
    [cssd(28794)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.020 seconds
    2011-07-02 23:07:35.621
    [cssd(28794)]CRS-1611:Network communication with node rac1 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 7.010 seconds
    2011-07-02 23:07:39.629
    [cssd(28794)]CRS-1610:Network communication with node rac1 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 3.000 seconds
    2011-07-02 23:07:42.641
    [cssd(28794)]CRS-1632:Node rac1 is being removed from the cluster in cluster incarnation 205080558
    2011-07-02 23:07:44.751
    [cssd(28794)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
    2011-07-02 23:07:45.326
    [ctssd(28861)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
    2011-07-04 19:46:26.008
    [ohasd(27261)]CRS-8011:reboot advisory message from host: rac1, component: mo155738, with time stamp: L-2011-07-04-19:44:43.318
    [ohasd(27261)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
    *[root@rac2 rac2]#* This log file start with complaint that OLR is not accessible. Here is what I see (rca2):
    -rw------- 1 root oinstall 272756736 Jul  2 18:18 /u01/grid/oracle/product/11.2.0/grid/cdata/rac2.olrAnd I guess rest of the problems start with this.

Maybe you are looking for

  • Iphone died and wont start up/detect in itunes

    Hey there! Over the weekend, my iphone 4's battery died. just before that, it showed a notification that is hould connect it to the computer to update its software. While the power was working then, i couldnt slide to unlock the phone. When i got hom

  • Recording Vocals On iPod Touch For Garage Band etc?

    When using the 'app' 'Garage Band' I want to be able to record people singing in a group and singles. I know there's a built in mic but, I understand this records ambient surrounding sounds. The iPod Touch doens't come with a plug in mic/headphone (h

  • Potential problem when using a template

    Hi, I am fairly new to dreamweaver and I have created a template for my website, I am now using that template to produce a number of other pages. When I come to save it I save it as a new file,it is saving as a HTM file rather than a HTML file, is th

  • Mp495 is printing document with double or shadow letter

    My canon mp495 is printing my documents with a shadow/double letters I aligned it and cleaned the roller, bottom plates,& deep cleaned the nozzle and still have same result please help

  • The Curspr doesn't shown in the fields , using Jinitiator ! PLEASE HELP

    Hi firends fro shwing forms in the client's browser , when i use Oracle JInitiator my big problem is that the cursor doesn't shown in the fields and i don't have clipboard action like Selection , Copy , and Paste on the fields !