CLSRSC-117: Failed to start Oracle Clusterware stack Oracle 12c clusterware

I'm installing Oracle 12c clusterware on 2 nodes.
i installed it on node 1 and after executing the ./root.sh is was executed successfully.
but after trying to execute it on the other node it fails with the next error:
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'dbrac1'
CRS-2883: Resource 'ora.cssdmonitor' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2015/02/11 01:53:15 CLSRSC-117: Failed to start Oracle Clusterware stack
my ASM disks are fine and i can see them via oracleasm scandisks and oracleasm listdiks.

Sorry it was successful on node 2 dbrac2 (which is my main node and has the oracle 12c sources)
here is the complete logs:
2015-02-11 00:58:11: Checking parameters from paramfile /u01/app/12.1.0/grid_1/crs/install/crsconfig_params to validate installer variables
2015-02-11 00:58:11: Set BC to 1
2015-02-11 00:58:11: ASM_CONFIG is near
2015-02-11 00:58:11: The configuration parameter file /u01/app/12.1.0/grid_1/crs/install/crsconfig_params  is valid
2015-02-11 00:58:11: ### Printing the configuration values from files:
2015-02-11 00:58:11:    /u01/app/12.1.0/grid_1/crs/install/crsconfig_params
2015-02-11 00:58:11:    /u01/app/12.1.0/grid_1/crs/install/s_crsconfig_defs
2015-02-11 00:58:11: ASM_AU_SIZE=1
2015-02-11 00:58:11: ASM_CONFIG=near
2015-02-11 00:58:11: ASM_CREDENTIALS=
2015-02-11 00:58:11: ASM_DISCOVERY_STRING=
2015-02-11 00:58:11: ASM_DISKS=ORCL:OC1,ORCL:OC2,ORCL:OC3
2015-02-11 00:58:11: ASM_DISK_GROUP=Votting
2015-02-11 00:58:11: ASM_REDUNDANCY=EXTERNAL
2015-02-11 00:58:11: ASM_SPFILE=
2015-02-11 00:58:11: ASM_UPGRADE=false
2015-02-11 00:58:11: BIG_CLUSTER=true
2015-02-11 00:58:11: CLSCFG_MISSCOUNT=
2015-02-11 00:58:11: CLUSTER_GUID=
2015-02-11 00:58:11: CLUSTER_NAME=dbrac-cluster
2015-02-11 00:58:11: CRFHOME="/u01/app/12.1.0/grid_1"
2015-02-11 00:58:11: CRS_LIMIT_CORE=unlimited
2015-02-11 00:58:11: CRS_LIMIT_MEMLOCK=unlimited
2015-02-11 00:58:11: CRS_LIMIT_OPENFILE=65536
2015-02-11 00:58:11: CRS_LIMIT_STACK=2048
2015-02-11 00:58:11: CRS_LSNR_STACK=10240
2015-02-11 00:58:11: CRS_NODEVIPS='AUTO/255.255.255.0/bond0,AUTO/255.255.255.0/bond0'
2015-02-11 00:58:11: CRS_STORAGE_OPTION=1
2015-02-11 00:58:11: CSS_LEASEDURATION=400
2015-02-11 00:58:11: DIRPREFIX=
2015-02-11 00:58:11: DISABLE_OPROCD=0
2015-02-11 00:58:11: EXTERNAL_ORACLE=/opt/oracle
2015-02-11 00:58:11: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
2015-02-11 00:58:11: GNS_ADDR_LIST=172.20.30.12
2015-02-11 00:58:11: GNS_ALLOW_NET_LIST=
2015-02-11 00:58:11: GNS_CONF=true
2015-02-11 00:58:11: GNS_CREDENTIALS=
2015-02-11 00:58:11: GNS_DENY_ITF_LIST=
2015-02-11 00:58:11: GNS_DENY_NET_LIST=
2015-02-11 00:58:11: GNS_DOMAIN_LIST=gns.kbe.com
2015-02-11 00:58:11: GNS_TYPE=local
2015-02-11 00:58:11: GPNPCONFIGDIR=/u01/app/12.1.0/grid_1
2015-02-11 00:58:11: GPNPGCONFIGDIR=/u01/app/12.1.0/grid_1
2015-02-11 00:58:11: GPNP_PA=
2015-02-11 00:58:11: HOST_NAME_LIST=dbrac2,dbrac1
2015-02-11 00:58:11: HUB_NODE_LIST=dbrac2,dbrac1
2015-02-11 00:58:11: HUB_NODE_VIPS=AUTO,AUTO
2015-02-11 00:58:11: HUB_SIZE=32
2015-02-11 00:58:11: ID=/etc/init.d
2015-02-11 00:58:11: INIT=/sbin/init
2015-02-11 00:58:11: INITCTL=/sbin/initctl
2015-02-11 00:58:11: ISROLLING=true
2015-02-11 00:58:11: IT=/etc/inittab
2015-02-11 00:58:11: JLIBDIR=/u01/app/12.1.0/grid_1/jlib
2015-02-11 00:58:11: JREDIR=/u01/app/12.1.0/grid_1/jdk/jre/
2015-02-11 00:58:11: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
2015-02-11 00:58:11: LISTENER_USERNAME=grid
2015-02-11 00:58:11: MGMT_DB=true
2015-02-11 00:58:11: MSGFILE=/var/adm/messages
2015-02-11 00:58:11: NETWORKS="bond0"/172.20.30.0:public,"bond1"/172.20.190.0:asm,"bond1"/172.20.190.0:cluster_interconnect
2015-02-11 00:58:11: NEW_HOST_NAME_LIST=
2015-02-11 00:58:11: NEW_NODEVIPS='AUTO/255.255.255.0/bond0,AUTO/255.255.255.0/bond0'
2015-02-11 00:58:11: NEW_NODE_NAME_LIST=
2015-02-11 00:58:11: NEW_PRIVATE_NAME_LIST=
2015-02-11 00:58:11: NODELIST=dbrac2,dbrac1
2015-02-11 00:58:11: NODE_NAME_LIST=dbrac2,dbrac1
2015-02-11 00:58:11: OCRCONFIG=/etc/oracle/ocr.loc
2015-02-11 00:58:11: OCRCONFIGDIR=/etc/oracle
2015-02-11 00:58:11: OCRID=
2015-02-11 00:58:11: OCRLOC=ocr.loc
2015-02-11 00:58:11: OCR_LOCATIONS=NO_VAL
2015-02-11 00:58:11: OLASTGASPDIR=/etc/oracle/lastgasp
2015-02-11 00:58:11: OLD_CRS_HOME=
2015-02-11 00:58:11: OLRCONFIG=/etc/oracle/olr.loc
2015-02-11 00:58:11: OLRCONFIGDIR=/etc/oracle
2015-02-11 00:58:11: OLRLOC=olr.loc
2015-02-11 00:58:11: OPROCDCHECKDIR=/etc/oracle/oprocd/check
2015-02-11 00:58:11: OPROCDDIR=/etc/oracle/oprocd
2015-02-11 00:58:11: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
2015-02-11 00:58:11: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
2015-02-11 00:58:11: ORACLE_BASE=/u01/app/grid
2015-02-11 00:58:11: ORACLE_HOME=/u01/app/12.1.0/grid_1
2015-02-11 00:58:11: ORACLE_OWNER=grid
2015-02-11 00:58:11: ORA_ASM_GROUP=asmadmin
2015-02-11 00:58:11: ORA_DBA_GROUP=oinstall
2015-02-11 00:58:11: PRIVATE_NAME_LIST=
2015-02-11 00:58:11: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
2015-02-11 00:58:11: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
2015-02-11 00:58:11: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
2015-02-11 00:58:11: RC_KILL=K15
2015-02-11 00:58:11: RC_KILL_OLD=K96
2015-02-11 00:58:11: RC_KILL_OLD2=K19
2015-02-11 00:58:11: RC_START=S96
2015-02-11 00:58:11: REUSEDG=false
2015-02-11 00:58:11: RIM_NODE_LIST=
2015-02-11 00:58:11: SCAN_NAME=kberac-scan.gns.kbe.com
2015-02-11 00:58:11: SCAN_PORT=1521
2015-02-11 00:58:11: SCRBASE=/etc/oracle/scls_scr
2015-02-11 00:58:11: SILENT=false
2015-02-11 00:58:11: SO_EXT=so
2015-02-11 00:58:11: SRVCFGLOC=srvConfig.loc
2015-02-11 00:58:11: SRVCONFIG=/var/opt/oracle/srvConfig.loc
2015-02-11 00:58:11: SRVCONFIGDIR=/var/opt/oracle
2015-02-11 00:58:11: TZ=Asia/Riyadh
2015-02-11 00:58:11: UPSTART_INIT_DIR=/etc/init
2015-02-11 00:58:11: USER_IGNORED_PREREQ=true
2015-02-11 00:58:11: VNDR_CLUSTER=false
2015-02-11 00:58:11: VOTING_DISKS=NO_VAL
2015-02-11 00:58:11: ### Printing other configuration values ###
2015-02-11 00:58:11: CLSCFG_EXTRA_PARMS=
2015-02-11 00:58:11: HAS_GROUP=oinstall
2015-02-11 00:58:11: HAS_USER=root
2015-02-11 00:58:11: HOST=dbrac1
2015-02-11 00:58:11: OLR_DIRECTORY=/u01/app/12.1.0/grid_1/cdata
2015-02-11 00:58:11: OLR_LOCATION=/u01/app/12.1.0/grid_1/cdata/dbrac1.olr
2015-02-11 00:58:11: ORA_CRS_HOME=/u01/app/12.1.0/grid_1
2015-02-11 00:58:11: SIHA=0
2015-02-11 00:58:11: SUCC_REBOOT=0
2015-02-11 00:58:11: SUPERUSER=root
2015-02-11 00:58:11: addfile=/u01/app/12.1.0/grid_1/crs/install/crsconfig_addparams
2015-02-11 00:58:11: crscfg_trace=1
2015-02-11 00:58:11: crscfg_trace_file=/u01/app/12.1.0/grid_1/cfgtoollogs/crsconfig/rootcrs_dbrac1_2015-02-11_00-58-11AM.log
2015-02-11 00:58:11: old_nodevips=
2015-02-11 00:58:11: osdfile=/u01/app/12.1.0/grid_1/crs/install/s_crsconfig_defs
2015-02-11 00:58:11: parameters_valid=1
2015-02-11 00:58:11: paramfile=/u01/app/12.1.0/grid_1/crs/install/crsconfig_params
2015-02-11 00:58:11: platform_family=unix
2015-02-11 00:58:11: srvctl_trc_suff=0
2015-02-11 00:58:11: stackStartLevel=11
2015-02-11 00:58:11: user_is_superuser=1
2015-02-11 00:58:11: ### Printing of configuration values complete ###
2015-02-11 00:58:11: Save the ASM password file location: +Votting/orapwASM
2015-02-11 00:58:11: Performing few checks before running scripts
2015-02-11 00:58:11: Attempt to get current working directory
2015-02-11 00:58:11: Running as user grid: pwd
2015-02-11 00:58:11: s_run_as_user2: Running /bin/su grid -c ' pwd '
2015-02-11 00:58:11: Removing file /tmp/fileCnF377
2015-02-11 00:58:11: Successfully removed file: /tmp/fileCnF377
2015-02-11 00:58:11: /bin/su successfully executed
2015-02-11 00:58:11: Pre-checkes for running scripts passed
2015-02-11 00:58:11: Instantiating scripts in GI home: /u01/app/12.1.0/grid_1
2015-02-11 00:58:11: saving current owner/permisssios of  parent dir of /u01/app/12.1.0
2015-02-11 00:58:11: saving parent dir ACLs in /u01/app/12.1.0/grid_1/crs/install/ParentDirPerm_dbrac1.txt
2015-02-11 00:58:11: loading... /u01/app/12.1.0/grid_1/crs/install/ParentDirPerm_dbrac1.txt
2015-02-11 00:58:11: Getting file permissions for /u01/app/12.1.0
2015-02-11 00:58:11: Getting file permissions for /u01/app
2015-02-11 00:58:11: Getting file permissions for /u01
2015-02-11 00:58:11: Executing cmd: /u01/app/12.1.0/grid_1/bin/sqlplus -V
2015-02-11 00:58:11: Command output:

>  SQL*Plus: Release 12.1.0.1.0 Production
>  
>End Command output
2015-02-11 00:58:11: Got CRS release version: 12.1.0.1.0
2015-02-11 00:58:11:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK
2015-02-11 00:58:11: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK
2015-02-11 00:58:11: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK '
2015-02-11 00:58:11: Removing file /tmp/fileBau02q
2015-02-11 00:58:11: Successfully removed file: /tmp/fileBau02q
2015-02-11 00:58:11: /bin/su successfully executed
2015-02-11 00:58:11: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK -pname VERSION
2015-02-11 00:58:11: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK -pname VERSION '
2015-02-11 00:58:12: Removing file /tmp/fileMP6nyK
2015-02-11 00:58:12: Successfully removed file: /tmp/fileMP6nyK
2015-02-11 00:58:12: /bin/su successfully executed
2015-02-11 00:58:12: The property VERSION for checkpoint:ROOTCRS_STACK is there
2015-02-11 00:58:12: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -listckpt -name ROOTCRS_STACK -pname VERSION
2015-02-11 00:58:12: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -listckpt -name ROOTCRS_STACK -pname VERSION '
2015-02-11 00:58:12: Removing file /tmp/filelarvE4
2015-02-11 00:58:12: Successfully removed file: /tmp/filelarvE4
2015-02-11 00:58:12: /bin/su successfully executed
2015-02-11 00:58:12: Succeeded to get property value:VERSION=12.1.0.1.0
2015-02-11 00:58:12: version1 is 12.1.0.1.0
2015-02-11 00:58:12: version2 is 12.1.0.1.0
2015-02-11 00:58:12: Version match status is 1
2015-02-11 00:58:12: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK -status
2015-02-11 00:58:12: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK -status '
2015-02-11 00:58:12: Removing file /tmp/file2xL0lp
2015-02-11 00:58:12: Successfully removed file: /tmp/file2xL0lp
2015-02-11 00:58:12: /bin/su successfully executed
2015-02-11 00:58:12: The 'ROOTCRS_STACK' status is FAILED
2015-02-11 00:58:12: USER_IGNORED_PREREQ  is set to 1
2015-02-11 00:58:12: Executing cmd: /u01/app/12.1.0/grid_1/bin/clsecho -p has -f clsrsc -m 363
2015-02-11 00:58:12: Command output:
>  CLSRSC-363: User ignored prerequisites during installation
>End Command output
2015-02-11 00:58:12: Executing cmd: /u01/app/12.1.0/grid_1/bin/clsecho -p has -f clsrsc -m 363
2015-02-11 00:58:12: Command output:
>  CLSRSC-363: User ignored prerequisites during installation
>End Command output
2015-02-11 00:58:12: CLSRSC-363: User ignored prerequisites during installation
2015-02-11 00:58:12: Set BC to 1
2015-02-11 00:58:12: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK -status
2015-02-11 00:58:12: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK -status '
2015-02-11 00:58:12: Removing file /tmp/fileUO8xNK
2015-02-11 00:58:12: Successfully removed file: /tmp/fileUO8xNK
2015-02-11 00:58:12: /bin/su successfully executed
2015-02-11 00:58:12: The 'ROOTCRS_STACK' status is FAILED
2015-02-11 00:58:12: Saving the configuration parameter file data
2015-02-11 00:58:12: checkpoint status of ROOTCRS_STACK is FAIL
2015-02-11 00:58:12:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_PARAM
2015-02-11 00:58:12: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_PARAM
2015-02-11 00:58:12: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_PARAM '
2015-02-11 00:58:13: Removing file /tmp/file53D9Q6
2015-02-11 00:58:13: Successfully removed file: /tmp/file53D9Q6
2015-02-11 00:58:13: /bin/su successfully executed
2015-02-11 00:58:13:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OSDSETUP
2015-02-11 00:58:13: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OSDSETUP
2015-02-11 00:58:13: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OSDSETUP '
2015-02-11 00:58:13: Removing file /tmp/fileJAZWvt
2015-02-11 00:58:13: Successfully removed file: /tmp/fileJAZWvt
2015-02-11 00:58:13: /bin/su successfully executed
2015-02-11 00:58:13: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OSDSETUP -status
2015-02-11 00:58:13: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OSDSETUP -status '
2015-02-11 00:58:13: Removing file /tmp/filesnoyLQ
2015-02-11 00:58:13: Successfully removed file: /tmp/filesnoyLQ
2015-02-11 00:58:13: /bin/su successfully executed
2015-02-11 00:58:13: The 'ROOTCRS_OSDSETUP' status is SUCCESS
2015-02-11 00:58:13: Oracle CRS home = /u01/app/12.1.0/grid_1
2015-02-11 00:58:13: Host name = dbrac1
2015-02-11 00:58:13: CRS user = grid
2015-02-11 00:58:13: Oracle CRS home = /u01/app/12.1.0/grid_1
2015-02-11 00:58:13: GPnP host = dbrac1
2015-02-11 00:58:13: Oracle GPnP home = /u01/app/12.1.0/grid_1/gpnp
2015-02-11 00:58:13: Oracle GPnP local home = /u01/app/12.1.0/grid_1/gpnp/dbrac1
2015-02-11 00:58:13: GPnP directories verified.
2015-02-11 00:58:13: Checking to see if Oracle CRS stack is already configured
2015-02-11 00:58:13: Oracle CRS stack is already configured and will be running under init (1M)
2015-02-11 00:58:13: ---Checking local gpnp setup...
2015-02-11 00:58:13: chk gpnphome /u01/app/12.1.0/grid_1/gpnp/dbrac1: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
2015-02-11 00:58:13: gpnptool: run /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/peer" -wu=peer
2015-02-11 00:58:13: Running as user grid: /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/peer" -wu=peer
2015-02-11 00:58:13: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/peer" -wu=peer '
2015-02-11 00:58:13: Removing file /tmp/fileC5RBEe
2015-02-11 00:58:13: Successfully removed file: /tmp/fileC5RBEe
2015-02-11 00:58:13: /bin/su successfully executed
2015-02-11 00:58:13: gpnptool: rc=0
2015-02-11 00:58:13: gpnptool output:
Profile signature is valid.
2015-02-11 00:58:13: Profile "/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/peer"
2015-02-11 00:58:13: gpnptool: run /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/prdr" -wu=peer
2015-02-11 00:58:13: Running as user grid: /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/prdr" -wu=peer
2015-02-11 00:58:13: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/prdr" -wu=peer '
2015-02-11 00:58:14: Removing file /tmp/filebJUCzD
2015-02-11 00:58:14: Successfully removed file: /tmp/filebJUCzD
2015-02-11 00:58:14: /bin/su successfully executed
2015-02-11 00:58:14: gpnptool: rc=0
2015-02-11 00:58:14: gpnptool output:
Profile signature is valid.
2015-02-11 00:58:14: Profile "/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/prdr"
2015-02-11 00:58:14: chk gpnphome /u01/app/12.1.0/grid_1/gpnp/dbrac1: OK
2015-02-11 00:58:14: gpnptool: run /u01/app/12.1.0/grid_1/bin/gpnptool getpval -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -o="/tmp/fileW3oHv3" -prf_cn -prf_cid -prf_sq -prf_pa
2015-02-11 00:58:14: Running as user grid: /u01/app/12.1.0/grid_1/bin/gpnptool getpval -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -o="/tmp/fileW3oHv3" -prf_cn -prf_cid -prf_sq -prf_pa
2015-02-11 00:58:14: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/gpnptool getpval -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -o="/tmp/fileW3oHv3" -prf_cn -prf_cid -prf_sq -prf_pa '
2015-02-11 00:58:14: Removing file /tmp/fileXuqOrt
2015-02-11 00:58:14: Successfully removed file: /tmp/fileXuqOrt
2015-02-11 00:58:14: /bin/su successfully executed
2015-02-11 00:58:14: gpnptool: rc=0
2015-02-11 00:58:14: gpnptool output:
2015-02-11 00:58:14: Removing file /tmp/fileW3oHv3
2015-02-11 00:58:14: Successfully removed file: /tmp/fileW3oHv3
2015-02-11 00:58:14: setting profile permissions
2015-02-11 00:58:14: resfiles are /u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/root/ewallet.p12 /u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/peer/cwallet.sso /u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/prdr/cwallet.sso /u01/app/12.1.0/grid_1/gpnp/dbrac1/wallets/pa/cwallet.sso
2015-02-11 00:58:14: GPnP Wallets ownership/permissions successfully set. [node-local=1]
2015-02-11 00:58:14: ---Checking cluster-wide gpnp setup...
2015-02-11 00:58:14: chk gpnphome /u01/app/12.1.0/grid_1/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
2015-02-11 00:58:14: gpnptool: run /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/wallets/peer" -wu=peer
2015-02-11 00:58:14: Running as user grid: /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/wallets/peer" -wu=peer
2015-02-11 00:58:14: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/wallets/peer" -wu=peer '
2015-02-11 00:58:15: Removing file /tmp/filefqLPlU
2015-02-11 00:58:15: Successfully removed file: /tmp/filefqLPlU
2015-02-11 00:58:15: /bin/su successfully executed
2015-02-11 00:58:15: gpnptool: rc=0
2015-02-11 00:58:15: gpnptool output:
Profile signature is valid.
2015-02-11 00:58:15: Profile "/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/app/12.1.0/grid_1/gpnp/wallets/peer"
2015-02-11 00:58:15: gpnptool: run /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/wallets/prdr" -wu=peer
2015-02-11 00:58:15: Running as user grid: /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/wallets/prdr" -wu=peer
2015-02-11 00:58:15: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/wallets/prdr" -wu=peer '
2015-02-11 00:58:15: Removing file /tmp/fileudaEhm
2015-02-11 00:58:15: Successfully removed file: /tmp/fileudaEhm
2015-02-11 00:58:15: /bin/su successfully executed
2015-02-11 00:58:15: gpnptool: rc=0
2015-02-11 00:58:15: gpnptool output:
Profile signature is valid.
2015-02-11 00:58:15: Profile "/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/app/12.1.0/grid_1/gpnp/wallets/prdr"
2015-02-11 00:58:15: chk gpnphome /u01/app/12.1.0/grid_1/gpnp: OK
2015-02-11 00:58:15: gpnptool: run /u01/app/12.1.0/grid_1/bin/gpnptool getpval -p="/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" -o="/tmp/fileYSmOeP" -prf_cn -prf_cid -prf_sq -prf_pa
2015-02-11 00:58:15: Running as user grid: /u01/app/12.1.0/grid_1/bin/gpnptool getpval -p="/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" -o="/tmp/fileYSmOeP" -prf_cn -prf_cid -prf_sq -prf_pa
2015-02-11 00:58:15: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/gpnptool getpval -p="/u01/app/12.1.0/grid_1/gpnp/profiles/peer/profile.xml" -o="/tmp/fileYSmOeP" -prf_cn -prf_cid -prf_sq -prf_pa '
2015-02-11 00:58:15: Removing file /tmp/filernP1bi
2015-02-11 00:58:15: Successfully removed file: /tmp/filernP1bi
2015-02-11 00:58:15: /bin/su successfully executed
2015-02-11 00:58:15: gpnptool: rc=0
2015-02-11 00:58:15: gpnptool output:
2015-02-11 00:58:15: Removing file /tmp/fileYSmOeP
2015-02-11 00:58:15: Successfully removed file: /tmp/fileYSmOeP
2015-02-11 00:58:15: setting profile permissions
2015-02-11 00:58:15: resfiles are /u01/app/12.1.0/grid_1/gpnp/wallets/root/ewallet.p12 /u01/app/12.1.0/grid_1/gpnp/wallets/peer/cwallet.sso /u01/app/12.1.0/grid_1/gpnp/wallets/prdr/cwallet.sso /u01/app/12.1.0/grid_1/gpnp/wallets/pa/cwallet.sso
2015-02-11 00:58:15: GPnP Wallets ownership/permissions successfully set. [node-local=0]
2015-02-11 00:58:15: gpnp setup checked: local valid? 1 cluster-wide valid? 1
2015-02-11 00:58:15: gpnptool: run /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/wallets/peer" -wu=peer
2015-02-11 00:58:15: Running as user grid: /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/wallets/peer" -wu=peer
2015-02-11 00:58:15: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/gpnptool verify -p="/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" -w="file:/u01/app/12.1.0/grid_1/gpnp/wallets/peer" -wu=peer '
2015-02-11 00:58:16: Removing file /tmp/fileaK8v7L
2015-02-11 00:58:16: Successfully removed file: /tmp/fileaK8v7L
2015-02-11 00:58:16: /bin/su successfully executed
2015-02-11 00:58:16: gpnptool: rc=0
2015-02-11 00:58:16: gpnptool output:
Profile signature is valid.
2015-02-11 00:58:16: Profile "/u01/app/12.1.0/grid_1/gpnp/dbrac1/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/app/12.1.0/grid_1/gpnp/wallets/peer"
2015-02-11 00:58:16: Local and Cluster-wide setups signed with same wallet.
gpnp setup: CLUSTERWIDE
2015-02-11 00:58:16: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl query crs releaseversion
2015-02-11 00:58:16: Command output:
>  Oracle High Availability Services release version on the local node is [12.1.0.1.0]
>End Command output
2015-02-11 00:58:16: Version String passed is: [Oracle High Availability Services release version on the local node is [12.1.0.1.0]]
2015-02-11 00:58:16: Version Info returned is : [12.1.0.1.0]
2015-02-11 00:58:16: Got CRS release version: 12.1.0.1.0
2015-02-11 00:58:16: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK -status
2015-02-11 00:58:16: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_STACK -status '
2015-02-11 00:58:16: Removing file /tmp/fileshWmsh
2015-02-11 00:58:16: Successfully removed file: /tmp/fileshWmsh
2015-02-11 00:58:16: /bin/su successfully executed
2015-02-11 00:58:16: The 'ROOTCRS_STACK' is either in START/FAILED state
2015-02-11 00:58:16:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OLR
2015-02-11 00:58:16: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OLR
2015-02-11 00:58:16: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OLR '
2015-02-11 00:58:16: Removing file /tmp/fileIB67nN
2015-02-11 00:58:16: Successfully removed file: /tmp/fileIB67nN
2015-02-11 00:58:16: /bin/su successfully executed
2015-02-11 00:58:16: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OLR -status
2015-02-11 00:58:16: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OLR -status '
2015-02-11 00:58:16: Removing file /tmp/filenbKqUj
2015-02-11 00:58:16: Successfully removed file: /tmp/filenbKqUj
2015-02-11 00:58:16: /bin/su successfully executed
2015-02-11 00:58:16: The 'ROOTCRS_OLR' status is SUCCESS
2015-02-11 00:58:16: 'ROOTCRS_OLR' state is 'SUCCESS'
2015-02-11 00:58:16: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OLR -status
2015-02-11 00:58:16: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OLR -status '
2015-02-11 00:58:17: Removing file /tmp/filevkRq2Q
2015-02-11 00:58:17: Successfully removed file: /tmp/filevkRq2Q
2015-02-11 00:58:17: /bin/su successfully executed
2015-02-11 00:58:17: The 'ROOTCRS_OLR' status is SUCCESS
2015-02-11 00:58:17: OLR is already initialized
2015-02-11 00:58:17: Executing /u01/app/12.1.0/grid_1/bin/crsctl create scr grid
2015-02-11 00:58:17: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl create scr grid
2015-02-11 00:58:17:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_GPNPSETUP
2015-02-11 00:58:17: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_GPNPSETUP
2015-02-11 00:58:17: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_GPNPSETUP '
2015-02-11 00:58:17: Removing file /tmp/filevuHPep
2015-02-11 00:58:17: Successfully removed file: /tmp/filevuHPep
2015-02-11 00:58:17: /bin/su successfully executed
2015-02-11 00:58:17: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_GPNPSETUP -status
2015-02-11 00:58:17: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_GPNPSETUP -status '
2015-02-11 00:58:17: Removing file /tmp/filegPte2X
2015-02-11 00:58:17: Successfully removed file: /tmp/filegPte2X
2015-02-11 00:58:17: /bin/su successfully executed
2015-02-11 00:58:17: The 'ROOTCRS_GPNPSETUP' status is SUCCESS
2015-02-11 00:58:17: 'ROOTCRS_GPNPSETUP' state is SUCCESS
2015-02-11 00:58:17: Local GPNP is already initialized
2015-02-11 00:58:17: Pin process list is osysmond.bin,ologgerd,ocssd.bin,cssdmonitor,cssdagent,mdb_pmon_-MGMTDB,kswapd0
2015-02-11 00:58:17:   copy "/u01/app/12.1.0/grid_1/crf/admin/crfdbrac1.ora" => "/tmp/DRDpfvHuaC"
2015-02-11 00:58:17: Copied existing orafile
2015-02-11 00:58:17: Creating CHM/OS config file /u01/app/12.1.0/grid_1/crf/admin/crfdbrac1.ora
2015-02-11 00:58:17: Trying to check if new CRS stack is partially up
2015-02-11 00:58:17: check new crs stack
2015-02-11 00:58:17: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl check resource ora.cssd -init
2015-02-11 00:58:17: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status resource ora.cssd -init
2015-02-11 00:58:18: Checking the status of ora.cssd
2015-02-11 00:58:23: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status resource ora.cssd -init
2015-02-11 00:58:23: Checking the status of ora.cssd
2015-02-11 00:58:28: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status resource ora.cssd -init
2015-02-11 00:58:28: Checking the status of ora.cssd
2015-02-11 00:58:33: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl check resource ora.crsd -init
2015-02-11 00:58:33: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status resource ora.crsd -init
2015-02-11 00:58:33: Checking the status of ora.crsd
2015-02-11 00:58:38: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status resource ora.crsd -init
2015-02-11 00:58:38: Checking the status of ora.crsd
2015-02-11 00:58:43: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status resource ora.crsd -init
2015-02-11 00:58:44: Checking the status of ora.crsd
2015-02-11 00:58:49: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl check resource ora.evmd -init
2015-02-11 00:58:49: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status resource ora.evmd -init
2015-02-11 00:58:49: Checking the status of ora.evmd
2015-02-11 00:58:54: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status resource ora.evmd -init
2015-02-11 00:58:54: Checking the status of ora.evmd
2015-02-11 00:58:59: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status resource ora.evmd -init
2015-02-11 00:58:59: Checking the status of ora.evmd
2015-02-11 00:59:04: Oracle Grid Infrastructure is not running from /u01/app/12.1.0/grid_1
2015-02-11 00:59:04: Deleting older CHM/OS repository at
2015-02-11 00:59:04:   copy "/tmp/DRDpfvHuaC" => "/u01/app/12.1.0/grid_1/crf/admin/crfdbrac1.ora"
2015-02-11 00:59:04: Check and delete older IPD/OS installation
2015-02-11 00:59:04: INFO: The OS Tool is not installed at /usr/lib/oracrf.
2015-02-11 00:59:04: Registering ohasd
2015-02-11 00:59:04:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OHASD
2015-02-11 00:59:04: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OHASD
2015-02-11 00:59:04: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OHASD '
2015-02-11 00:59:05: Removing file /tmp/fileF65ZJD
2015-02-11 00:59:05: Successfully removed file: /tmp/fileF65ZJD
2015-02-11 00:59:05: /bin/su successfully executed
2015-02-11 00:59:05: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OHASD -status
2015-02-11 00:59:05: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_OHASD -status '
2015-02-11 00:59:05: Removing file /tmp/filewDoW1j
2015-02-11 00:59:05: Successfully removed file: /tmp/filewDoW1j
2015-02-11 00:59:05: /bin/su successfully executed
2015-02-11 00:59:05: The 'ROOTCRS_OHASD' status is SUCCESS
2015-02-11 00:59:05: 'ROOTCRS_OHASD' state is SUCCESS
2015-02-11 00:59:05: ohasd already registered
2015-02-11 00:59:05: Starting ohasd
2015-02-11 00:59:05: Checking the status of ohasd
2015-02-11 00:59:05: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl check has
2015-02-11 00:59:05: ohasd is already running
2015-02-11 00:59:05: Checking the status of ohasd
2015-02-11 00:59:05: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl check has
2015-02-11 00:59:05: ohasd started successfully
2015-02-11 00:59:05: Creating OHASD resources and dependencies
2015-02-11 00:59:05: Configuring HASD
2015-02-11 00:59:05:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_INITRES
2015-02-11 00:59:05: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_INITRES
2015-02-11 00:59:05: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_INITRES '
2015-02-11 00:59:05: Removing file /tmp/fileral9N1
2015-02-11 00:59:05: Successfully removed file: /tmp/fileral9N1
2015-02-11 00:59:05: /bin/su successfully executed
2015-02-11 00:59:05: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_INITRES -status
2015-02-11 00:59:05: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_INITRES -status '
2015-02-11 00:59:06: Removing file /tmp/fileL9iNaK
2015-02-11 00:59:06: Successfully removed file: /tmp/fileL9iNaK
2015-02-11 00:59:06: /bin/su successfully executed
2015-02-11 00:59:06: The 'ROOTCRS_INITRES' status is SUCCESS
2015-02-11 00:59:06: 'ROOTCRS_INITRES' state is SUCCESS
2015-02-11 00:59:06: OHASD Resources are already configured
2015-02-11 00:59:06: Successfully created OHASD resources for cluster and ASM
2015-02-11 00:59:06:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_AFDINST
2015-02-11 00:59:06: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_AFDINST
2015-02-11 00:59:06: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_AFDINST '
2015-02-11 00:59:06: Removing file /tmp/fileqVR08s
2015-02-11 00:59:06: Successfully removed file: /tmp/fileqVR08s
2015-02-11 00:59:06: /bin/su successfully executed
2015-02-11 00:59:06: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_AFDINST -status
2015-02-11 00:59:06: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_AFDINST -status '
2015-02-11 00:59:06: Removing file /tmp/fileOqJoHc
2015-02-11 00:59:06: Successfully removed file: /tmp/fileOqJoHc
2015-02-11 00:59:06: /bin/su successfully executed
2015-02-11 00:59:06: The 'ROOTCRS_AFDINST' status is SUCCESS
2015-02-11 00:59:06:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_ACFSINST
2015-02-11 00:59:06: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_ACFSINST
2015-02-11 00:59:06: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_ACFSINST '
2015-02-11 00:59:06: Removing file /tmp/file1r4ZQW
2015-02-11 00:59:06: Successfully removed file: /tmp/file1r4ZQW
2015-02-11 00:59:06: /bin/su successfully executed
2015-02-11 00:59:06: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_ACFSINST -status
2015-02-11 00:59:06: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_ACFSINST -status '
2015-02-11 00:59:06: Removing file /tmp/filePXzECH
2015-02-11 00:59:06: Successfully removed file: /tmp/filePXzECH
2015-02-11 00:59:06: /bin/su successfully executed
2015-02-11 00:59:06: The 'ROOTCRS_ACFSINST' status is SUCCESS
2015-02-11 00:59:06: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl stop crs -f
2015-02-11 00:59:09: Command output:
>  CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbrac1'
>  CRS-2673: Attempting to stop 'ora.evmd' on 'dbrac1'
>  CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'dbrac1'
>  CRS-2673: Attempting to stop 'ora.mdnsd' on 'dbrac1'
>  CRS-2673: Attempting to stop 'ora.gipcd' on 'dbrac1'
>  CRS-2673: Attempting to stop 'ora.gpnpd' on 'dbrac1'
>  CRS-2677: Stop of 'ora.cssdmonitor' on 'dbrac1' succeeded
>  CRS-2677: Stop of 'ora.gipcd' on 'dbrac1' succeeded
>  CRS-2677: Stop of 'ora.evmd' on 'dbrac1' succeeded
>  CRS-2677: Stop of 'ora.mdnsd' on 'dbrac1' succeeded
>  CRS-2677: Stop of 'ora.gpnpd' on 'dbrac1' succeeded
>  CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbrac1' has completed
>  CRS-4133: Oracle High Availability Services has been stopped.
>End Command output
2015-02-11 00:59:09: The return value of stop of CRS: 0
2015-02-11 00:59:09: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl check crs
2015-02-11 00:59:09: Command output:
>  CRS-4639: Could not contact Oracle High Availability Services
>End Command output
2015-02-11 00:59:09: Oracle CRS stack has been shut down
2015-02-11 00:59:09:      ckpt: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_BOOTCFG
2015-02-11 00:59:09: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_BOOTCFG
2015-02-11 00:59:09: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_BOOTCFG '
2015-02-11 00:59:10: Removing file /tmp/file9QLcAB
2015-02-11 00:59:10: Successfully removed file: /tmp/file9QLcAB
2015-02-11 00:59:10: /bin/su successfully executed
2015-02-11 00:59:10: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_BOOTCFG -status
2015-02-11 00:59:10: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -chkckpt -name ROOTCRS_BOOTCFG -status '
2015-02-11 00:59:10: Removing file /tmp/fileMeQD8v
2015-02-11 00:59:10: Successfully removed file: /tmp/fileMeQD8v
2015-02-11 00:59:10: /bin/su successfully executed
2015-02-11 00:59:10: The 'ROOTCRS_BOOTCFG' status is SUCCESS
2015-02-11 00:59:10: 'ROOTCRS_BOOTCFG' state is SUCCESS
2015-02-11 00:59:10: Node specific initial boot configuration already completed
2015-02-11 00:59:10: Startup level is 11
2015-02-11 00:59:10: Attempt to start the whole CRS stack
2015-02-11 00:59:10: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl start crs -wait
2015-02-11 01:09:57: Command output:
>  CRS-4123: Starting Oracle High Availability Services-managed resources
>  CRS-2672: Attempting to start 'ora.mdnsd' on 'dbrac1'
>  CRS-2672: Attempting to start 'ora.evmd' on 'dbrac1'
>  CRS-2676: Start of 'ora.mdnsd' on 'dbrac1' succeeded
>  CRS-2676: Start of 'ora.evmd' on 'dbrac1' succeeded
>  CRS-2672: Attempting to start 'ora.gpnpd' on 'dbrac1'
>  CRS-2676: Start of 'ora.gpnpd' on 'dbrac1' succeeded
>  CRS-2672: Attempting to start 'ora.gipcd' on 'dbrac1'
>  CRS-2676: Start of 'ora.gipcd' on 'dbrac1' succeeded
>  CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbrac1'
>  CRS-2676: Start of 'ora.cssdmonitor' on 'dbrac1' succeeded
>  CRS-2672: Attempting to start 'ora.cssd' on 'dbrac1'
>  CRS-2672: Attempting to start 'ora.diskmon' on 'dbrac1'
>  CRS-2676: Start of 'ora.diskmon' on 'dbrac1' succeeded
>  CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'dbrac1'
>  CRS-2883: Resource 'ora.cssdmonitor' failed during Clusterware stack start.
>  CRS-4406: Oracle High Availability Services synchronous start failed.
>  CRS-4000: Command Start failed, or completed with errors.
>End Command output
2015-02-11 01:09:57: The return value of blocking start of CRS: 1
2015-02-11 01:09:57: Failed to start the Oracle CRS stack
2015-02-11 01:09:57: Executing cmd: /u01/app/12.1.0/grid_1/bin/clsecho -p has -f clsrsc -m 117
2015-02-11 01:09:57: Command output:
>  CLSRSC-117: Failed to start Oracle Clusterware stack
>End Command output
2015-02-11 01:09:57: Executing cmd: /u01/app/12.1.0/grid_1/bin/clsecho -p has -f clsrsc -m 117
2015-02-11 01:09:57: Command output:
>  CLSRSC-117: Failed to start Oracle Clusterware stack
>End Command output
2015-02-11 01:09:57: CLSRSC-117: Failed to start Oracle Clusterware stack
2015-02-11 01:09:57: ###### Begin DIE Stack Trace ######
2015-02-11 01:09:57:     Package         File                 Line Calling  
2015-02-11 01:09:57:     --------------- -------------------- ---- ----------
2015-02-11 01:09:57:  1: main            rootcrs.pl            211 crsutils::dietrap
2015-02-11 01:09:57:  2: crsinstall      crsinstall.pm         941 main::__ANON__
2015-02-11 01:09:57:  3: crsinstall      crsinstall.pm         857 crsinstall::start_cluster
2015-02-11 01:09:57:  4: crsinstall      crsinstall.pm         281 crsinstall::prepare_to_start_clusterware
2015-02-11 01:09:57:  5: crsinstall      crsinstall.pm         219 crsinstall::CRSInstall
2015-02-11 01:09:57:  6: main            rootcrs.pl            334 crsinstall::new
2015-02-11 01:09:57: ####### End DIE Stack Trace #######
2015-02-11 01:09:57: ROOTCRS_STACK checkpoint has failed
2015-02-11 01:09:57: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_STACK -state FAIL
2015-02-11 01:09:57: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_STACK -state FAIL '
2015-02-11 01:09:57: Removing file /tmp/fileJKiQVi
2015-02-11 01:09:57: Successfully removed file: /tmp/fileJKiQVi
2015-02-11 01:09:57: /bin/su successfully executed
2015-02-11 01:09:57: Succeeded in writing the checkpoint:'ROOTCRS_STACK' with status:FAIL
2015-02-11 01:09:57: Running as user grid: /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_STACK -state FAIL
2015-02-11 01:09:57: s_run_as_user2: Running /bin/su grid -c ' /u01/app/12.1.0/grid_1/bin/cluutil -ckpt -oraclebase /u01/app/grid -writeckpt -name ROOTCRS_STACK -state FAIL '
2015-02-11 01:09:58: Removing file /tmp/file3h3DN6
2015-02-11 01:09:58: Successfully removed file: /tmp/file3h3DN6
2015-02-11 01:09:58: /bin/su successfully executed
2015-02-11 01:09:58: Succeeded in writing the checkpoint:'ROOTCRS_STACK' with status:FAIL

Similar Messages

  • Failed to start tns listener on oracle xe on redhat 3

    hi.. i stopped the listener manually
    when i tried to start it, i got:
    Starting /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/bin/tnslsnr: please wait...
    TNSLSNR for Linux: Version 10.2.0.1.0 - Production
    NL-00280: error creating log stream /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/network/log/listener.log
    NL-00278: cannot open log file
    SNL-00016: snlfohd: error opening file
    Linux Error: 13: Permission denied
    i'm starting as root (can't enter user oracle, even though it's not locked accroding to passwd -s)

    >
    but i still cant' login as oracle using login oracle, it still gives me incorrect login.
    >
    This looks like a different problem and would normally require a new thread, but still...can you give us more details, maybe we can help...?
    What's the error you get? Any relevant entries in log files? Did you perform any change on this account lately? if so, how did you do that?

  • Error creating ASComponent oid1,Cause:Failed to start the componentAction

    Hi,
    On 64-bit Windows-7 machine, I have installed Oracle DB (11.1.1.5.0), RCU (11.1.1.5.0),
    Weblogic Server (10.3.5.0) and then I was trying to install Oracle IDM (11.1.1.5.0).
    During IDM installation, I selected the option "Install and Configure".
    Installation set up is successful.
    But it failed while configuring Oracle Identity Management Components - step2 - Start Oracle Internet Directory - Failed.
    Error :
    Error creating ASComponent oid1.
    Cause:
    An internal operation has failed: Failed to start the componentAction
    Logs:
    [OOB IDM CONFIG EVENT] onConfigurationError -> set RETRY ExecutionResponse..
    E:\Oracle\Middleware\Oracle_IDM2 E:\Oracle\Middleware\asinst_1 asinst_1
    oracle.as.config.ProvisionException: Failed to start the component
         at oracle.as.provisioning.fmwadmin.ASComponentProv.createComponent(ASComponentProv.java:157)
         at oracle.as.provisioning.fmwadmin.ASComponentProv.createComponent(ASComponentProv.java:73)
         at oracle.as.provisioning.fmwadmin.ASInstanceProv._createComponent(ASInstanceProv.java:401)
         at oracle.as.provisioning.fmwadmin.ASInstanceProv.createComponent(ASInstanceProv.java:364)
         at oracle.as.provisioning.fmwadmin.ASInstanceProv.createInstanceAndComponents(ASInstanceProv.java:136)
         at oracle.as.provisioning.engine.WorkFlowExecutor._createASInstancesAndComponents(WorkFlowExecutor.java:523)
         at oracle.as.provisioning.engine.WorkFlowExecutor.executeWLSWorkFlow(WorkFlowExecutor.java:439)
         at oracle.as.provisioning.engine.Config.executeConfigWorkflow_WLS(Config.java:866)
         at oracle.as.idm.install.config.IdMDirectoryServicesManager.doExecute(IdMDirectoryServicesManager.java:951)
         at oracle.as.install.engine.modules.configuration.client.ConfigAction.execute(ConfigAction.java:335)
         at oracle.as.install.engine.modules.configuration.action.TaskPerformer.run(TaskPerformer.java:87)
         at oracle.as.install.engine.modules.configuration.action.TaskPerformer.startConfigAction(TaskPerformer.java:104)
         at oracle.as.install.engine.modules.configuration.action.ActionRequest.perform(ActionRequest.java:15)
         at oracle.as.install.engine.modules.configuration.action.RequestQueue.perform(RequestQueue.java:63)
         at oracle.as.install.engine.modules.configuration.standard.StandardConfigActionManager.start(StandardConfigActionManager.java:158)
         at oracle.as.install.engine.modules.configuration.boot.ConfigurationExtension.kickstart(ConfigurationExtension.java:81)
         at oracle.as.install.engine.modules.configuration.ConfigurationModule.run(ConfigurationModule.java:83)
         at java.lang.Thread.run(Thread.java:662)
    oracle.as.provisioning.util.ConfigException:
    创建 ASComponent oid1 时出错。
    Cause: 内部操作失败: Failed to start the componentAction: 有关详细信息, 请查看日志。
         at oracle.as.provisioning.util.ConfigException.createConfigException(ConfigException.java:123)
         at oracle.as.provisioning.fmwadmin.ASInstanceProv._createComponent(ASInstanceProv.java:414)
         at oracle.as.provisioning.fmwadmin.ASInstanceProv.createComponent(ASInstanceProv.java:364)
         at oracle.as.provisioning.fmwadmin.ASInstanceProv.createInstanceAndComponents(ASInstanceProv.java:136)
         at oracle.as.provisioning.engine.WorkFlowExecutor._createASInstancesAndComponents(WorkFlowExecutor.java:523)
         at oracle.as.provisioning.engine.WorkFlowExecutor.executeWLSWorkFlow(WorkFlowExecutor.java:439)
         at oracle.as.provisioning.engine.Config.executeConfigWorkflow_WLS(Config.java:866)
         at oracle.as.idm.install.config.IdMDirectoryServicesManager.doExecute(IdMDirectoryServicesManager.java:951)
         at oracle.as.install.engine.modules.configuration.client.ConfigAction.execute(ConfigAction.java:335)
         at oracle.as.install.engine.modules.configuration.action.TaskPerformer.run(TaskPerformer.java:87)
         at oracle.as.install.engine.modules.configuration.action.TaskPerformer.startConfigAction(TaskPerformer.java:104)
         at oracle.as.install.engine.modules.configuration.action.ActionRequest.perform(ActionRequest.java:15)
         at oracle.as.install.engine.modules.configuration.action.RequestQueue.perform(RequestQueue.java:63)
         at oracle.as.install.engine.modules.configuration.standard.StandardConfigActionManager.start(StandardConfigActionManager.java:158)
         at oracle.as.install.engine.modules.configuration.boot.ConfigurationExtension.kickstart(ConfigurationExtension.java:81)
         at oracle.as.install.engine.modules.configuration.ConfigurationModule.run(ConfigurationModule.java:83)
         at java.lang.Thread.run(Thread.java:662)
    Caused by: oracle.as.config.ProvisionException: Failed to start the component
         at oracle.as.provisioning.fmwadmin.ASComponentProv.createComponent(ASComponentProv.java:157)
         at oracle.as.provisioning.fmwadmin.ASComponentProv.createComponent(ASComponentProv.java:73)
         at oracle.as.provisioning.fmwadmin.ASInstanceProv._createComponent(ASInstanceProv.java:401)
         ... 15 more
    progress in calculate progress50

    Hi,
    Please try
    "Error Creating ASComponent Oid1" Error During OID Configuration When Installing OID11g (Doc ID 1427253.1) that should resolve the issue.

  • RAC, ASM failed to start up on second node , ORA-03113: end-of-file on comm

    i'm installing an RAC with 2 nodes on top of ASM
    when creating ASM Diskgroup , it failed and reported error CRS-0215 failed to start asm on node2
    Oracle 10.2.0.1
    linux CentOs 4.x
    u01/app/oracle/product/10.2.0/db_1/bin/dbca  -progress_only   -configureASM -templateName NO_VALUE -gdbName NO -sid NO      -emConf
    iguration NONE    -diskList /dev/raw/raw2,/dev/raw/raw3  -diskGroupName DATA -datafileJarLocation /u01/app/oracle/product/10.2.0/db_
    1/assistants/dbca/templates  -responseFile NO_VALUE  -nodeinfo node1,node2    -obfuscatedPasswords true   -oratabLocation /u01/app/o
    racle/product/10.2.0/db_1/install/oratab   -asmSysPassword 05dbb0be38ecf8cca822cf3cf99e675448  -redundancy EXTERNA
    [oracle@node2 bin]$ ./crs_stat -t -v
    Name           Type           R/RA   F/FT   Target    State     Host       
    ora....SM1.asm application    0/5    0/0    ONLINE    ONLINE    node1      
    ora....E1.lsnr application    0/5    0/0    ONLINE    ONLINE    node1      
    ora.node1.gsd  application    0/5    0/0    ONLINE    ONLINE    node1      
    ora.node1.ons  application    0/3    0/0    ONLINE    ONLINE    node1      
    ora.node1.vip  application    0/0    0/0    ONLINE    ONLINE    node1      
    ora....SM2.asm application    0/5    0/0    OFFLINE   OFFLINE              
    ora....E2.lsnr application    0/5    0/0    ONLINE    ONLINE    node2      
    ora.node2.gsd  application    0/5    0/0    ONLINE    ONLINE    node2      
    ora.node2.ons  application    0/3    0/0    ONLINE    ONLINE    node2      
    ora.node2.vip  application    0/0    0/0    ONLINE    ONLINE    node2  
    i checked the status , asm is able to start on both nodes if not at the same time ,
    when trying to start the second node , with srvctl or sqlplus , each give the error 03113
    can anyone suggest me of how to bring up both instances ,
    thanks~
    [oracle@node2 bin]$ srvctl stop asm -n node1
    [oracle@node2 bin]$ srvctl start asm -n node1
    [oracle@node2 bin]$ srvctl start asm -n node2
    PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Wed May 27 16:14:50 2009
    node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:Copyright (c) 1982, 2005, Oracle.  All rights reserved.
    node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:Enter user-name: Connected to an idle instance.
    node2:ora.node2.ASM2.asm:
    node2:ora.node2.ASM2.asm:SQL> ORA-03113: end-of-file on communication channel
    node2:ora.node2.ASM2.asm:SQL> Disconnected
    node2:ora.node2.ASM2.asm:
    [code/]
    Edited by: zs_hzh on May 27, 2009 1:25 AM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

    Is it possible to start ASM on second node with SQL*Plus in NOMOUNT state?

  • The Script root.sh problem - ora.asm and ASM and Clusterware Stack failed

    Folks,
    Hello. I am installing Oracle 11gR2 RAC using 2 VMs (rac1 and rac2) whose OS are Oracle Linux 5.6 in VMPlayer according to the website http://appsdbaworkshop.blogspot.com/2011/10/11gr2-rac-on-linux-56-using-vmware.html
    I am installing Grid infrastructure. On step 9 of 10 - execute script /u01/app/grid/root.sh for 2 VMs rac1 and rac2.
    After run root.sh in rac1 successfully. I run root.sh in rac2 and get an error as below:
    [root@rac2 grid]# ./root.sh
    Running Oracle 11g root.sh script...
    The following environment variables are set as:
    ORACLE_OWNER= ora11g
    ORACLE_HOME= /u01/app/grid
    Enter the full pathname of the local bin directory: [usr/local/bin]: /usr/local/bin
    Copying dbhome to /usr/local/bin ...
    Copying oraenv to /usr/local/bin ...
    Copying coraenv to /usr/local/bin ...
    Creating /etc/oratab file...
    Entries will be added to the /etc/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    2012-03-05 16:32:52: Parsing the host name
    2012-03-05 16:32:52: Checking for super user privileges
    2012-03-05 16:32:52: User has super user privileges
    Using configuration parameter file: /u01/app/grid/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Adding daemon to inittab
    CRS-4123: Oracle High Availability Services has been started.
    ohasd is starting
    CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
    An active cluster was found during exclusive startup, restarting to join the cluster
    CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
    CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
    CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
    CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
    CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
    CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
    CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
    Start action for octssd aborted
    CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
    CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac2'
    CRS-2672: Attempting to start 'ora.asm' on 'rac2'
    CRS-2676: Start of 'ora.drivers.acfs' on 'rac2' succeeded
    CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
    CRS-2664: Resource 'ora.ctssd' is already running on 'rac2'
    CRS-4000: Command Start failed, or completed with errors.
    Command return code of 1 (256) from command: /u01/app/grid/bin/crsctl start resource ora.asm -init
    Start of resource "ora.asm -init" failed
    Failed to start ASM
    Failed to start Oracle Clusterware stack
    [root@rac2 grid]#
    As we see the output above, at the end of the output
    1) Start of resource ora.asm -init failed
    2) Failed to start ASM
    3) Failed to start Oracle Clusterware stack
    The runInstaller is in the first VM rac1. My question is:
    Do any folk understand how to solve the script root.sh in rac2 problem ( 3 fails of ora.asm, ASM and Clusterware stack as above) ?
    Thanks.

    Please check there is no firewall exist:
    try this like:
    root.sh fails on second node
    MOS note:
    11gR2 Grid: root.sh Fails to Start the Clusterware on the Second Node Due to Firewall on Private Network [ID 981357.1]
    Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1] (Most probabily this issue)

  • Clusterware Install:root.sh- Failure at final check of Oracle CRS stack. 10

    Hello All,
    Image: !http://systemwars.com/rac/cluster_back.jpg!
    I was attempting to perform the steps in:
    Link: http://www.oracle-base.com/articles/11g/OracleDB11gR1RACInstallationOnLinuxUsingNFS.php
    The only difference is that I decided to use fedora core 12 instead. I did this because I added a second NIC card (USB) and only FC12 would recognize it. I tried to get it to work on Cent 5 but it just wouldn't. The second nic on each machine eth1 are connected via crossover cable, and the interfaces can ping each other just fine, rac1-priv and rac2-priv.
    So here is my setup:
    # Public
    192.168.2.11 rac1.localdomain rac1
    192.168.2.12 rac2.localdomain rac2
    #Private
    192.168.0.11 rac1-priv.localdomain rac1-priv
    192.168.0.12 rac2-priv.localdomain rac2-priv
    #Virtual
    192.168.2.111 rac1-vip.localdomain rac1-vip
    192.168.2.112 rac2-vip.localdomain rac2-vip
    #NAS
    192.168.2.10 mini.localdomain mini
    Mini refers to my Mac mini which I decided to use as the 3rd "server" in the group. I was able to mount/read & write to the file systems just fine. As you can see.
    [root@rac1 ~]# df
    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/mapper/vg_rac1-lv_root
    8063408 5156268 2497540 68% /
    tmpfs 1417456 0 1417456 0% /dev/shm
    /dev/sda1 198337 22080 166017 12% /boot
    mini:/shared_config 488050688 76719808 411074880 16% /u01/shared_config
    mini:/shared_crs 488050688 76719808 411074880 16% /u01/app/crs/product/11.1.0/crs
    mini:/shared_home 488050688 76719808 411074880 16% /u01/app/oracle/product/11.1.0/db_1
    mini:/shared_data 488050688 76719808 411074880 16% /u01/oradata
    [root@rac1 ~]# ssh rac2
    Last login: Mon Dec 21 19:33:38 2009 from rac1.localdomain
    [root@rac2 ~]# df
    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/mapper/vg_rac2-lv_root
    8063408 4958008 2695800 65% /
    tmpfs 1417456 0 1417456 0% /dev/shm
    /dev/sda1 198337 22063 166034 12% /boot
    mini:/shared_config 488050688 76719808 411074880 16% /u01/shared_config
    mini:/shared_crs 488050688 76719808 411074880 16% /u01/app/crs/product/11.1.0/crs
    mini:/shared_home 488050688 76719808 411074880 16% /u01/app/oracle/product/11.1.0/db_1
    mini:/shared_data 488050688 76719808 411074880 16% /u01/oradata[color]
    CLUSTER VERIFY SEEMS OK APART FROM ONE WARNING
    WARNING:
    Could not find a suitable set of interfaces for VIPs.
    which according to this link, "can be safety ignored", although I noticed in the link its an actual ERROR and not a WARNING => http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_11.shtml . I also noted that it saw the public IPs as the possible priv IPs, which I also thought could safety be ignored.
    oracle@rac1 clusterware]$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose
    Performing pre-checks for cluster services setup
    Checking node reachability...
    Check: Node reachability from node "rac1"
      Destination Node                      Reachable?             
      rac2                                  yes                    
      rac1                                  yes                    
    Result: Node reachability check passed from node "rac1".
    Checking user equivalence...
    Check: User equivalence for user "oracle"
      Node Name                             Comment                
      rac2                                  passed                 
      rac1                                  passed                 
    Result: User equivalence check passed for user "oracle".
    Checking administrative privileges...
    Check: Existence of user "oracle"
      Node Name     User Exists               Comment                
      rac2          yes                       passed                 
      rac1          yes                       passed                 
    Result: User existence check passed for "oracle".
    Check: Existence of group "oinstall"
      Node Name     Status                    Group ID               
      rac2          exists                    501                    
      rac1          exists                    501                    
    Result: Group existence check passed for "oinstall".
    Check: Membership of user "oracle" in group "oinstall" [as Primary]
      Node Name         User Exists   Group Exists  User in Group  Primary       Comment    
      rac2              yes           yes           yes           yes           passed     
      rac1              yes           yes           yes           yes           passed     
    Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed.
    Administrative privileges check passed.
    Checking node connectivity...
    Interface information for node "rac2"
      Interface Name    IP Address    Subnet        Subnet Gateway  Default Gateway  Hardware Address
      eth0              192.168.2.12  192.168.2.0   0.0.0.0       192.168.2.1   00:01:6C:XXXX
      eth2              192.168.0.12  192.168.0.0   0.0.0.0       192.168.2.1   00:25:4B:XXXX
    Interface information for node "rac1"
      Interface Name    IP Address    Subnet        Subnet Gateway  Default Gateway  Hardware Address
      eth0              192.168.2.11  192.168.2.0   0.0.0.0       192.168.2.1   00:01:6CXXXXX
      eth1              192.168.0.11  192.168.0.0   0.0.0.0       192.168.2.1   00:25:4B:XXXX
    Check: Node connectivity of subnet "192.168.2.0"
      Source                          Destination                     Connected?     
      rac2:eth0                       rac1:eth0                       yes            
    Result: Node connectivity check passed for subnet "192.168.2.0" with node(s) rac2,rac1.
    Check: Node connectivity of subnet "192.168.0.0"
      Source                          Destination                     Connected?     
      rac2:eth2                       rac1:eth1                       yes            
    Result: Node connectivity check passed for subnet "192.168.0.0" with node(s) rac2,rac1.
    Interfaces found on subnet "192.168.2.0" that are likely candidates for a private interconnect:
    rac2 eth0:192.168.2.12
    rac1 eth0:192.168.2.11
    WARNING:
    Could not find a suitable set of interfaces for VIPs.
    Result: Node connectivity check passed.
    Checking system requirements for 'crs'...
    Check: Total memory
      Node Name     Available                 Required                  Comment  
      rac2          2.7GB (2834912KB)         1GB (1048576KB)           passed   
      rac1          2.7GB (2834912KB)         1GB (1048576KB)           passed   
    Result: Total memory check passed.
    Check: Free disk space in "/tmp" dir
      Node Name     Available                 Required                  Comment  
      rac2          4.58GB (4805204KB)        400MB (409600KB)          passed   
      rac1          10.51GB (11015624KB)      400MB (409600KB)          passed   
    Result: Free disk space check passed.
    Check: Swap space
      Node Name     Available                 Required                  Comment  
      rac2          2GB (2097144KB)           1.5GB (1572864KB)         passed   
      rac1          3GB (3145720KB)           1.5GB (1572864KB)         passed   
    Result: Swap space check passed.
    Check: System architecture
      Node Name     Available                 Required                  Comment  
      rac2          i686                      i686                      passed   
      rac1          i686                      i686                      passed   
    Result: System architecture check passed.
    Check: Kernel version
      Node Name     Available                 Required                  Comment  
      rac2          2.6.31.5-127.fc12.i686.PAE  2.6.9                     passed   
      rac1          2.6.31.5-127.fc12.i686.PAE  2.6.9                     passed   
    Result: Kernel version check passed.
    Check: Package existence for "make-3.81"
      Node Name                       Status                          Comment        
      rac2                            make-3.81-18.fc12.i686          passed         
      rac1                            make-3.81-18.fc12.i686          passed         
    Result: Package existence check passed for "make-3.81".
    Check: Package existence for "binutils-2.17.50.0.6"
      Node Name                       Status                          Comment        
      rac2                            binutils-2.19.51.0.14-34.fc12.i686  passed         
      rac1                            binutils-2.19.51.0.14-34.fc12.i686  passed         
    Result: Package existence check passed for "binutils-2.17.50.0.6".
    Check: Package existence for "gcc-4.1.1"
      Node Name                       Status                          Comment        
      rac2                            gcc-4.4.2-7.fc12.i686           passed         
      rac1                            gcc-4.4.2-7.fc12.i686           passed         
    Result: Package existence check passed for "gcc-4.1.1".
    Check: Package existence for "libaio-0.3.106"
      Node Name                       Status                          Comment        
      rac2                            libaio-0.3.107-9.fc12.i686      passed         
      rac1                            libaio-0.3.107-9.fc12.i686      passed         
    Result: Package existence check passed for "libaio-0.3.106".
    Check: Package existence for "libaio-devel-0.3.106"
      Node Name                       Status                          Comment        
      rac2                            libaio-devel-0.3.107-9.fc12.i686  passed         
      rac1                            libaio-devel-0.3.107-9.fc12.i686  passed         
    Result: Package existence check passed for "libaio-devel-0.3.106".
    Check: Package existence for "libstdc++-4.1.1"
      Node Name                       Status                          Comment        
      rac2                            libstdc++-4.4.2-7.fc12.i686     passed         
      rac1                            libstdc++-4.4.2-7.fc12.i686     passed         
    Result: Package existence check passed for "libstdc++-4.1.1".
    Check: Package existence for "elfutils-libelf-devel-0.125"
      Node Name                       Status                          Comment        
      rac2                            elfutils-libelf-devel-0.143-1.fc12.i686  passed         
      rac1                            elfutils-libelf-devel-0.143-1.fc12.i686  passed         
    Result: Package existence check passed for "elfutils-libelf-devel-0.125".
    Check: Package existence for "sysstat-7.0.0"
      Node Name                       Status                          Comment        
      rac2                            sysstat-9.0.4-4.fc12.i686       passed         
      rac1                            sysstat-9.0.4-4.fc12.i686       passed         
    Result: Package existence check passed for "sysstat-7.0.0".
    Check: Package existence for "compat-libstdc++-33-3.2.3"
      Node Name                       Status                          Comment        
      rac2                            compat-libstdc++-33-3.2.3-68.i686  passed         
      rac1                            compat-libstdc++-33-3.2.3-68.i686  passed         
    Result: Package existence check passed for "compat-libstdc++-33-3.2.3".
    Check: Package existence for "libgcc-4.1.1"
      Node Name                       Status                          Comment        
      rac2                            libgcc-4.4.2-7.fc12.i686        passed         
      rac1                            libgcc-4.4.2-7.fc12.i686        passed         
    Result: Package existence check passed for "libgcc-4.1.1".
    Check: Package existence for "libstdc++-devel-4.1.1"
      Node Name                       Status                          Comment        
      rac2                            libstdc++-devel-4.4.2-7.fc12.i686  passed         
      rac1                            libstdc++-devel-4.4.2-7.fc12.i686  passed         
    Result: Package existence check passed for "libstdc++-devel-4.1.1".
    Check: Package existence for "unixODBC-2.2.11"
      Node Name                       Status                          Comment        
      rac2                            unixODBC-2.2.14-6.fc12.i686     passed         
      rac1                            unixODBC-2.2.14-9.fc12.i686     passed         
    Result: Package existence check passed for "unixODBC-2.2.11".
    Check: Package existence for "unixODBC-devel-2.2.11"
      Node Name                       Status                          Comment        
      rac2                            unixODBC-devel-2.2.14-6.fc12.i686  passed         
      rac1                            unixODBC-devel-2.2.14-9.fc12.i686  passed         
    Result: Package existence check passed for "unixODBC-devel-2.2.11".
    Check: Package existence for "glibc-2.5-12"
      Node Name                       Status                          Comment        
      rac2                            glibc-2.11-2.i686               passed         
      rac1                            glibc-2.11-2.i686               passed         
    Result: Package existence check passed for "glibc-2.5-12".
    Check: Group existence for "dba"
      Node Name     Status                    Comment                
      rac2          exists                    passed                 
      rac1          exists                    passed                 
    Result: Group existence check passed for "dba".
    Check: Group existence for "oinstall"
      Node Name     Status                    Comment                
      rac2          exists                    passed                 
      rac1          exists                    passed                 
    Result: Group existence check passed for "oinstall".
    Check: User existence for "nobody"
      Node Name     Status                    Comment                
      rac2          exists                    passed                 
      rac1          exists                    passed                 
    Result: User existence check passed for "nobody".
    System requirement passed for 'crs'
    Pre-check for cluster services setup was successful.  So now here is the actual problem:
    After the installation and during the run of the root.sh I get:
    Failure at final check of Oracle CRS stack.
    10
    [root@rac1 crs]# ./root.sh
    WARNING: directory '/u01/app/crs/product/11.1.0' is not owned by root
    WARNING: directory '/u01/app/crs/product' is not owned by root
    WARNING: directory '/u01/app/crs' is not owned by root
    WARNING: directory '/u01/app' is not owned by root
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up Network socket directories
    Oracle Cluster Registry configuration upgraded successfully
    The directory '/u01/app/crs/product/11.1.0' is not owned by root. Changing owner to root
    The directory '/u01/app/crs/product' is not owned by root. Changing owner to root
    The directory '/u01/app/crs' is not owned by root. Changing owner to root
    The directory '/u01/app' is not owned by root. Changing owner to root
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: rac1 rac1-priv rac1
    node 2: rac2 rac2-priv rac2
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    Now formatting voting device: /u01/shared_config/voting_disk
    Format of 1 voting devices complete.
    Startup will be queued to init within 30 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Failure at final check of Oracle CRS stack.
    10According to this link => http://blog.contractoracle.com/2009/01/failure-at-final-check-of-oracle-crs.html
    To recover from a status 10, one must check:
    check firewall / routing / iptables issues
    Now I have turned iptables off completely it doesnt even start up at boot time, so I know it can't be that.
    ROUTE
    [oracle@rac1 clusterware]$ route
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    192.168.2.0 * 255.255.255.0 U 1 0 0 eth0
    192.168.0.0 * 255.255.255.0 U 1 0 0 eth1
    default 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
    [oracle@rac2 ~]$ route
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    192.168.2.0 * 255.255.255.0 U 1 0 0 eth0
    192.168.0.0 * 255.255.255.0 U 1 0 0 eth2
    default 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
    [oracle@rac1 clusterware]$ traceroute rac2
    traceroute to rac2 (192.168.2.12), 30 hops max, 60 byte packets
    1 rac2.localdomain (192.168.2.12) 0.424 ms 0.427 ms 0.096 ms
    [oracle@rac1 clusterware]$ traceroute rac2-priv
    traceroute to rac2-priv (192.168.0.12), 30 hops max, 60 byte packets
    1 rac2-priv.localdomain (192.168.0.12) 1.336 ms 1.238 ms 1.188 ms
    [oracle@rac1 clusterware]$ traceroute rac2-vip
    traceroute to rac2-vip (192.168.2.112), 30 hops max, 60 byte packets
    1 rac1.localdomain (192.168.2.11) 2999.599 ms !H 2999.560 ms !H 2999.523 ms !H
    [oracle@rac1 bin]$ ./crs_stat -t
    CRS-0184: Cannot communicate with the CRS daemon.
    Both rac1 and rac2 get the same output above with the -vip getting !H => !H, !N, or !P (host, network or protocol unreachable), I am assuming this is normal as CRS install did not complete successfully and the virtual IP is not bound yet.
    Im pretty sure I have some kind of networking issue here, but I cant put my finger on it. I have tried absolutely everything that is suggested on the internet that I could find. Even deleting the /tmp/.oracle and /var/tmp/.oracle but nothing works. Ssh keys for root and oracle users exist and Ive connected using every possible combination to avoid that first time ssh prompt so users oracle on each node goes directly into rac1/rac2 rac1-priv/rac2-priv & actual IPs as well. Any ideas?
    Edited by: Javier on Dec 30, 2009 12:34 PM
    Edited by: Javier on Dec 30, 2009 6:58 PM

    Hello
    Note 370605.1 (Clusterware Intermittently Hangs And Commands Fail With CRS-184) is telling this.
    "This is caused by a cron job that cleans up the /tmp directory which also removes the Oracle socket files in /tmp/.oracle
    Do not remove /tmp/.oracle or /var/tmp/.oracle or its files while Oracle Clusterware is up."
    Best Regards...

  • While installing grid software in rac in vmware we got the below error, failed to start the Clusterware

    hi,
    this is malli,
    i got below error while installing grid software in rac.
    os versin redhat 11inux 5.5
    oracle 11gr2 cluster rac
    thanks in advance....please do needful
    Failed to start the Clusterware. Last 20 lines of the alert log follow:
    2014-12-24 01:05:40.100
    [client(3850)]CRS-2101:The OLR was formatted using version 3.
    2014-12-24 01:05:41.923
    [client(3852)]CRS-2317:Fatal error: cannot get local GPnP security keys (wallet).
    2014-12-24 01:05:41.923
    [client(3852)]CRS-2316:Fatal error: cannot initialize GPnP, CLSGPNP_ERR (Generic GPnP error).
    2014-12-24 01:05:41.954
    [client(3852)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid_home/log/lnx01/client/clscfg.log.

    [client(3852)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid_home/log/lnx01/client/clscfg.log.
    Have you looked in this log file to see why the OCR is inaccessible? Have you looked in the ASM instance's alert log for other clues?
    Cheers,
    Brian

  • Multiple Oracle Homes - Oracle Listener failes to start after installation

    Just in case the listener fails to start after an oracle installation, please check the oracle ports in the listener.ora and tnsnames.ora, both of which are present in the following directory:
    ...\oracle\<SID>\<Ver>\NETWORK\ADMIN
    All installations should have separate listener ports eg. If the first installation has port 1527, the next one should be 1528 and so on..
    If they do not, manually change the ports and start the listeners.
    Also check SAP Note 98252. Although I didn't find it necessary (I think it applies only if you have a single listener for all oracle homes), it did give me an idea that there some config problem in the .ora files.
    Also see [this thread|Oracle Listener error in Import ABAP phase?; for related details

    hi
    thanks for the help ,
    I have been using the SQL Server , i am totally blank about Oracle
    i am totally confused with this user, and hoststring
    i logged in to the enterprise manager using System and manager password
    But i cannot connect to any database using SQL Server
    i am not remembering any user name or password i have given during installation.
    the only thing i remember is
    Global databasename = (globedb) i have given
    For SID = globedb the same name i have given
    for sys ,i have given sysdb as password
    and for system i have given systemdb as password
    with this can i do anything
    when expand the schemas ,i can see the XDB,SCOTT and SYS . how can i login to this .
    if i want to login to the scott
    what would be the username,password,and hoststring. since i haven't set any password and username ,there would be some default username ,password, and hoststring.
    if any one can help me please help me
    thanks regards

  • Oracle 9i Report Server Failed to start

    Hi,
    My Oracle 9i Report Server Failed to start, I think it is due to patch problem with OS
    It was workin fine before we upgraded patch to Sun Solaris Machine.
    Now the Patch set In my OS is
    The patched installed are as follows
    112138-01 Synopsis: SunOS 5.8:: usr/bin/domainname patch
    108652-98 X11 6.4.1: Xsun patch
    108921-25 CDE 1.4: dtwm patch
    108940-72 Motif 1.2.7 and 2.1.1: Runtime library patch for Solaris 8
    OS: SunOS 5.8 Generic_117350-37 sun4u sparc SUNW, Ultra-4
    When i try to start the application Server it shows the below error pls Help.
    ......Starting Oracle HTTP Server.....
    ADMN-100999
    ......Started Oracle HTTP Server.....
    ......Started OC4J instance.....
    ADMN-100999
    ......Started OC4J instance.....
    .....Checking OHS and OC4J status.....
    ADMN-100999
    See base exception for details.
    Base Exception:
    oracle.ias.repository.schema.SchemaException:Unable to connect to Directory Server:javax.naming.CommunicationException: it15242.chennai.ford.com:4032 [Root exception is java.net.ConnectException: Connection refused]
    See base exception for details.
    .....PRESS ENTER TO CONTINUE.......
    ......Starting Web Cache...........
    Oracle9iAS Web Cache, Version 9.0.2.0.0
    Copyright: Oracle Corporation, 1999-2001
    Build CALYPSO_9.0.0_SOLARIS_020411
    Admin Server failed to start
    ......Started Web Cache...........
    .....PRESS ENTER TO EXIT.......
    Please Help me out to solve this issue.

    I am assuming you are using 9iR2:
    1. start your infrastructure database
    2. start oid
    nohup oidmon start
    nohup oidctl server=oidldapd configset=0 instance=1 start
    3. check to see if things seem to be working properly in the infrastructure before starting the midtier
    dcmctl getstate -v (this will give you the status of the infrastructure components). If this comes back in error, you will need to solve this first.

  • Oracle Service Bus 11g (11.1.1.3.0) failing to start

    I have installed Oracle Weblogic 10.3.3 (11g with Patch Set2) on a Windows XP SP2 32-bit machine. I was able to configure and start the AdminServer without any issues. Next, I installed Oracle Service Bus 11g (11.1.1.3.0) on top of WLS 10.3.3 (11g) on the same machine. I extended the previous weblogic domain (AdminServer only) to include Oracle Service Bus (OSB) features. OSB server was configured successfully.
    However, the AdminServer and OSB server is now failing to start successfully. It is failing to deploy and load the OSB applications (e.g. - OSB Console, File Adapter, SB Transport Adapter etc. ). While configuring the OSB domain, OSB application got targeted to both AdminServer and OSB server. Have you encountered this issue before? I tried to start the managed server from command prompt and Admin console

    Few links which you may refer -
    auto start weblogic 11gR1 as windows service?
    http://blogs.oracle.com/jamesbayer/2010/01/weblogic_nodemanager_quick_sta.html
    http://blogs.oracle.com/middleware/2010/09/startup_scripts_for_weblogic_and_soa_suite.html
    Regards,
    Anuj

  • Oracle Process Manager failed to start(11.1.2)

    Hi all!
    I've installed and configured 11.1.2 on my VM(Win 2003 SP2)
    Then i've been trying to start system but during launching Oracle Process manager I get the following error
    Could not start Oracle Process Manage service(ohsInstance3250625901) on local computer
    Error 1053: The service did not respond to the start request in a timely fashion
    Then i checked Event log
    The Oracle Process Manager (EPM_epmsystem1) service failed to start due to the following error:
    Timeout (30000 milliseconds) waiting for the Oracle Process Manager (EPM_epmsystem1) service to connect.
    The Oracle Process Manager (ohsInstance3250625901) service failed to start due to the following error:
    The service did not respond to the start or control request in a timely fashion.
    Please help?
    Regards
    Alexander
    Edited by: Softperson on 9/8/2010 15:36

    Hi Jasmine
    I've found a lot of errors in event viewer
    Some of them bellow
    "The Oracle Process Manager (EPM_epmsystem1) service failed to start due to the following error:
    Timeout (30000 milliseconds) waiting for the Oracle Process Manager (EPM_epmsystem1) service to connect.
    The Oracle Process Manager (ohsInstance3250625901) service failed to start due to the following error:
    The service did not respond to the start or control request in a timely fashion."
    Regards
    Alexander

  • Installing Oracle Application Server 10.1.3.1 - failed to start opmnctl

    I received the following error when trying to install Oracle Application Server 10.1.3.1 on Linux Redhat 5.2:
    failed to start a managed process after maximum retry limit
    Log: /as/10.1.3.1/OracleAS_1/opmn/logs//HTTP_Server~1.log
    In the HTTP_Server~1.log:
    Start Process
    /as/10.1.3.1/OracleAS_1/Apache/bin/apachectl startssl: execing httpd
    Please help! Thanks in advance.

    hi,
    First check the log files why http is not starting up thro opmnctl cmd.
    In order to change the port..
    Take the back up of Original httpd.conf file and opmn.xml
    and then in $ORACLE_HOME/Apache/Apache/conf/httpd.conf file
    you will get parameter "port" and "listen"..so enter the desired port no
    Regards
    Fabian

  • Oracle 10 isqlplussvc fails to start

    I am working on building a new ECC 6 EHP4 system on Windows Server 2008 64 bit. I am installing Oracle 10.2.0.4.0 and at the end of the installation I receive the following error:
    "Failed to start the service, Error: 0, the operation completed successfully."
    The service "OracleX20102iSQL*Plus" is always stuck in "starting" status. I ran a netstat -ab from the command line to make sure there are no other services running on port 5560 and there are not any. I did however see a lot of the following error:
    "windows sockets initialization failed 5"
    The only other thing to note is that this is being installed on a virtual machine. I have not had this issue when installing to physical machines. Are there any suggestions?

    Below is the error message that I receive from the sapinst_dev log.  The message shows up during step 17, Import ABAP.  I also included the errors for the particular imports that failed:
    Loading of 'PCL2' import package: ERROR
    Loading of 'PCL4' import package: ERROR
    Loading of 'AGR_HIERT' import package: ERROR
    Import Monitor jobs: running 0, waiting 1, completed 101, failed 3, total 105.
    TRACE      2010-12-01 21:39:17.915
    Function setMessageIdOfExceptionMessage: nw.programError
    INFO[E]    2010-12-01 21:39:17.915
               CJSlibModule::writeError_impl()
    CJS-30022  Program 'Migration Monitor' exits with error code 103. For details see log file(s) import_monitor.java.log, import_monitor.log.
    TRACE      2010-12-01 21:39:18.55 [iaxxejsbas.hpp:483]
               EJS_Base::dispatchFunctionCall()
    JS Callback has thrown unknown exception. Rethrowing.
    ERROR      2010-12-01 21:39:18.134 [sixxcstepexecute.cpp:950]
    FCO-00011  The step runMigrationMonitor with step key |NW_ABAP_OneHost|ind|ind|ind|ind|0|0|NW_Onehost_System|ind|ind|ind|ind|1|0|NW_CreateDBandLoad|ind|ind|ind|ind|10|0|NW_ABAP_Import_Dialog|ind|ind|ind|ind|5|0|NW_ABAP_Import|ind|ind|ind|ind|0|0|runMigrationMonitor was executed with status ERROR .
    TRACE      2010-12-01 21:39:18.181 [iaxxgenimp.cpp:752]
                CGuiEngineImp::showMessageBox
    ERROR      2010-12-01 21:39:18.134 [sixxcstepexecute.cpp:950]
    FCO-00011  The step runMigrationMonitor with step key |NW_ABAP_OneHost|ind|ind|ind|ind|0|0|NW_Onehost_System|ind|ind|ind|ind|1|0|NW_CreateDBandLoad|ind|ind|ind|ind|10|0|NW_ABAP_Import_Dialog|ind|ind|ind|ind|5|0|NW_ABAP_Import|ind|ind|ind|ind|0|0|runMigrationMonitor was executed with status ERROR .
    <html> <head> </head> <body> <p> An error occurred while processing option Enhancement Package 4 for SAP ERP 6.0 > SAP Application Server ABAP > Oracle > Central System > Central System. You can now: </p> <ul> <li> Choose <i>Retry</i> to repeat the current step. </li> <li> Choose <i>View Log</i> to get more information about the error. </li> <li> Stop the option and continue with it later. </li> </ul> <p> Log files are written to C:/Program Files/sapinst_instdir/ERPEhP4/AS-ABAP/ORA/CENTRAL. </p> </body></html>
    TRACE      2010-12-01 21:39:18.181 [iaxxgenimp.cpp:1255]
               CGuiEngineImp::acceptAnswerForBlockingRequest
    Waiting for an answer from GUI
    Edited by: Jeff Kinard on Dec 2, 2010 12:04 AM

  • OPMN Process Manger failed to start during Oracle 10gAS installation

    Hi,
    I am installing OracleAS 10.1.2.0.2 on AIX 5.3.
    I am facing this problem while installing infrastructure
    services.
    I am facing an issue like
    "OPMN Process Manger failed to start"
    Please help me to solve this.
    regards,
    Mat

    Hi,
    Thanks for the replay. I am doing the installation once again. Now I am not facing OPMN problem.
    Now I am facing some another problem. That DBCA fails to create infrastructre metadata respository database. Why it is happening?
    regards,
    Mat.

  • TCP Extend Server - Failed to start Service - Oracle Coherence GE 3.5.2/463

    Hello,
    We are about to go to production I see Failed to start Service in TCP Extend Server (Storage disable node).
    Regards
    /AG
    My Configuration look like the following
    <?xml version="1.0" encoding="windows-1252" ?>
    <!DOCTYPE cache-config SYSTEM "cache-config.dtd">
    <cache-config>
    <caching-scheme-mapping>
    </caching-scheme-mapping>
    <caching-schemes>
    <distributed-scheme>
    <scheme-name>distributedCache</scheme-name>
    <service-name>distributedCache</service-name>
    <backing-map-scheme>
    <local-scheme>
    <eviction-policy>HYBRID</eviction-policy>
    <high-units>500</high-units>
    <low-units>375</low-units>
    <unit-calculator>BINARY</unit-calculator>
    <unit-factor>1048576</unit-factor>
    </local-scheme>
    </backing-map-scheme>
    </distributed-scheme>
    <proxy-scheme>
    <service-name>ExtendTcpProxyService</service-name>
    <thread-count>15</thread-count>
    <acceptor-config>
    <tcp-acceptor>
    <local-address>
    <address system-property="proxy.listen.address">....</address>
    <port system-property="proxy.listen.port">....</port>
    </local-address>
    </tcp-acceptor>
    </acceptor-config>
    <autostart>true</autostart>
    </proxy-scheme>
    </caching-schemes>
    </cache-config>
    ------------------------------------------------------------------------- And the Log looks like the following ------------------------------------
    2009-12-04 16:21:54.056/25821.278 Oracle Coherence GE 3.5.2/463 <D6> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=12): Closed: Channel(Id=193159068
    6, Open=false)
    2009-12-04 16:21:54.058/25821.280 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:4, member=12): Repeating SizeReques
    t due to the re-distribution of PartitionSet{220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242
    , 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256}
    2009-12-04 16:21:54.058/25821.280 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:12, member=12): Repeating SizeReque
    st due to the re-distribution of PartitionSet{220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 24
    2, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256}
    2009-12-04 16:21:54.058/25821.280 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:11, member=12): Repeating SizeReque
    st due to the re-distribution of PartitionSet{220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 24
    2, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256}
    2009-12-04 16:21:54.058/25821.280 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:7, member=12): Repeating SizeReques
    t due to the re-distribution of PartitionSet{220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242
    , 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256}
    2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:10, member=12): An exception occurr
    ed while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:107)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
    at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
    at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
    at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
    at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
    at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
    at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
    at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
    at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
    at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
    at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
    at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:96)
    ... 18 more
    2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:9, member=12): An exception occurre
    d while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:107)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
    at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
    at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
    at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
    at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
    at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
    at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
    at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
    at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
    at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
    at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
    at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:96)
    ... 18 more
    2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:13, member=12): An exception occurr
    ed while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:107)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
    at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
    at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
    at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
    at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
    at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
    at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
    at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
    at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
    at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
    at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
    at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:96)
    ... 18 more
    2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:11, member=12): An exception occurr
    ed while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:107)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
    at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
    at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
    at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
    at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
    at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
    at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
    at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
    at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
    at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
    at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
    at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:96)
    ... 18 more
    2009-12-04 16:21:54.175/25821.397 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:2, member=12): An exception occurre
    d while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:107)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureRequestTarget(DistributedCache.CDB:15)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.sendPartitionedRequest(DistributedCache.CDB:31)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.size(DistributedCache.CDB:13)
    at com.tangosol.util.ConverterCollections$ConverterMap.size(ConverterCollections.java:1470)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.size(DistributedCache.CDB:1)
    at com.tangosol.coherence.component.util.SafeNamedCache.size(SafeNamedCache.CDB:1)
    at com.tangosol.coherence.component.util.collections.WrapperMap.size(WrapperMap.CDB:1)
    at com.tangosol.coherence.component.net.extend.messageFactory.NamedCacheFactory$SizeRequest.onRun(NamedCacheFactory.CDB:7)
    at com.tangosol.coherence.component.net.extend.message.Request.run(Request.CDB:4)
    at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.onMessage(NamedCacheProxy.CDB:11)
    at com.tangosol.coherence.component.net.extend.Channel.execute(Channel.CDB:28)
    at com.tangosol.coherence.component.net.extend.Channel.receive(Channel.CDB:26)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer$DaemonPool$WrapperTask.run(Peer.CDB:9)
    at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
    at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
    at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:96)
    ... 18 more
    2009-12-04 16:21:54.176/25821.398 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:6, member=12): An exception occurre
    d while processing a SizeRequest for Service=Proxy:ExtendTcpProxyService:TcpAcceptor: (Wrapped) java.lang.InterruptedException
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:293)
    at com.tangosol.util.Base.ensureRuntimeException(Base.java:269)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForPartitionRedistribution(DistributedCache
    .CDB:107)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.waitForRedistribution(DistributedCache.CDB:34)
    :$
    2009-12-04 16:21:54.259/25821.481 Oracle Coherence GE 3.5.2/463 <D4> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:8, member=12): Daemon caught an unh
    andled exception (com.tangosol.net.messaging.ConnectionException: channel is closed) while exiting.
    2009-12-04 16:21:54.264/25821.486 Oracle Coherence GE 3.5.2/463 <D4> (thread=Proxy:ExtendTcpProxyService:TcpAcceptorWorker:3, member=12): Daemon caught an unh
    andled exception (com.tangosol.net.messaging.ConnectionException: channel is closed) while exiting.
    2009-12-04 16:21:54.330/25821.552 Oracle Coherence GE 3.5.2/463 <D5> (thread=Proxy:ExtendTcpProxyService:TcpAcceptor, member=12): Stopped: TcpAcceptor{Name=Pr
    oxy:ExtendTcpProxyService:TcpAcceptor, State=(SERVICE_STOPPED), ThreadCount=0, Codec=Codec(Format=POF), PingInterval=0, PingTimeout=0, RequestTimeout=0, Local
    Address=[nybc94lxb01/10.12.101.81:21005], LocalAddressReusable=false, KeepAliveEnabled=true, TcpDelayEnabled=false, ReceiveBufferSize=0, SendBufferSize=0, Lis
    tenBacklog=0, LingerTimeout=-1, BufferPoolIn=BufferPool(BufferSize=2KB, BufferType=DIRECT, Capacity=Unlimited), BufferPoolOut=BufferPool(BufferSize=2KB, Buffe
    rType=DIRECT, Capacity=Unlimited)}
    Exception in thread "Thread-2" java.lang.RuntimeException: Failed to start Service "Proxy:ExtendTcpProxyService:TcpAcceptor" (ServiceState=SERVICE_STOPPED)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
    at com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:10)
    at com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.closeChannel(Peer.CDB:18)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.closeChannel(Peer.CDB:1)
    at com.tangosol.coherence.component.net.extend.Channel.close(Channel.CDB:20)
    at com.tangosol.coherence.component.net.extend.Channel.close(Channel.CDB:1)
    at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.run(NamedCacheProxy.CDB:30)
    at java.lang.Thread.run(Thread.java:619)
    Exception in thread "Thread-3" java.lang.RuntimeException: Failed to start Service "Proxy:ExtendTcpProxyService:TcpAcceptor" (ServiceState=SERVICE_STOPPED)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
    at com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:10)
    at com.tangosol.coherence.component.net.extend.Channel.request(Channel.CDB:1)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.closeChannel(Peer.CDB:18)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Peer.closeChannel(Peer.CDB:1)
    at com.tangosol.coherence.component.net.extend.Channel.close(Channel.CDB:20)
    at com.tangosol.coherence.component.net.extend.Channel.close(Channel.CDB:1)
    at com.tangosol.coherence.component.net.extend.proxy.NamedCacheProxy.run(NamedCacheProxy.CDB:30)
    at java.lang.Thread.run(Thread.java:619)
    Edited by: Anand Gupta on Dec 4, 2009 5:10 PM

    David,
    Thanks for your detailed response. I will try all that you suggested and let you know my observation at the end. To give a back ground of the application I am working on.
    Background
    Try to replace home grown cache (very well optimized uses custom serialization using NIO bytebuffer etc.,) my mandate is to replace the legacy cache with coherence (I love coherence product (at least 3.4) from my previous job experience). Since all kind of optimization is done in legacy application I am converting NIO byteBuffer generated by legacy serialization to byteArray and converting ByteArray into POF. Also for various other reasons, I have to send entry processor to do even get and put data from the cluster.
    Raw Data Set Size
    My data size is just 500MB (In Production). I put enough cache nodes so that raw data is just 50MB per node. (Since it is compute heavy with entry processor and the number of clients are going to be ~2500)
    What was happening when I got the exception?
    I was doing stress testing of the cache. Not sure if that put lot of garbage in some of the node hence may be the garbage collection pause triggered the re-distribution of partition set. Since the stress testing was on the way it might had cascading effect on all the nodes.
    Datagram Test result.
    Tx summary 4 peers:
    life: 96 MB/sec, 68774 packets/sec
    now: 100 MB/sec, 71378 packets/sec, packets/burst: 1029, bursts/second: 69.41594
    Success rate ranges from 0.87 to 1.0
    Is a success rate less than 0.98 a cause for concern?
    Regarding Size (to keep the extend client always connected to the proxy)
    I do size request every 5 sec. if there is not other request has gone to the cluster in the past 5 sec.
    In one of the oracle presentation it was said that "If you have map listener's and no other request from a long period of time extend client has to do a periodic size request to make sure that connection to proxy is alive"
    In this regard my question is "Can the same effect achieved by doing member listener on the extend client and on disconnection do the size request?"
    Proxy Size guide lines
    In production number of extend clients are going to be ~2500 connected all the time.
    Of which 2000 clients with few map listener + entry processor gets (Each of the clients will have distinct and disjoint set of data on which they work and listener)
    And the remaining ~500 clients will do all kinds of request on the entire data set.
    In this circumstance is there any recommendation for Number of extend-client : Number of extend-server : Number of threads ratio?
    Regards
    /Anand

Maybe you are looking for

  • Satellite P300-1CN - How to get replacemant?

    Hi, About two weeks ago I have started a thread on this forum. [http://forums.computers.toshiba-europe.com/forums/thread.jspa?threadID=56949&tstart=15] Below the main post: +My laptop started switching itself off every time I put it away for a couple

  • Ziio: Composite video out?

    In the initial specsheets released for the Ziio 7" a composite video out was mentioned. It seems that the current specs does not mention this? Is composite video out present on current Ziio's? /lars

  • Posting Date in MB5S

    Hello gurus, Does anyone knows how I can add posting date to MM reprt MB5S?

  • Filenames are numerical code- need last four #'s to be the first four #'s

    How do I switch the .jpg of numbers to the beginning of filenames? I have several hundred files whose filenames have three sets of numbers separated by underscores. Unfortunately, the last four numbers need to be placed at the beginning of the filena

  • ITune Purchases did not import over to iTune 7.2

    Just updated to iTune 7.2 and all library items imported OK, however, none of my purchased movies imported, says that I am not authorized to play on my iPod? What gives with this? PLEASE HELP