Parallel queries are failing in 8 node RAC DB
While running queries with parallel hints , the queries are failing with
ORA-12805 parallel query server died unexpectedly
Upon checking the alert logs, I couldnt find any thing about ORA-12805, But the i find this error: Please help me to fix this problem
Fatal NI connect error 12537, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for Linux: Version 11.1.0.7.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.1.0.7.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.1.0.7.0 - Production
Time: 15-MAY-2012 16:49:15
Tracing not turned on.
Tns error struct:
ns main err code: 12537
TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
ORA-609 : opiodr aborting process unknown ospid (18807_47295439087424)
Tue May 15 16:49:16 2012
A couple of thoughts come immediately to mind:
1. When I read ... "Tracing not turned on" ... I wonder to myself ... why not turn on tracing?
2. When I read ... "Version 11.1.0.7.0" ... I wonder to myself ... why not apply all of the patches Oracle has created in the last 3 years and see if having a fully patched version addresses the issue?
3. When I read ... "parallel query server died" ... I wonder whether you have gone to support.oracle.com and looked up the causes and solutions for Parallel Query Server dying?
Of course I also wonder why you have an 8 node cluster as that is adding substantial complexity and which leads me to wonder ... "is it happening on only one node or all nodes?"
Hope this helps.
Similar Messages
-
PCS calls are failing through "SendToVRU" node.
Hi All
we are using icm enabled PCS and getting problem that calls are failing through the send to VRU node.
Actually we are getting Handoff.tcl errors on our monitoring system. From CVP error log we found that those errors are against PCS calls only and from ICM script we found the above.For reference i am attaching screenshot.
Kindly help if anyone faced the same issue and got the solution.Not much info is provided, but are the calls which failing belong to a particular language? check below mentioned link and see if it applies for you.
https://tools.cisco.com/bugsearch/bug/CSCtk13452/?referring_site=ss -
Parallel queries are suspending
A procedure start two concurrent jobs. Each of them runs identical queries but the name of partition of destination table and bind variables defining partitions of source tables.
On schema A(test) the procedure works fine for a few minutes, on schema B(business) queries are suspended.
Any suggestions greatly appreciated.
Query generated by job:
BEGIN
MERGE /*+ APPEND USE_HASH(s d)*/ INTO D_SERVICE_FACT_BUF ::PART_DEF d
USING
SELECT /*+ USE_NL( TSRF DSUB ) USE_NL( TSRF NTRR ) USE_NL( TSRF NBLP ) USE_NL( TSRF NTKG1 ) USE_NL( TSRF NTKG2 ) */
COL1,
COL2,
FROM
<SCHEMA_NAME>.T_SERVICE_FACT TSRF,
<SCHEMA_NAME>.D_SUBSCRIPTION_ACT DSUB,
<SCHEMA_NAME>.N_TARIFF_RULE_ON_CALLS NTRR,
<SCHEMA_NAME>.N_BILLING_PROCEDURE NBLP,
<SCHEMA_NAME>.N_TRUNKGROUP NTKG1,
<SCHEMA_NAME>.N_TRUNKGROUP NTKG2,
<SCHEMA_NAME>.N_NUMZONE NNMZ
WHERE
1 = 1 AND
TSRF.SRC = DSUB.SRC AND
TSRF.SRC = NTRR.SRC(+) AND
TSRF.SRC = NBLP.SRC(+) AND
TSRF.SRC = NNMZ.SRC(+) AND
TSRF.SRF_RF_SUBSCRIPTION = DSUB.SUB_ID AND
TSRF.SRF_RF_TARIFF_RULE = NTRR.TRR_ID(+) AND
TSRF.SRF_DT_START BETWEEN DSUB.FD AND DSUB.TD AND
TSRF.SRF_DT_START BETWEEN NTRR.FD(+) AND NTRR.TD(+) AND
TSRF.SRF_DT_START BETWEEN NTKG1.FD(+) AND NTKG1.TD(+) AND
TSRF.SRF_DT_START BETWEEN NTKG2.FD(+) AND NTKG2.TD(+) AND
TSRF.SRF_DT_START BETWEEN NNMZ.FD(+) AND NNMZ.TD(+) AND
TSRF.SRF_RF_BILLING_PROC = NBLP.BLP_ID(+) AND
TSRF.SRF_RF_TRUNKGROUP_IN = NTKG1.TKG_ID(+) AND
TSRF.SRF_RF_TRUNKGROUP_OUT = NTKG2.TKG_ID(+) AND
TSRF.SRF_RF_NUMZONE = NNMZ.NMZ_ID(+) AND
:LDN = :LDN AND
:LDN_UP = :LDN_UP AND
TSRF.HASH_KEY = :SRF_HK AND
1 = 1 ) s
ON ( 1 = 0 )
WHEN MATCHED THEN UPDATE SET
d.src = 0
WHEN NOT MATCHED THEN INSERT
<DST_COLS_LIST>
VALUES
<RES_COL_LIST>
END;Hi,
Suspending means are you getting error after some time if yes (then what is the error ), or it just hang and does not give you output, then please post the wait event through V$session_wait or session trace.
Regards
Anurag Tibrewal -
Multi-table INSERT with PARALLEL hint on 2 node RAC
Multi-table INSERT statement with parallelism set to 5, works fine and spawns multiple parallel
servers to execute. Its just that it sticks on to only one instance of a 2 node RAC. The code I
used is what is given below.
create table t1 ( x int );
create table t2 ( x int );
insert /*+ APPEND parallel(t1,5) parallel (t2,5) */
when (dummy='X') then into t1(x) values (y)
when (dummy='Y') then into t2(x) values (y)
select dummy, 1 y from dual;
I can see multiple sessions using the below query, but on only one instance only. This happens not
only for the above statement but also for a statement where real time table(as in table with more
than 20 million records) are used.
select p.server_name,ps.sid,ps.qcsid,ps.inst_id,ps.qcinst_id,degree,req_degree,
sql.sql_text
from Gv$px_process p, Gv$sql sql, Gv$session s , gv$px_session ps
WHERE p.sid = s.sid
and p.serial# = s.serial#
and p.sid = ps.sid
and p.serial# = ps.serial#
and s.sql_address = sql.address
and s.sql_hash_value = sql.hash_value
and qcsid=945
Won't parallel servers be spawned across instances for multi-table insert with parallelism on RAC?
Thanks,
MaheshPlease take a look at these 2 articles below
http://christianbilien.wordpress.com/2007/09/12/strategies-for-rac-inter-instance-parallelized-queries-part-12/
http://christianbilien.wordpress.com/2007/09/14/strategies-for-parallelized-queries-across-rac-instances-part-22/
thanks
http://swervedba.wordpress.com -
All connections are connecting to 2nd node only in a 2 Node RAC Cluster
Hello,
I have a 10.2.0.3 database on a two node RAC Cluster with only one service configured. This service set to be preferred on both nodes.
However, all the connections are falling on Node2 only. Any idea where to look.
$> srvctl config service -d PSDB
psdbsrv1 PREF: psdb1 psdb2 AVAIL:
Thanks,
MMApplication is using the following connection string.
jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = PQ2-PS-db-01-vip)(PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = PQ2-PS-db-02-vip)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PSDBSRV1)(FAILOVER_MODE =(TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5))))
--MM -
How to detect sessions that are currently running parallel queries?
Hi everyone,
How to detect session that are currently running parallel queries?
- The only way i can think of is querying pdml_Status from gv$session?
- Is there a better way to do this?
Follow up question:
After detecting sessions that are running parallel queries how do i identify which sessions are slaves of which session?
thanks!Start with V$PX_SESSION, however also take a look at V$PQ_* and V$PX_* tables.
-
UCCE calls are failing at the RunExternalScript node
Dear members,
It is a UCCE Lab environment with CVP version 9.0
My issue is that all the calls are failing at the RunExternalScript node.
In the CVP logs, the below error appears.
External VXML located at http://192.168.2.215:7000/CVP/en-us/app/Server?_dnis=3000&application=Test&callid=565&_ani=565 resulted in a bad fetch
Attached are the ICM script, CVP script, CVP logs and rtr logs.
Is there any variable that needs to be added in the ICM script?
Thank you in advance for your help.
LaraThank you Jameson for replying.
I was using the variable user.microappp.sys_media_lib instead of user.microapp.app_media_lib.
But even after I corrected this variable, I'm still facing a bad fetch error.
70: 192.168.2.215: Mar 09 2015 11:24:49.294 +0200: %CVP_9_0_IVR-3-CALL_ERROR: CALLGUID=0145638000010000000000421500A8C0 DNIS=012345678918 CVP VXML Server encountered a Bad-Fetch Error - URL: http://192.168.2.215:7000/CVP/en-us/../Server?_dnis=3000&application=Test&callid=565&_ani=565 (Client: 10.1.1.103) [id:3023]
71: 192.168.2.215: Mar 09 2015 11:24:49.294 +0200: %CVP_9_0_IVR-3-CALL_ERROR: RunScript Error from 10.1.1.103 [CVP_BAD_FETCH(45)] CALLGUID: 0145638000010000000000421500A8C0 DNIS=012345678918 {VRUScriptName: 'GS,Server,V' ConfigParam: ''} [id:3023]
108: 192.168.2.215: Mar 09 2015 11:24:49.310 +0200: %CVP_9_0_SIP-3-SIP_CALL_ERROR: CALLGUID = 0145638000010000000000421500A8C0 LEGID = 50331337-C57A11E4-BCB6D284-70B23AF8 - [INBOUND] - DIALOGUE_FAILURE from ICM Router sends 404 rejection to call. errorcode=15 [id:5004]
111: 192.168.2.215: Mar 09 2015 11:24:49.310 +0200: %CVP_9_0_SIP-3-SIP_CALL_ERROR: CALLGUID = 0145638000010000000000421500A8C0 LEGID = 50331337-C57A11E4-BCB6D284-70B23AF8 - [INBOUND] - ABNORMALLY ENDING - SIP code [404], Reason Hdr [SIP;cause=404] Not Found, GW call using SURV TCL flag [false], NON NORMAL flag [true], DNIS [3000], ANI [565] with AGE (msecs) 54334 and Call History : 012345678918|-1; [id:5004]
120: 192.168.2.215: Mar 09 2015 11:24:50.152 +0200: %CVP_9_0_SIP-3-SIP_CALL_ERROR: CALLGUID = 0145638000010000000000421500A8C0 LEGID = 50331337-C57A11E4-BCB6D284-70B23AF8 - [INBOUND]: Refer failed with 503 - Service Unavailable. May be a problem with Routing Configuration or Gateway Dial-Peer. [id:5004]
122: 192.168.2.215: Mar 09 2015 11:24:50.152 +0200: %CVP_9_0_SIP-3-SIP_CALL_ERROR: CALLGUID = 0145638000010000000000421500A8C0 LEGID = 50331337-C57A11E4-BCB6D284-70B23AF8 - [INBOUND] - ABNORMALLY ENDING - SIP code [404], Reason Hdr [SIP;cause=404] Not Found, GW call using SURV TCL flag [false], NON NORMAL flag [true], DNIS [3000], ANI [565] with AGE (msecs) 55176 and Call History : 012345678918|-1; [id:5004]
I can play the .wav file from the browser but the calls are still failing at the Run External Script node.
I restarted the CVP Call/VXML server but with no luck.
Your help is highly appreciated.
Thank you.
Lara -
Dbms_schduler job is not running on a 2 node rac when 1st node fails
Hi,
I want to create a dbms_scheduler job in a 2 node RAC and the job should always run on the node1 and if node1 is down then it should run on node2. This is Oracle 10gR2 (10.2.0.3 in WINDOWS) .In order to do the same I did following
-- First Step
Using DBCA- Service Managment - Created a service (BATCH_SERVICE) and given node1 as preferred and node2 as available. This created following entry in tnsnames.ora in both nodes.
BATCH_SERVICE =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = BATCH_SERVICE)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 180)
(DELAY = 5)
--- Step 2
-- Created BATCH job classes.
BEGIN
DBMS_SCHEDULER.create_job_class(
job_class_name => 'BATCH_JOB_CLASS',
service => 'BATCH_SERVICE');
END;
-- Step 3 -- created a job using job_class as BATCH_JOB_CLASS
begin
dbms_scheduler.create_job(
job_name => 'oltp_job_test'
,job_type => 'STORED_PROCEDURE'
,job_action => 'schema1.P1'
,start_date => systimestamp at time zone 'US/Central'
,repeat_interval => 'FREQ=DAILY;BYHOUR=11;BYMINUTE=30;'
,job_class => 'BATCH_JOB_CLASS'
,enabled => TRUE
,comments => 'New Job.');
end;
Now when I monitor this job it runs on node1. Now I started testing for failover. I manually shutdown 1st instance. Then as per my understanding job should run on 2nd node. But job is not picking up.
when I run the followign command
srvctl status service -d db -s BATCH_SERVICE
service BATCH_SERVICE is running on instance node2.
Any help is really appreciated.It does not show that whether job is running or broken.
-
FAILED_REQ# of PS enqueue is increasing(2 node RAC).
Hi, all.
The database is 2 node RAC database (10.2.0.2.0) on 32-bit windows 2003 EE SP1.
Our database is suffering database hang on a regular basis.
I checked v$enqueue_stat by using following sql.
select EQ_TYPE,TOTAL_REQ#,TOTAL_WAIT#,SUCC_REQ#,FAILED_REQ#,CUM_WAIT_TIME
from v$enqueue_stat
where failed_req# >0
order by eq_type,inst_id
● PS type enqueue
The value of FAILED_REQ# of node 2 is steady, not increasing.
However, the value of FAILED_REQ# of node 1 is gradually increasing.
I think it has some relation with PZnn process.
Sometimes, the database hangs caused by "CKPT hangs"
ORACLE.EXE (DM00)
ORACLE.EXE (q000)
ORACLE.EXE (q004)
expdp.exe
ORACLE.EXE (q002)
ORACLE.EXE (q003)
ORACLE.EXE (J000)
ORACLE.EXE (CKPT)
library cache lock
reliable message
library cache lock
library cache lock
enq: UL - contention
library cache lock
library cache lock
gc buffer busy
gc cr request
Could you give me any clue to solving this issue??
I am looking forwared to hearing your experience and advice.
Thanks and Regards.Did you attempt to generate an AWR/ADDM report to see if it gives more information on the performance problems you are facing. Enqueue type - PS simply means a "parallel operation lock". May be AWR will highlight the queries which are being impacted by this enqueue and you start from there to see how you can reduce the contention/locking/enqueue issues. Just a thought.
Thanks
Chandra -
Root.sh failed on second node while installing CRS 10g on centos 5.5
root.sh failed on second node while installing CRS 10g
Hi all,
I am able to install Oracle 10g RAC clusterware on first node of the cluster. However, when I run the root.sh script as root
user on second node of the cluster, it fails with following error message:
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Failure at final check of Oracle CRS stack.
10
and run cluvfy stage -post hwos -n all -verbose,it show message:
ERROR:
Could not find a suitable set of interfaces for VIPs.
Result: Node connectivity check failed.
Checking shared storage accessibility...
Disk Sharing Nodes (2 in count)
/dev/sda db2 db1
and run cluvfy stage -pre crsinst -n all -verbose,it show message:
ERROR:
Could not find a suitable set of interfaces for VIPs.
Result: Node connectivity check failed.
Checking system requirements for 'crs'...
No checks registered for this product.
and run cluvfy stage -post crsinst -n all -verbose,it show message:
Result: Node reachability check passed from node "DB2".
Result: User equivalence check passed for user "oracle".
Node Name CRS daemon CSS daemon EVM daemon
db2 no no no
db1 yes yes yes
Check: Health of CRS
Node Name CRS OK?
db1 unknown
Result: CRS health check failed.
check crsd.log and show message:
clsc_connect: (0x143ca610) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_db2_crs))
clsssInitNative: connect failed, rc 9
Any help would be greatly appreciated.
Edited by: 868121 on 2011-6-24 上午12:31Hello, it took a little searching, but I found this in a note in the GRID installation guide for Linux/UNIX:
Public IP addresses and virtual IP addresses must be in the same subnet.
In your case, you are using two different subnets for the VIPs. -
Error - convert single node-RAC-ConvertTomydb.xml -
my single node init.ora file:
# Copyright (c) 1991, 2001, 2002 by Oracle Corporation
# Cache and I/O
db_block_size=8192
db_file_multiblock_read_count=16
# Cursors and Library Cache
open_cursors=300
# Database Identification
db_domain=""
db_name=mydb
# Diagnostics and Statistics
background_dump_dest=/u01/app/oracle/admin/mydb/bdump
core_dump_dest=/u01/app/oracle/admin/mydb/cdump
user_dump_dest=/u01/app/oracle/admin/mydb/udump
# File Configuration
control_files=("/u01/app/oracle/oradata/mydb/control01.ctl", "/u01/app/oracle/oradata/mydb/control02.ctl", "/u01/app/oracle/oradata/mydb/control03.ctl")
# Job Queues
job_queue_processes=10
# Miscellaneous
compatible=10.2.0.1.0
# Processes and Sessions
processes=150
# SGA Memory
sga_target=1083179008
# Security and Auditing
audit_file_dest=/u01/app/oracle/admin/mydb/adump
remote_login_passwordfile=EXCLUSIVE
# Shared Server
dispatchers="(PROTOCOL=TCP) (SERVICE=mydbXDB)"
# Sort, Hash Joins, Bitmap Indexes
pga_aggregate_target=360710144
# System Managed Undo and Rollback Segments
undo_management=AUTO
undo_tablespace=UNDOTBS1
my ConvertTomydb.xml ------------which is copy of ConvertToRAC.xml file
<?xml version="1.0" encoding="UTF-8" ?>
- <n:RConfig xmlns:n="http://www.oracle.com/rconfig" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.oracle.com/rconfig">
- <n:ConvertToRAC>
- <!-- Verify does a precheck to ensure all pre-requisites are met, before the conversion is attempted. Allowable values are: YES|NO|ONLY
-->
- <n:Convert verify="ONLY">
- <!-- Specify current OracleHome of non-rac database for SourceDBHome
-->
<n:SourceDBHome>/u01/app/oracle/product/10.2.0/db_1</n:SourceDBHome>
- <!-- Specify OracleHome where the rac database should be configured. It can be same as SourceDBHome
-->
<n:TargetDBHome>/u01/app/oracle/product/10.2.0/db_1</n:TargetDBHome>
- <!-- Specify SID of non-rac database and credential. User with sysdba role is required to perform conversion
-->
- <n:SourceDBInfo SID="mydb">
- <n:Credentials>
<n:User>sys</n:User>
<n:Password>oracle</n:Password>
<n:Role>sysdba</n:Role>
</n:Credentials>
</n:SourceDBInfo>
- <!-- ASMInfo element is required only if the current non-rac database uses ASM Storage
-->
- <n:ASMInfo SID="+ASM1">
- <n:Credentials>
<n:User>sys</n:User>
<n:Password>oracle</n:Password>
<n:Role>sysdba</n:Role>
</n:Credentials>
</n:ASMInfo>
- <!-- Specify the list of nodes that should have rac instances running. LocalNode should be the first node in this nodelist.
-->
- <n:NodeList>
<n:Node name="linux1" />
<n:Node name="linux2" />
</n:NodeList>
- <!-- Specify prefix for rac instances. It can be same as the instance name for non-rac database or different. The instance number will be attached to this prefix.
-->
<n:InstancePrefix>mydb</n:InstancePrefix>
- <!-- Specify port for the listener to be configured for rac database.If port="", alistener existing on localhost will be used for rac database.The listener will be extended to all nodes in the nodelist
-->
<n:Listener port="1521" />
- <!-- Specify the type of storage to be used by rac database. Allowable values are CFS|ASM. The non-rac database should have same storage type.
-->
- <n:SharedStorage type="ASM">
- <!-- Specify Database Area Location to be configured for rac database.If this field is left empty, current storage will be used for rac database. For CFS, this field will have directory path.
-->
<n:TargetDatabaseArea>+ORCL_DATA1</n:TargetDatabaseArea>
- <!-- Specify Flash Recovery Area to be configured for rac database. If this field is left empty, current recovery area of non-rac database will be configured for rac database. If current database is not using recovery Area, the resulting rac database will not have a recovery area.
-->
<n:TargetFlashRecoveryArea>+FLASH_RECOVERY_AREA</n:TargetFlashRecoveryArea>
</n:SharedStorage>
</n:Convert>
</n:ConvertToRAC>
</n:RConfig>
Ran the xml file
$ rconfig ConvertTomydb.xml
Got this below error.
[oracle@linux1 bin]$ sh rconfig ConvertTomydb.xml
<?xml version="1.0" ?>
<RConfig>
<ConvertToRAC>
<Convert>
<Response>
<Result code="1" >
Operation Failed
</Result>
<ErrorDetails>
Clusterware is not configured
</ErrorDetails>
</Response>
</Convert>
</ConvertToRAC></RConfig>
[oracle@linux1 bin]$
Log file from /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/rconfig/rconfig.log
[main] [0:14:4:4] [RuntimeExec.runCommand:175] Returning from RunTimeExec.runCommand
oracle.ops.mgmt.cluster.RemoteShellException: PRKC-1044 : Failed to check remote command execution setup for node linux2 using shells /usr/bin/ssh and /usr/bin/rsh
linux2.com: Connection refused
at oracle.ops.mgmt.nativesystem.UnixSystem.checkRemoteExecutionSetup(UnixSystem.java:1880)
at oracle.ops.mgmt.nativesystem.UnixSystem.getRemoteShellCmd(UnixSystem.java:1634)
at oracle.ops.mgmt.nativesystem.UnixSystem.createCommand(UnixSystem.java:614)
at oracle.ops.mgmt.nativesystem.UnixSystem.removeFile(UnixSystem.java:622)
at oracle.ops.mgmt.nativesystem.UnixSystem.isSharedPath(UnixSystem.java:1352)
at oracle.ops.mgmt.cluster.Cluster.isSharedPath(Cluster.java:916)
at oracle.ops.mgmt.cluster.Cluster.isSharedPath(Cluster.java:859)
at oracle.sysman.assistants.util.ClusterUtils.areSharedPaths(ClusterUtils.java:570)
at oracle.sysman.assistants.util.ClusterUtils.isShared(ClusterUtils.java:501)
at oracle.sysman.assistants.util.ClusterUtils.isShared(ClusterUtils.java:457)
at oracle.sysman.assistants.util.attributes.CommonOPSAttributes.updateShared(CommonOPSAttributes.java:724)
at oracle.sysman.assistants.util.attributes.CommonOPSAttributes.setNodeNames(CommonOPSAttributes.java:207)
at oracle.sysman.assistants.rconfig.engine.Context.<init>(Context.java:54)
at oracle.sysman.assistants.rconfig.engine.ASMInstance.createUtilASMInstanceRAC(ASMInstance.java:109)
at oracle.sysman.assistants.rconfig.engine.Step.execute(Step.java:245)
at oracle.sysman.assistants.rconfig.engine.Request.execute(Request.java:73)
at oracle.sysman.assistants.rconfig.engine.RConfigEngine.execute(RConfigEngine.java:65)
at oracle.sysman.assistants.rconfig.RConfig.<init>(RConfig.java:85)
at oracle.sysman.assistants.rconfig.RConfig.<init>(RConfig.java:51)
at oracle.sysman.assistants.rconfig.RConfig.main(RConfig.java:130)
[main] [0:14:4:16] [UnixSystem.isSharedPath:1356] UnixSystem.isShared: creating file /u01/app/oracle/product/10.2.0/db_1/CFSFileName126249561289258204.tmp
[main] [0:14:4:17] [UnixSystem.checkRemoteExecutionSetup:1817] checkRemoteExecutionSetup:: Checking user equivalence using Secured Shell '/usr/bin/ssh'
[main] [0:14:4:17] [UnixSystem.checkRemoteExecutionSetup:1819] checkRemoteExecutionSetup:: Running Unix command: /usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=yes -o NumberOfPasswordPrompts=0 linux2 /bin/true
oracle.ops.mgmt.cluster.SharedDeviceException: PRKC-1044 : Failed to check remote command execution setup for node linux2 using shells /usr/bin/ssh and /usr/bin/rsh
linux2.com: Connection refused
at oracle.ops.mgmt.nativesystem.UnixSystem.testCFSFile(UnixSystem.java:1444)
at oracle.ops.mgmt.nativesystem.UnixSystem.isSharedPath(UnixSystem.java:1402)
at oracle.ops.mgmt.cluster.Cluster.isSharedPath(Cluster.java:916)
at oracle.ops.mgmt.cluster.Cluster.isSharedPath(Cluster.java:859)
at oracle.sysman.assistants.util.ClusterUtils.areSharedPaths(ClusterUtils.java:570)
at oracle.sysman.assistants.util.ClusterUtils.isShared(ClusterUtils.java:501)
at oracle.sysman.assistants.util.ClusterUtils.isShared(ClusterUtils.java:457)
at oracle.sysman.assistants.util.attributes.CommonOPSAttributes.updateShared(CommonOPSAttributes.java:724)
at oracle.sysman.assistants.util.attributes.CommonOPSAttributes.setNodeNames(CommonOPSAttributes.java:207)
at oracle.sysman.assistants.rconfig.engine.Context.<init>(Context.java:54)
at oracle.sysman.assistants.rconfig.engine.ASMInstance.createUtilASMInstanceRAC(ASMInstance.java:109)
at oracle.sysman.assistants.rconfig.engine.Step.execute(Step.java:245)
at oracle.sysman.assistants.rconfig.engine.Request.execute(Request.java:73)
at oracle.sysman.assistants.rconfig.engine.RConfigEngine.execute(RConfigEngine.java:65)
at oracle.sysman.assistants.rconfig.RConfig.<init>(RConfig.java:85)
at oracle.sysman.assistants.rconfig.RConfig.<init>(RConfig.java:51)
at oracle.sysman.assistants.rconfig.RConfig.main(RConfig.java:130)
[main] [0:14:35:152] [Version.isPre10i:189] isPre10i.java: Returning FALSE
[main] [0:14:35:152] [UnixSystem.getCSSConfigType:1985] configFile=/etc/oracle/ocr.loc
[main] [0:14:35:157] [Utils.getPropertyValue:221] keyName=ocrconfig_loc props.val=/u02/oradata/orcl/OCRFile propValue=/u02/oradata/orcl/OCRFile
[main] [0:14:35:157] [Utils.getPropertyValue:221] keyName=ocrmirrorconfig_loc props.val=/u02/oradata/orcl/OCRFile_mirror propValue=/u02/oradata/orcl/OCRFile_mirror
[main] [0:14:35:157] [Utils.getPropertyValue:292] propName=local_only propValue=FALSE
[main] [0:14:35:157] [UnixSystem.getCSSConfigType:2029] configType=false
[main] [0:14:35:158] [Version.isPre10i:189] isPre10i.java: Returning FALSE
[main] [0:14:35:168] [OCRTree.init:201] calling OCRTree.init
[main] [0:14:35:169] [Version.isPre10i:189] isPre10i.java: Returning FALSE
[main] [0:14:35:177] [OCRTree.<init>:157] calling OCR.init at level 7
[main] [0:14:35:177] [HASContext.getInstance:190] Module init : 24
[main] [0:14:35:177] [HASContext.getInstance:214] Local Module init : 0
[main] [0:14:35:177] [HASContext.getInstance:249] HAS Context Allocated: 4 to oracle.ops.mgmt.has.ClusterLock@f47bf5
[main] [0:14:35:177] [ClusterLock.<init>:60] ClusterLock Instance created.
[main] [0:14:35:178] [OCR.getKeyValue:411] OCR.getKeyValue(SYSTEM.local_only)
[main] [0:14:35:178] [nativesystem.OCRNative.Native] getKeyValue: procr_open_key retval = 0
[main] [0:14:35:179] [nativesystem.OCRNative.Native] getKeyValue: procr_get_value retval = 0, size = 6
[main] [0:14:35:179] [nativesystem.OCRNative.Native] getKeyValue: value is [false] dtype = 3
[main] [0:14:35:179] [OCRTreeHA.getLocalOnlyKeyValue:1697] OCRTreeHA localOnly string = false
[main] [0:14:35:180] [HASContext.getInstance:190] Module init : 6
[main] [0:14:35:180] [HASContext.getInstance:214] Local Module init : 0
[main] [0:14:35:180] [HASContext.getInstance:249] HAS Context Allocated: 5 to oracle.ops.mgmt.has.Util@f6438d
[main] [0:14:35:180] [Util.<init>:86] Util Instance created.
[main] [0:14:35:180] [has.UtilNative.Native] prsr_trace: Native: hasHAPrivilege
[main] [0:14:35:184] [HASContext.getInstance:190] Module init : 56
[main] [0:14:35:184] [HASContext.getInstance:214] Local Module init : 32
[main] [0:14:35:184] [has.HASContextNative.Native] prsr_trace: Native: allocHASContext
[main] [0:14:35:184] [has.HASContextNative.Native]
allocHASContext: Came in
[main] [0:14:35:184] [has.HASContextNative.Native] prsr_trace: Native: prsr_initCLSR
[main] [0:14:35:185] [has.HASContextNative.Native]
allocHASContext: CLSR context [1]
[main] [0:14:35:185] [has.HASContextNative.Native]
allocHASContext: retval [1]
[main] [0:14:35:185] [HASContext.getInstance:249] HAS Context Allocated: 6 to oracle.ops.mgmt.has.ClusterAlias@18825b3
[main] [0:14:35:185] [ClusterAlias.<init>:85] ClusterAlias Instance created.
[main] [0:14:35:185] [has.UtilNative.Native] prsr_trace: Native: getCRSHome
[main] [0:14:35:186] [has.UtilNative.Native] prsr_trace: Native: getCRSHome crs_home=/u01/app/oracle/product/10.2.0/crs(**)
[main] [0:14:35:280] [ASMTree.getASMInstanceOracleHome:1328] DATABASE.ASM.linux1.+asm1 does exist
[main] [0:14:35:280] [ASMTree.getASMInstanceOracleHome:1329] Acquiring shared CSS lock SRVM.ASM.DATABASE.ASM.linux1.+asm1
[main] [0:14:35:280] [has.ClusterLockNative.Native] prsr_trace: Native: acquireShared
[main] [0:14:35:281] [OCR.getKeyValue:411] OCR.getKeyValue(DATABASE.ASM.linux1.+asm1.ORACLE_HOME)
[main] [0:14:35:281] [nativesystem.OCRNative.Native] getKeyValue: procr_open_key retval = 0
[main] [0:14:35:282] [nativesystem.OCRNative.Native] getKeyValue: procr_get_value retval = 0, size = 36
[main] [0:14:35:282] [nativesystem.OCRNative.Native] getKeyValue: value is [u01/app/oracle/product/10.2.0/db_1] dtype = 3
[main] [0:14:35:282] [ASMTree.getASMInstanceOracleHome:1346] getASMInstanceOracleHome:ohome=/u01/app/oracle/product/10.2.0/db_1
[main] [0:14:35:282] [ASMTree.getASMInstanceOracleHome:1367] Releasing shared CSS lock SRVM.ASM.DATABASE.ASM.linux1.+asm1
[main] [0:14:35:282] [has.ClusterLockNative.Native] prsr_trace: Native: unlock
[main] [0:14:35:802] [nativesystem.OCRNative.Native] keyExists: procr_close_key retval = 0
[main] [0:14:35:802] [ASMTree.getNodes:1236] DATABASE.ASM does exist
[main] [0:14:35:802] [ASMTree.getNodes:1237] Acquiring shared CSS lock SRVM.ASM.DATABASE.ASM
[main] [0:14:35:802] [has.ClusterLockNative.Native] prsr_trace: Native: acquireShared
[main] [0:14:35:803] [OCR.listSubKeys:615] OCR.listSubKeys(DATABASE.ASM)
[main] [0:14:35:803] [nativesystem.OCRNative.Native] listSubKeys: key_name=[DATABASE.ASM]
[main] [0:14:35:809] [GetASMNodeListOperation.run:78] Got nodes=[Ljava.lang.String;@11a75a2
[main] [0:14:35:809] [GetASMNodeListOperation.run:91] result status 0
[main] [0:14:35:809] [LocalCommand.execute:56] LocalCommand.execute: Returned from run method
[main] [0:14:35:810] [ASMInstanceRAC.loadDiskGroups:2260] diskgroup: instName=+ASM2, diskGroupName=FLASH_RECOVERY_AREA, size=95378, freeSize=88454, type=EXTERN, state=MOUNTED
[main] [0:14:35:811] [ASMInstanceRAC.loadDiskGroups:2260] diskgroup: instName=+ASM1, diskGroupName=FLASH_RECOVERY_AREA, size=95378, freeSize=88454, type=EXTERN, state=MOUNTED
[main] [0:14:35:811] [ASMInstanceRAC.loadDiskGroups:2260] diskgroup: instName=+ASM2, diskGroupName=ORCL_DATA1, size=95384, freeSize=39480, type=NORMAL, state=MOUNTED
[main] [0:14:35:811] [ASMInstanceRAC.loadDiskGroups:2260] diskgroup: instName=+ASM1, diskGroupName=ORCL_DATA1, size=95384, freeSize=39480, type=NORMAL, state=MOUNTED
[main] [0:14:35:858] [ASMInstance.setBestDiskGroup:1422] sql to be executed:=select name from v$asm_diskgroup where free_mb= (select max(free_mb) from v$asm_diskgroup)
[main] [0:14:35:864] [ASMInstance.setBestDiskGroup:1426] Setting best diskgroup....
[main] [0:14:35:888] [SQLEngine.doSQLSubstitution:2165] The substituted sql statement:=select t1.name from v$asm_template t1, v$asm_diskgroup t2 where t1.group_number=t2.group_number and t2.name='FLASH_RECOVERY_AREA'
[main] [0:14:35:888] [ASMInstance.setTemplates:1345] sql to be executed:=select t1.name from v$asm_template t1, v$asm_diskgroup t2 where t1.group_number=t2.group_number and t2.name='FLASH_RECOVERY_AREA'
[main] [0:14:35:892] [ASMInstance.setTemplates:1349] Getting templates for diskgroup: oracle.sysman.assistants.util.asm.DiskGroup@170888e
[main] [0:14:35:892] [ASMInstance.setTemplates:1357] template: PARAMETERFILE
[main] [0:14:35:893] [ASMInstance.setTemplates:1357] template: DUMPSET
[main] [0:14:35:893] [ASMInstance.setTemplates:1357] template: DATAGUARDCONFIG
[main] [0:14:35:893] [ASMInstance.setTemplates:1357] template: FLASHBACK
[main] [0:14:35:893] [ASMInstance.setTemplates:1357] template: CHANGETRACKING
[main] [0:14:35:893] [ASMInstance.setTemplates:1357] template: XTRANSPORT
[main] [0:14:35:893] [ASMInstance.setTemplates:1357] template: AUTOBACKUP
[main] [0:14:35:893] [ASMInstance.setTemplates:1357] template: BACKUPSET
[main] [0:14:35:894] [ASMInstance.setTemplates:1357] template: TEMPFILE
[main] [0:14:35:894] [ASMInstance.setTemplates:1357] template: DATAFILE
[main] [0:14:35:894] [ASMInstance.setTemplates:1357] template: ONLINELOG
[main] [0:14:35:894] [ASMInstance.setTemplates:1357] template: ARCHIVELOG
[main] [0:14:35:894] [ASMInstance.setTemplates:1357] template: CONTROLFILE
[main] [0:14:35:894] [ASMInstance.createUtilASMInstanceRAC:113] Diskgroups loaded
[main] [0:14:35:894] [LocalNodeCheck.checkLocalNode:107] Performing LocalNodeCheck
[main] [0:14:35:894] [OracleHome.getNodeNames:270] inside getNodeNames
[main] [0:14:36:116] [OracleHome.isClusterInstalled:252] bClusterInstalled=false
[main] [0:14:36:120] [Step.execute:251] STEP Result=Clusterware is not configured
[main] [0:14:36:121] [Step.execute:280] Returning result:Operation Failed
[main] [0:14:36:121] [RConfigEngine.execute:67] bAsyncJob=false
[main] [0:14:36:124] [RConfigEngine.execute:76] Result=<?xml version="1.0" ?>
<RConfig>
<ConvertToRAC>
<Convert>
<Response>
<Result code="1" >
Operation Failed
</Result>
<ErrorDetails>
Clusterware is not configured
</ErrorDetails>
</Response>
</Convert>
</ConvertToRAC></RConfig>
Log file from /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/rconfig/mydb/sqllog
MYDB mydb 2622254467
10.2.0.1.0 ACTIVE
cluster_database FALSE
undo_management AUTO
db_domain
dispatchers (PROTOCOL=TCP) (SERVICE=mydbXDB)
background_dump_dest /u01/app/oracle/admin/mydb/bdump
user_dump_dest /u01/app/oracle/admin/mydb/udump
core_dump_dest /u01/app/oracle/admin/mydb/cdump
audit_file_dest /u01/app/oracle/admin/mydb/adump
MYDB mydb 2622254467
10.2.0.1.0 ACTIVE
cluster_database FALSE
undo_management AUTO
db_domain
dispatchers (PROTOCOL=TCP) (SERVICE=mydbXDB)
background_dump_dest /u01/app/oracle/admin/mydb/bdump
user_dump_dest /u01/app/oracle/admin/mydb/udump
core_dump_dest /u01/app/oracle/admin/mydb/cdump
audit_file_dest /u01/app/oracle/admin/mydb/adump
MYDB mydb 2622254467
10.2.0.1.0 ACTIVE
cluster_database TRUE
undo_management AUTO
db_domain
dispatchers (PROTOCOL=TCP) (SERVICE=mydbXDB)
background_dump_dest /u01/app/oracle/admin/mydb/bdump
user_dump_dest /u01/app/oracle/admin/mydb/udump
core_dump_dest /u01/app/oracle/admin/mydb/cdump
audit_file_dest /u01/app/oracle/admin/mydb/adump
MYDB mydb 2622254467
10.2.0.1.0 ACTIVE
cluster_database TRUE
undo_management AUTO
db_domain
dispatchers (PROTOCOL=TCP) (SERVICE=mydbXDB)
background_dump_dest /u01/app/oracle/admin/mydb/bdump
user_dump_dest /u01/app/oracle/admin/mydb/udump
core_dump_dest /u01/app/oracle/admin/mydb/cdump
audit_file_dest /u01/app/oracle/admin/mydb/adump
MYDB mydb 2622254467
10.2.0.1.0 ACTIVE
cluster_database TRUE
undo_management AUTO
db_domain
dispatchers (PROTOCOL=TCP) (SERVICE=mydbXDB)
background_dump_dest /u01/app/oracle/admin/mydb/bdump
user_dump_dest /u01/app/oracle/admin/mydb/udump
core_dump_dest /u01/app/oracle/admin/mydb/cdump
audit_file_dest /u01/app/oracle/admin/mydb/adump
Please help me where I am making mistake.
Thanks1) I have created single node standard database called mydb in /u01/app/oracle/product/10.2.0/db_1 home (hostname linux1)
2) installed crs and asm on linux1 and linux2 and shared storag on ASM(which external HD running ieee1294 cards and ports) . no database is created on linux1 or linux2
3) I want to convert mydb to RAC DATABASE called mydb1 instance on linux1 and mydb2 on linux2 machine respectively.
4) copied and modifed xml as you see above called ConvertTomydb.xml to $ORACLE_HOME/bin directory
5) when I run
$rconfig ConvertTomydb.xml from $ORACLE_HOME/bin directory , i get the following error
<ConvertToRAC>
<Convert>
<Response>
<Result code="1" >
Operation Failed
</Result>
<ErrorDetails>
Clusterware is not configured
</ErrorDetails>
</Response>
</Convert>
</ConvertToRAC>
$
Please see my crs_stat -t command output
Name Type Target State Host
ora....SM1.asm application ONLINE ONLINE linux1
ora....X1.lsnr application ONLINE ONLINE linux1
ora.linux1.gsd application ONLINE ONLINE linux1
ora.linux1.ons application ONLINE ONLINE linux1
ora.linux1.vip application ONLINE ONLINE linux1
ora....SM2.asm application ONLINE ONLINE linux2
ora....X2.lsnr application ONLINE ONLINE linux2
ora.linux2.gsd application ONLINE ONLINE linux2
ora.linux2.ons application ONLINE ONLINE linux2
ora.linux2.vip application ONLINE ONLINE linux2
ora.orcl.db application ONLINE ONLINE linux1
ora....l1.inst application ONLINE ONLINE linux1
ora....l2.inst application ONLINE ONLINE linux2
ora....test.cs application ONLINE ONLINE linux1
ora....cl1.srv application ONLINE ONLINE linux1
ora....cl2.srv application ONLINE UNKNOWN linux2
please see the output from olsnodes command
[oracle@linux1 bin]$ olsnodes
linux1
linux2
[oracle@linux1 bin]$
What is your cache fusion interconnect strategy?
I don't about this, please let me know where can i find the answers. what kind of command do i have to use get the answer
damorgan , Please let me know, if I gave answers to your questions. if not please let me know, i can give as much as possible. i really appreciate for your help
Thanks -
Root.ah fails on 2nd node(rac2) with [ ORA-15018,ORA-15017,ORA-15003 ]
Hi All,
I m trying to setup 11gR2 Grid installation on two-node Rac . When it comes to running root.sh on second node (i.e. rac2) it fails with below error. Could please anyone help me out. This is my 3rd attempt and all fails with below errors on node 2.
rac2:
[root@rac2 grid_home]# ./root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/grid_home
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2013-07-10 18:53:15: Parsing the host name
2013-07-10 18:53:15: Checking for super user privileges
2013-07-10 18:53:15: User has super user privileges
Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
DiskGroup CRS creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15017: diskgroup "CRS" cannot be mounted
ORA-15003: diskgroup "CRS" already mounted in another lock name space
Configuration of ASM failed, see logs for details
Did not succssfully configure and start ASM
CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init
Stop of resource "ora.crsd -init" failed
Failed to stop CRSD
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
Initial cluster configuration failed. See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details
[root@rac2 grid_home]#
rac2 alertrac2.log
[root@rac2 rac2]# cat -n alertrac2.log
1 Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2 2013-07-10 18:53:16.145
3 [client(13088)]CRS-2106:The OLR location /u01/grid_home/cdata/rac2.olr is inaccessible. Details in /u01/grid_home/log/rac2/client/ocrconfig_13088.log.
4 2013-07-10 18:53:16.228
5 [client(13088)]CRS-2101:The OLR was formatted using version 3.
6 2013-07-10 18:53:31.734
7 [ohasd(13132)]CRS-2112:The OLR service started on node rac2.
8 2013-07-10 18:53:31.893
9 [ohasd(13132)]CRS-2772:Server 'rac2' has been assigned to pool 'Free'.
10 2013-07-10 18:53:53.762
11 [ohasd(13132)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
12 2013-07-10 18:53:55.381
13 [cssd(14409)]CRS-1713:CSSD daemon is started in exclusive mode
14 2013-07-10 18:54:01.530
15 [cssd(14409)]CRS-1709:Lease acquisition failed for node rac2 because no voting file has been configured; Details at (:CSSNM00031:) in /u01/grid_home/log/rac2/cssd/ocssd.log
16 2013-07-10 18:54:19.113
17 [cssd(14409)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac2 .
18 2013-07-10 18:54:19.910
19 [ctssd(14465)]CRS-2403:The Cluster Time Synchronization Service on host rac2 is in observer mode.
20 2013-07-10 18:54:19.920
21 [ctssd(14465)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
22 2013-07-10 18:54:20.903
23 [ctssd(14465)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
24 [client(14715)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
25 [client(14719)]CRS-10001:ACFS-9322: done.
26 2013-07-10 18:54:47.104
27 [ctssd(14465)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user
28 2013-07-10 18:54:55.837
29 [cssd(14409)]CRS-1603:CSSD on node rac2 shutdown by user.
rac2 rootcrs logfile
[root@rac2 rac2]# cat /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log
2013-07-10 18:53:15: The configuration parameter file /u01/grid_home/crs/install/crsconfig_params is valid
2013-07-10 18:53:15: Checking for super user privileges
2013-07-10 18:53:15: User has super user privileges
2013-07-10 18:53:15: ### Printing the configuration values from files:
2013-07-10 18:53:15: /u01/grid_home/crs/install/crsconfig_params
2013-07-10 18:53:15: /u01/grid_home/crs/install/s_crsconfig_defs
2013-07-10 18:53:15: ASM_DISCOVERY_STRING=
2013-07-10 18:53:15: ASM_DISKS=ORCL:CRS1
2013-07-10 18:53:15: ASM_DISK_GROUP=CRS
2013-07-10 18:53:15: ASM_REDUNDANCY=EXTERNAL
2013-07-10 18:53:15: ASM_SPFILE=
2013-07-10 18:53:15: ASM_UPGRADE=false
2013-07-10 18:53:15: CLSCFG_MISSCOUNT=
2013-07-10 18:53:15: CLUSTER_GUID=
2013-07-10 18:53:15: CLUSTER_NAME=rac-scan
2013-07-10 18:53:15: CRS_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
2013-07-10 18:53:15: CRS_STORAGE_OPTION=1
2013-07-10 18:53:15: CSS_LEASEDURATION=400
2013-07-10 18:53:15: DIRPREFIX=
2013-07-10 18:53:15: DISABLE_OPROCD=0
2013-07-10 18:53:15: EMBASEJAR_NAME=oemlt.jar
2013-07-10 18:53:15: EWTJAR_NAME=ewt3.jar
2013-07-10 18:53:15: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
2013-07-10 18:53:15: GNS_ADDR_LIST=
2013-07-10 18:53:15: GNS_ALLOW_NET_LIST=
2013-07-10 18:53:15: GNS_CONF=false
2013-07-10 18:53:15: GNS_DENY_ITF_LIST=
2013-07-10 18:53:15: GNS_DENY_NET_LIST=
2013-07-10 18:53:15: GNS_DOMAIN_LIST=
2013-07-10 18:53:15: GPNPCONFIGDIR=/u01/grid_home
2013-07-10 18:53:15: GPNPGCONFIGDIR=/u01/grid_home
2013-07-10 18:53:15: GPNP_PA=
2013-07-10 18:53:15: HELPJAR_NAME=help4.jar
2013-07-10 18:53:15: HOST_NAME_LIST=rac1,rac2
2013-07-10 18:53:15: ID=/etc/init.d
2013-07-10 18:53:15: INIT=/sbin/init
2013-07-10 18:53:15: IT=/etc/inittab
2013-07-10 18:53:15: JEWTJAR_NAME=jewt4.jar
2013-07-10 18:53:15: JLIBDIR=/u01/grid_home/jlib
2013-07-10 18:53:15: JREDIR=/u01/grid_home/jdk/jre/
2013-07-10 18:53:15: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
2013-07-10 18:53:15: MSGFILE=/var/adm/messages
2013-07-10 18:53:15: NETCFGJAR_NAME=netcfg.jar
2013-07-10 18:53:15: NETWORKS="eth0"/192.168.0.0:public,"eth1"/192.168.1.0:cluster_interconnect
2013-07-10 18:53:15: NEW_HOST_NAME_LIST=
2013-07-10 18:53:15: NEW_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
2013-07-10 18:53:15: NEW_NODE_NAME_LIST=
2013-07-10 18:53:15: NEW_PRIVATE_NAME_LIST=
2013-07-10 18:53:15: NODELIST=rac1,rac2
2013-07-10 18:53:15: NODE_NAME_LIST=rac1,rac2
2013-07-10 18:53:15: OCFS_CONFIG=
2013-07-10 18:53:15: OCRCONFIG=/etc/oracle/ocr.loc
2013-07-10 18:53:15: OCRCONFIGDIR=/etc/oracle
2013-07-10 18:53:15: OCRID=
2013-07-10 18:53:15: OCRLOC=ocr.loc
2013-07-10 18:53:15: OCR_LOCATIONS=NO_VAL
2013-07-10 18:53:15: OLASTGASPDIR=/etc/oracle/lastgasp
2013-07-10 18:53:15: OLRCONFIG=/etc/oracle/olr.loc
2013-07-10 18:53:15: OLRCONFIGDIR=/etc/oracle
2013-07-10 18:53:15: OLRLOC=olr.loc
2013-07-10 18:53:15: OPROCDCHECKDIR=/etc/oracle/oprocd/check
2013-07-10 18:53:15: OPROCDDIR=/etc/oracle/oprocd
2013-07-10 18:53:15: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
2013-07-10 18:53:15: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
2013-07-10 18:53:15: ORACLE_BASE=/u01/11.2.0
2013-07-10 18:53:15: ORACLE_HOME=/u01/grid_home
2013-07-10 18:53:15: ORACLE_OWNER=grid
2013-07-10 18:53:15: ORA_ASM_GROUP=asmadmin
2013-07-10 18:53:15: ORA_DBA_GROUP=oinstall
2013-07-10 18:53:15: PRIVATE_NAME_LIST=
2013-07-10 18:53:15: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
2013-07-10 18:53:15: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
2013-07-10 18:53:15: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
2013-07-10 18:53:15: RC_KILL=K19
2013-07-10 18:53:15: RC_KILL_OLD=K96
2013-07-10 18:53:15: RC_START=S96
2013-07-10 18:53:15: SCAN_NAME=rac-scan.naveed.com
2013-07-10 18:53:15: SCAN_PORT=1521
2013-07-10 18:53:15: SCRBASE=/etc/oracle/scls_scr
2013-07-10 18:53:15: SHAREJAR_NAME=share.jar
2013-07-10 18:53:15: SILENT=false
2013-07-10 18:53:15: SO_EXT=so
2013-07-10 18:53:15: SRVCFGLOC=srvConfig.loc
2013-07-10 18:53:15: SRVCONFIG=/var/opt/oracle/srvConfig.loc
2013-07-10 18:53:15: SRVCONFIGDIR=/var/opt/oracle
2013-07-10 18:53:15: VNDR_CLUSTER=false
2013-07-10 18:53:15: VOTING_DISKS=NO_VAL
2013-07-10 18:53:15: ### Printing other configuration values ###
2013-07-10 18:53:15: CLSCFG_EXTRA_PARMS=
2013-07-10 18:53:15: CRSDelete=0
2013-07-10 18:53:15: CRSPatch=0
2013-07-10 18:53:15: DEBUG=
2013-07-10 18:53:15: DOWNGRADE=
2013-07-10 18:53:15: HAS_GROUP=oinstall
2013-07-10 18:53:15: HAS_USER=root
2013-07-10 18:53:15: HOST=rac2
2013-07-10 18:53:15: IS_SIHA=0
2013-07-10 18:53:15: OLR_DIRECTORY=/u01/grid_home/cdata
2013-07-10 18:53:15: OLR_LOCATION=/u01/grid_home/cdata/rac2.olr
2013-07-10 18:53:15: ORA_CRS_HOME=/u01/grid_home
2013-07-10 18:53:15: SUPERUSER=root
2013-07-10 18:53:15: UPGRADE=
2013-07-10 18:53:15: VF_DISCOVERY_STRING=
2013-07-10 18:53:15: addfile=/u01/grid_home/crs/install/crsconfig_addparams
2013-07-10 18:53:15: crscfg_trace=1
2013-07-10 18:53:15: crscfg_trace_file=/u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log
2013-07-10 18:53:15: hosts=
2013-07-10 18:53:15: oldcrshome=
2013-07-10 18:53:15: oldcrsver=
2013-07-10 18:53:15: osdfile=/u01/grid_home/crs/install/s_crsconfig_defs
2013-07-10 18:53:15: parameters_valid=1
2013-07-10 18:53:15: paramfile=/u01/grid_home/crs/install/crsconfig_params
2013-07-10 18:53:15: platform_family=unix
2013-07-10 18:53:15: srvctl_trc_suff=0
2013-07-10 18:53:15: unlock_crshome=
2013-07-10 18:53:15: user_is_superuser=1
2013-07-10 18:53:15: ### Printing of configuration values complete ###
2013-07-10 18:53:15: Oracle CRS stack is not configured yet
2013-07-10 18:53:15: CRS is not yet configured. Hence, will proceed to configure CRS
2013-07-10 18:53:15: Cluster-wide one-time actions... Done!
2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home
2013-07-10 18:53:15: Host name = rac2
2013-07-10 18:53:15: CRS user = grid
2013-07-10 18:53:15: Oracle CRS home = /u01/grid_home
2013-07-10 18:53:15: GPnP host = rac2
2013-07-10 18:53:15: Oracle GPnP home = /u01/grid_home/gpnp
2013-07-10 18:53:15: Oracle GPnP local home = /u01/grid_home/gpnp/rac2
2013-07-10 18:53:15: GPnP directories verified.
2013-07-10 18:53:15: Checking to see if Oracle CRS stack is already configured
2013-07-10 18:53:15: Oracle CRS stack is not configured yet
2013-07-10 18:53:15: ---Checking local gpnp setup...
2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" does not exist
2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" does not exist
2013-07-10 18:53:15: The setup file "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" does not exist
2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp/rac2: INVALID (bad profile/wallet)
2013-07-10 18:53:15: ---Checking cluster-wide gpnp setup...
2013-07-10 18:53:15: chk gpnphome /u01/grid_home/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer
2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer
2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/peer" -wu=peer '
2013-07-10 18:53:15: Removing file /tmp/file0qKE0c
2013-07-10 18:53:15: Successfully removed file: /tmp/file0qKE0c
2013-07-10 18:53:15: /bin/su successfully executed
2013-07-10 18:53:15: gpnptool: rc=0
2013-07-10 18:53:15: gpnptool output:
Profile signature is valid.
2013-07-10 18:53:15: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/peer"
2013-07-10 18:53:15: gpnptool: run /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer
2013-07-10 18:53:15: Running as user grid: /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer
2013-07-10 18:53:15: s_run_as_user2: Running /bin/su grid -c ' /u01/grid_home/bin/gpnptool verify -p="/u01/grid_home/gpnp/profiles/peer/profile.xml" -w="file:/u01/grid_home/gpnp/wallets/prdr" -wu=peer '
2013-07-10 18:53:16: Removing file /tmp/filebkOtBv
2013-07-10 18:53:16: Successfully removed file: /tmp/filebkOtBv
2013-07-10 18:53:16: /bin/su successfully executed
2013-07-10 18:53:16: gpnptool: rc=0
2013-07-10 18:53:16: gpnptool output:
Profile signature is valid.
2013-07-10 18:53:16: Profile "/u01/grid_home/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/grid_home/gpnp/wallets/prdr"
2013-07-10 18:53:16: chk gpnphome /u01/grid_home/gpnp: OK
2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.
2013-07-10 18:53:16: gpnp setup checked: local valid? 0 cluster-wide valid? 1
2013-07-10 18:53:16: Taking cluster-wide setup as local
2013-07-10 18:53:16: copy "/u01/grid_home/gpnp/profiles/peer/profile.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml"
2013-07-10 18:53:16: set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile.xml" => (grid,oinstall)
2013-07-10 18:53:16: copy "/u01/grid_home/gpnp/wallets/peer/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso"
2013-07-10 18:53:16: set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cwallet.sso" => (grid,oinstall)
2013-07-10 18:53:16: copy "/u01/grid_home/gpnp/wallets/prdr/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso"
2013-07-10 18:53:16: set ownership on "/u01/grid_home/gpnp/rac2/wallets/prdr/cwallet.sso" => (grid,oinstall)
2013-07-10 18:53:16: copy "/u01/grid_home/gpnp/profiles/peer/profile_orig.xml" => "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml"
2013-07-10 18:53:16: set ownership on "/u01/grid_home/gpnp/rac2/profiles/peer/profile_orig.xml" => (grid,oinstall)
2013-07-10 18:53:16: copy "/u01/grid_home/gpnp/wallets/root/ewallet.p12" => "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12"
2013-07-10 18:53:16: set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/ewallet.p12" => (grid,oinstall)
2013-07-10 18:53:16: copy "/u01/grid_home/gpnp/wallets/pa/cwallet.sso" => "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso"
2013-07-10 18:53:16: set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cwallet.sso" => (grid,oinstall)
2013-07-10 18:53:16: copy "/u01/grid_home/gpnp/wallets/root/b64certificate.txt" => "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt"
2013-07-10 18:53:16: set ownership on "/u01/grid_home/gpnp/rac2/wallets/root/b64certificate.txt" => (grid,oinstall)
2013-07-10 18:53:16: copy "/u01/grid_home/gpnp/wallets/peer/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt"
2013-07-10 18:53:16: set ownership on "/u01/grid_home/gpnp/rac2/wallets/peer/cert.txt" => (grid,oinstall)
2013-07-10 18:53:16: copy "/u01/grid_home/gpnp/wallets/pa/cert.txt" => "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt"
2013-07-10 18:53:16: set ownership on "/u01/grid_home/gpnp/rac2/wallets/pa/cert.txt" => (grid,oinstall)
2013-07-10 18:53:16: GPnP Wallets ownership/permissions successfully set.
2013-07-10 18:53:16: gpnp setup: GOTCLUSTERWIDE
2013-07-10 18:53:16: Validating for SI-CSS configuration
2013-07-10 18:53:16: Retrieving OCR main disk location
2013-07-10 18:53:16: Opening file OCRCONFIG
2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc
2013-07-10 18:53:16: Unable to retrieve ocr disk info
2013-07-10 18:53:16: Checking to see if any 9i GSD is up
2013-07-10 18:53:16: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so
2013-07-10 18:53:16: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so
2013-07-10 18:53:16: SKGXN library file does not exists
2013-07-10 18:53:16: OLR location = /u01/grid_home/cdata/rac2.olr
2013-07-10 18:53:16: Oracle CRS Home = /u01/grid_home
2013-07-10 18:53:16: Validating /etc/oracle/olr.loc file for OLR location /u01/grid_home/cdata/rac2.olr
2013-07-10 18:53:16: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
2013-07-10 18:53:16: Oracle CRS home = /u01/grid_home
2013-07-10 18:53:16: Oracle cluster name = rac-scan
2013-07-10 18:53:16: OCR locations = +CRS
2013-07-10 18:53:16: Validating OCR
2013-07-10 18:53:16: Retrieving OCR location used by previous installations
2013-07-10 18:53:16: Opening file OCRCONFIG
2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc
2013-07-10 18:53:16: Opening file OCRCONFIG
2013-07-10 18:53:16: Value () is set for key=ocrmirrorconfig_loc
2013-07-10 18:53:16: Opening file OCRCONFIG
2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc3
2013-07-10 18:53:16: Opening file OCRCONFIG
2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc4
2013-07-10 18:53:16: Opening file OCRCONFIG
2013-07-10 18:53:16: Value () is set for key=ocrconfig_loc5
2013-07-10 18:53:16: Checking if OCR sync file exists
2013-07-10 18:53:16: No need to sync OCR file
2013-07-10 18:53:16: OCR_LOCATION=+CRS
2013-07-10 18:53:16: OCR_MIRROR_LOCATION=
2013-07-10 18:53:16: OCR_MIRROR_LOC3=
2013-07-10 18:53:16: OCR_MIRROR_LOC4=
2013-07-10 18:53:16: OCR_MIRROR_LOC5=
2013-07-10 18:53:16: Current OCR location=
2013-07-10 18:53:16: Current OCR mirror location=
2013-07-10 18:53:16: Current OCR mirror loc3=
2013-07-10 18:53:16: Current OCR mirror loc4=
2013-07-10 18:53:16: Current OCR mirror loc5=
2013-07-10 18:53:16: Verifying current OCR settings with user entered values
2013-07-10 18:53:16: Setting OCR locations in /etc/oracle/ocr.loc
2013-07-10 18:53:16: Validating OCR locations in /etc/oracle/ocr.loc
2013-07-10 18:53:16: Checking for existence of /etc/oracle/ocr.loc
2013-07-10 18:53:16: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
2013-07-10 18:53:16: Setting ocr location +CRS
2013-07-10 18:53:16: Creating or upgrading Oracle Local Registry (OLR)
2013-07-10 18:53:16: OLR successfully created or upgraded
2013-07-10 18:53:16: /u01/grid_home/bin/clscfg -localadd
2013-07-10 18:53:16: Keys created in the OLR successfully
2013-07-10 18:53:16: GPnP setup state: new-cluster-wide
2013-07-10 18:53:16: GPnP cluster configuration already performed
2013-07-10 18:53:16: Registering ohasd
2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/init.ohasd
2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/init.ohasd to /etc/init.d directory
2013-07-10 18:53:16: Setting init.ohasd permission in /etc/init.d directory
2013-07-10 18:53:16: init file = /u01/grid_home/crs/init/ohasd
2013-07-10 18:53:16: Copying file /u01/grid_home/crs/init/ohasd to /etc/init.d directory
2013-07-10 18:53:16: Setting ohasd permission in /etc/init.d directory
2013-07-10 18:53:16: Removing "/etc/rc.d/rc3.d/S96ohasd"
2013-07-10 18:53:16: Removing file /etc/rc.d/rc3.d/S96ohasd
2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
2013-07-10 18:53:16: Failed to remove file:
2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
2013-07-10 18:53:16: Removing "/etc/rc.d/rc5.d/S96ohasd"
2013-07-10 18:53:16: Removing file /etc/rc.d/rc5.d/S96ohasd
2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
2013-07-10 18:53:16: Failed to remove file:
2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
2013-07-10 18:53:16: Removing "/etc/rc.d/rc0.d/K19ohasd"
2013-07-10 18:53:16: Removing file /etc/rc.d/rc0.d/K19ohasd
2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
2013-07-10 18:53:16: Failed to remove file:
2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
2013-07-10 18:53:16: Removing "/etc/rc.d/rc1.d/K19ohasd"
2013-07-10 18:53:16: Removing file /etc/rc.d/rc1.d/K19ohasd
2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
2013-07-10 18:53:16: Failed to remove file:
2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
2013-07-10 18:53:16: Removing "/etc/rc.d/rc2.d/K19ohasd"
2013-07-10 18:53:16: Removing file /etc/rc.d/rc2.d/K19ohasd
2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
2013-07-10 18:53:16: Failed to remove file:
2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
2013-07-10 18:53:16: Removing "/etc/rc.d/rc4.d/K19ohasd"
2013-07-10 18:53:16: Removing file /etc/rc.d/rc4.d/K19ohasd
2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
2013-07-10 18:53:16: Failed to remove file:
2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
2013-07-10 18:53:16: Removing "/etc/rc.d/rc6.d/K19ohasd"
2013-07-10 18:53:16: Removing file /etc/rc.d/rc6.d/K19ohasd
2013-07-10 18:53:16: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
2013-07-10 18:53:16: Failed to remove file:
2013-07-10 18:53:16: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
2013-07-10 18:53:16: The file ohasd has been successfully linked to the RC directories
2013-07-10 18:53:16: Starting ohasd
2013-07-10 18:53:16: itab entries=
2013-07-10 18:53:21: Created backup /etc/inittab.no_crs
2013-07-10 18:53:21: Appending to /etc/inittab.tmp:
2013-07-10 18:53:21: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
2013-07-10 18:53:21: Done updating /etc/inittab.tmp
2013-07-10 18:53:21: Saved /etc/inittab.crs
2013-07-10 18:53:21: Installed new /etc/inittab
2013-07-10 18:53:36: ohasd is starting
2013-07-10 18:53:36: Checking ohasd
2013-07-10 18:53:37: ohasd started successfully
2013-07-10 18:53:37: Creating CRS resources and dependencies
2013-07-10 18:53:37: Configuring HASD
2013-07-10 18:53:37: Registering type ora.daemon.type
2013-07-10 18:53:37: Registering type ora.mdns.type
2013-07-10 18:53:37: Registering type ora.gpnp.type
2013-07-10 18:53:38: Registering type ora.gipc.type
2013-07-10 18:53:38: Registering type ora.cssd.type
2013-07-10 18:53:38: Registering type ora.cssdmonitor.type
2013-07-10 18:53:39: Registering type ora.crs.type
2013-07-10 18:53:39: Registering type ora.evm.type
2013-07-10 18:53:39: Registering type ora.ctss.type
2013-07-10 18:53:40: Registering type ora.asm.type
2013-07-10 18:53:40: Registering type ora.drivers.acfs.type
2013-07-10 18:53:40: Registering type ora.diskmon.type
2013-07-10 18:53:51: ADVM/ACFS is configured
2013-07-10 18:53:51: Successfully created CRS resources for cluster daemon and ASM
2013-07-10 18:53:51: Checking if initial configuration has been performed
2013-07-10 18:53:51: Starting CSS in exclusive mode
2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
2013-07-10 18:54:19: CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
2013-07-10 18:54:19: CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
2013-07-10 18:54:19: CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
2013-07-10 18:54:19: CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
2013-07-10 18:54:19: CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
2013-07-10 18:54:19: CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
2013-07-10 18:54:19: Querying for existing CSS voting disks
2013-07-10 18:54:19: Performing initial configuration for cluster
2013-07-10 18:54:21: Start of resource "ora.ctssd -init" Succeeded
2013-07-10 18:54:21: Configuring ASM via ASMCA
2013-07-10 18:54:21: Executing as grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM
2013-07-10 18:54:21: Running as user grid: /u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM
2013-07-10 18:54:21: Invoking "/u01/grid_home/bin/asmca -silent -diskGroupName CRS -diskList ORCL:CRS1 -redundancy EXTERNAL -configureLocalASM" as user "grid"
2013-07-10 18:54:40: Configuration of ASM failed, see logs for details
2013-07-10 18:54:40: Did not succssfully configure and start ASM
2013-07-10 18:54:40: Exiting exclusive mode
2013-07-10 18:54:40: Command return code of 1 (256) from command: /u01/grid_home/bin/crsctl stop resource ora.crsd -init
2013-07-10 18:54:40: Stop of resource "ora.crsd -init" failed
2013-07-10 18:54:40: Failed to stop CRSD
2013-07-10 18:55:04: Initial cluster configuration failed. See /u01/grid_home/cfgtoollogs/crsconfig/rootcrs_rac2.log for details
Also below are some of the configs related to rac2 node
[root@rac2 rac2]# rpm -qa | grep oracleasm
oracleasmlib-2.0.4-1.el5
oracleasm-support-2.1.8-1.el5
oracleasm-2.6.18-274.el5xen-2.0.5-1.el5
oracleasm-2.6.18-274.el5-2.0.5-1.el5
oracleasm-2.6.18-274.el5debug-2.0.5-1.el5
oracleasm-2.6.18-274.el5-debuginfo-2.0.5-1.el5
[root@rac2 rac2]# /usr/sbin/oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
[root@rac2 rac2]# /usr/sbin/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@rac2 rac2]# /usr/sbin/oracleasm listdisks
CRS1
DATA1
FRA1
[root@rac2 rac2]# ls -l /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmadmin 8, 17 Jul 10 18:35 CRS1
brw-rw---- 1 grid asmadmin 8, 33 Jul 10 18:36 DATA1
brw-rw---- 1 grid asmadmin 8, 49 Jul 10 18:36 FRA1
[root@rac2 rac2]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
#Public IP's(eth0)
192.168.0.101 rac1.naveed.com rac1
192.168.0.102 rac2.naveed.com rac2
#Private IP's(eth1)
192.168.1.101 rac1-prv.naveed.com rac1-prv
192.168.1.102 rac2-prv.naveed.com rac2-prv
#VIPS
192.168.0.221 rac1-vip.naveed.com rac1-vip
192.168.0.222 rac2-vip.naveed.com rac2-vip
#DNS server IP
192.168.0.10 naveeddns.naveed.com naveeddns
[root@rac2 rac2]#
Thanks in advanceHi,
First of all thanks a lot for the response. You wont't beleive this is my 7th fresh installation and everytime in node 2 i m hit with this same error.
Also i tried below procedure instead of fresh installation
once i deconfig & rerun (./rootcrs.pl -verbose -deconfig -force) on node 2
Using configuration parameter file: ./crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac2'
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac2' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node
[root@rac2 grid_home]# ./root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/grid_home
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/grid_home/crs/install/crsconfig_params
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
ASM created and started successfully.
Disk Group CRS mounted successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Successful addition of voting disk 636af26485ef4f27bfec31523aaa0660.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE 636af26485ef4f27bfec31523aaa0660 (ORCL:CRS1) [CRS]
Located 1 voting disk(s).
Start of resource "ora.crsd" failed
CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'rac2'
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Grid Infrastructure stack
Failed to start Cluster Ready Services at /u01/grid_home/crs/install/crsconfig_lib.pm line 1286.
/u01/grid_home/perl/bin/perl -I/u01/grid_home/perl/lib -I/u01/grid_home/crs/install /u01/grid_home/crs/install/rootcrs.pl execution failed -
When we create one service as preferred on both nodes in two node RAC
How to configure listener,tnsnames.ora & listener file When we create one service as preferred on both nodes in two node RAC ... ( I don't need load balancing here but i just want to create service as preferred on both nodes)
please some one help me in this ..Thanks alot Sebastain for your reply..
I am using 10.2.0.4 version and below tns entry is from my client side tns entry ..
M4AMPRD_TEST=
(DESCRIPTION=
(ADDRESS= (PROTOCOL=TCP) (HOST=153.88.184.228) (PORT=1521))
(ADDRESS= (PROTOCOL=TCP) (HOST=153.88.184.229) (PORT=1521))
(FAILOVER=ON)
(CONNECT_DATA=(SERVICE_NAME=M4AMPRD_TEST)
(FAILOVER_MODE=
(TYPE=SELECT)
(METHOD=BASIC)
(RETRIES=20)
(DELAY=5)
service creation: srvctl add service -d M4AMPRD -s M4AMPRD_TEST -r M4AMPRD1,M4AMPRD2
But when i connect to database usign above service from client some times its working fine and some times its failing please see below log with timings how it is behaving ..
SQL> set time on
18:39:46 SQL> CONN m4owner/iamm4amdev!@M4AMPRD_TEST
Connected.
18:39:48 SQL> CONN m4owner/iamm4amdev!@M4AMPRD_TEST
ERROR:
ORA-12545: Connect failed because target host or object does not exist
Warning: You are no longer connected to ORACLE.
18:39:52 SQL> CONN m4owner/iamm4amdev!@M4AMPRD_TEST
Connected.
18:39:53 SQL> CONN m4owner/iamm4amdev!@M4AMPRD_TEST
Connected.
18:39:55 SQL> CONN m4owner/iamm4amdev!@M4AMPRD_TEST
Connected.
18:39:57 SQL> CONN m4owner/iamm4amdev!@M4AMPRD_TEST
ERROR:
ORA-12545: Connect failed because target host or object does not exist
Warning: You are no longer connected to ORACLE.
18:39:59 SQL> CONN m4owner/iamm4amdev!@M4AMPRD_TEST
Connected.
Thanks for your help in advance
Anil Vejendla.. -
Processes in v$process that do not exist in v$session (2 node RAC)!
Hi, all.
The database is 2 node RAC database (10.2.0.2.0)
on 32 bit windows 2003 EE SP1.
Our database is suffering "CKPT hang" from time to time.
I checked v$process and v$session on both node by the following sql.
select addr,pid,spid,username, program
from v$process
where addr not in (select paddr from v$session)
ADDR PID SPID USERNAME PROGRAM
56E2 1 PSEUDO
56E2 18 3984 SYSTEM ORACLE.EXE (D000)
56E2 19 4020 SYSTEM ORACLE.EXE (S000)
56E2 27 3176 SYSTEM ORACLE.EXE (PZ99)
56E3 39 2296 SYSTEM ORACLE.EXE (PZ97)
●select * from v$px_process
SERVER_NAME STATUS PID SPID SID SERIAL#
PZ97 AVAILABLE 39 2296
PZ99 AVAILABLE 27 3176
●select * from V$PX_SESSION
--> no rows
●select slave_name,status from v$pq_slave
SLAVE_NAME STATUS
PZ97 IDLE
PZ99 IDLE
I found the above processes which do not exist in v$session.
Is this normal??
Thanks and Regards.you see nothing in v$session because there is nothing to see ...
the Parallel servers are AVAILABLE , ie no session are running parallel executions (as shown in V$PX_SESSION)
PID is not SID -
Platform: "HP-UX 11.23 64-bit"
Database: "10.2.0.4 64-bit"
RAC: 2 Node RAC setup
Our RAC setup has been properly done and RAC is working fine with load balancing i.e clients are getting connection on both instances. BUT the issue I am facing with my RAC setup is High Availability testing. When I send reboot signal to "Node-2" and the "Node-1" is up what I observe and receive complain from clients that they have lost connection with database ALSO no new connections are being allowed. When I see the alert log of "Node-1" I see the following abnormal messages reported in it:
List of nodes:
0 1
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Tue Aug 9 04:02:15 2011
LMS 2: 0 GCS shadows cancelled, 0 closed
Tue Aug 9 04:02:15 2011
LMS 0: 0 GCS shadows cancelled, 0 closed
Tue Aug 9 04:02:15 2011
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Tue Aug 9 04:02:15 2011
LMS 1: 1908 GCS shadows traversed, 1076 replayed
Tue Aug 9 04:02:15 2011
LMS 2: 1911 GCS shadows traversed, 1086 replayed
Tue Aug 9 04:02:15 2011
LMS 0: 1899 GCS shadows traversed, 1164 replayed
Tue Aug 9 04:02:15 2011
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Tue Aug 9 04:02:16 2011
ARCH shutting down
ARC2: Archival stopped
Tue Aug 9 04:02:21 2011
Redo thread 2 internally enabled
Tue Aug 9 04:02:35 2011
Reconfiguration started (old inc 4, new inc 6)
List of nodes:
0
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Tue Aug 9 04:02:35 2011
LMS 1: 0 GCS shadows cancelled, 0 closed
Tue Aug 9 04:02:35 2011
LMS 2: 0 GCS shadows cancelled, 0 closed
Tue Aug 9 04:02:35 2011
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Tue Aug 9 04:02:35 2011
Instance recovery: looking for dead threads
Tue Aug 9 04:02:35 2011
Beginning instance recovery of 1 threads
Tue Aug 9 04:02:35 2011
LMS 1: 1908 GCS shadows traversed, 0 replayed
Tue Aug 9 04:02:35 2011
LMS 2: 1907 GCS shadows traversed, 0 replayed
Tue Aug 9 04:02:35 2011
LMS 0: 1899 GCS shadows traversed, 0 replayed
Tue Aug 9 04:02:35 2011
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Tue Aug 9 04:02:37 2011
parallel recovery started with 11 processes
Tue Aug 9 04:02:37 2011
Started redo application at
Thread 2: logseq 6, block 2, scn 1837672332
Tue Aug 9 04:02:37 2011
Errors in file /u01/app/oracle/product/10.2.0/db/admin/BAF/bdump/baf1_smon_10253.trc:
ORA-00600: internal error code, arguments: [kcratr2_onepass], [], [], [], [], [], [], []
Tue Aug 9 04:02:38 2011
Errors in file /u01/app/oracle/product/10.2.0/db/admin/BAF/bdump/baf1_smon_10253.trc:
ORA-00600: internal error code, arguments: [kcratr2_onepass], [], [], [], [], [], [], []
Tue Aug 9 04:02:38 2011
Errors in file /u01/app/oracle/product/10.2.0/db/admin/BAF/bdump/baf1_smon_10253.trc:
ORA-00600: internal error code, arguments: [kcratr2_onepass], [], [], [], [], [], [], []
SMON: terminating instance due to error 600
Tue Aug 9 04:02:38 2011
Dump system state for local instance only
System State dumped to trace file /u01/app/oracle/product/10.2.0/db/admin/BAF/bdump/baf1_diag_10229.trc
Tue Aug 9 04:02:38 2011
Instance terminated by SMON, pid = 10253
Tue Aug 9 04:04:09 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 lan3 192.168.1.0 configured from OCR for use as a cluster interconnect
Interface type 1 lan2 172.20.21.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned off.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 300
sessions = 335
timed_statistics = TRUE
Kindly help me to get rid out of this issue. Waiting for the quick and helpful response from the gurus in the forum. Thanks in advance.
Regards,if above were really 100% correct, you would not be here posting about errors!Definitely but these situations could become the cause for new BUGS, isn't it?
I don't know what is real & what is unnecessary obfuscation.What part of the thread you didn't understand.
It is not a good idea to have subtraction sign/dash character as part of object/host name; i.e. "Node-1"."Node-1" is not the hostname it is just to make clear understanding. the hostname is "sdupn101" for node-1 and "sdupn102" for node-2.
ORA-00600/ORA-07445/ORA-03113 = Oracle bug => search on Metalink and/or call Oracle supportNewbie is my status on this forum but I have little bit ethics of using forums and suppot blogs. I searched but unfortunately didn't find any matching solution.
Anyway will update you once find any solution so that you can assist someone else in future.
Maybe you are looking for
-
Recorded presentation - can't open file
Let me know if anyone can help. We recorded a presentation which created a 150MB .cp file. Unfortunately we can't open the file to be able to publish it. Not sure if it is too large in size. This was recorded using Captivate 4 and there is no erro
-
Sudden, unexplainable lag and performance problems making gaming impossible
Last week I was happily gaming with all the settings on high. I went away for the weekend and when I came back, it is absolutely impossible to play because of lag. When I put all the settings at their very lowest, it is almost possible to play but,
-
Hello, I am integrating XMLMILL (www.xmlmill.com) into EP6 to provide PDF rendering on the fly from xml+xsl data. I finally got it working and I am now able to render impressive PDF files but seems xmlmill is not fully compatible with EP6 from differ
-
Why can't i log into my accounts on the internet it keeps going back to the log in page
my internet works fine and i like the phone but there has to be a setting or something that i am missing because any time i try to sign into any website that has a log in it just goes right back to the log in page over and over. it looks like i put
-
Apple TV-playing itunes library through TV speakers
I have just installed apple TV-initially it showed as an alternative speaker in my itunes so that I could play my "unsynced" tunes through the TV speakers-now it seems to have disappeared leaving just the "computer" as the only speaker option....what