Sqlplus is hanging on RAC1 node

Hi all,
I have 3 Node RAC DB (11.2.0.3). When i am trying to connect sqlplus to rac 1 , there is no error but i am not getting sqlplus prompt.
RAC 1 : sqlplus "/as sysdba" -- Hangs
RAC 2 : sqlplus "/as sysdba" -- getting prompt
RAC 3 : sqlplus "/as sysdba" -- getting prompt
from RAC 2 : sqlplus sys/password@rac1 as sysdba -- Hangs
RAC 3 : sqlplus sys/password@rac1 as sysdba -- Hangs
RAC 2 : sqlplus "/as sysdba" -- getting prompt but when i query any thing from gv$session, then it hangs.
there is no error in alert log and cput utilization on RAC 1 is 5-10% only (Which is normal).
Kindly inform me how i can resolve this issue.
Thanks
Krishna

Hello All,
My operating system is Linux 64 Bit. and all the mount points have sufficient space. There is no message in DB alert log/listener log.
Kindly advice me if i can check some else.
Thanks
Krishna

Similar Messages

R3trans and sqlplus/@SID hang when the AS is started

Hi all,
I have migrated one platform on Windows 2003 with SAP 4.6C Oracle 10.2.2 and client 920
Everything is working fine with the DBCI + the first AS
SAP + DB + DB COnnect ...
I can perform an R3tans with return code 000 from the DBCI , the first AS and the 2nd AS as well
But When I start the 2nd Application Server ; the R3trans hangs, and I can't connect to the Database (sqlplus /@SDI hangs) through the 2nd AS, the first and the DBCI neighter.
If someone has encountered this issue, thanks for your help
Cheers
Yacine

Hi Eric,
Many thanks for your answer and sorry for the delay of my anser.
Indeed we have adjusted these parameters and deceased the WP for both AS.
Currently it's a closing period for us so the 2 AS si stopped for security, but for the next month we will start it whith these changes.
Thanks to all for your help
Yacine

SQLPlus login hangs because of "Thread 1 cannot allocate new log"

Hi to everybody!
Could you please give an advice how to overcome the following situation.
In my 10GR2 database I have destination for archive logs placed in ASM disk group. When suddenly this disk group becomes overfilled I can not login into the database even "/as sysdba" locally. The best solution is to switch log_archive_dest_1 to some other place, but as I mentioned I even can not login into the database to do this change. Another solution is to free some space in ASM disk group by deletion of some archive logs. But if I have a Standby DB and if I delete some archive logs I'll lose my Standby DB.
I will be glad to hear your real life's experience :)
Thank's in advance!

Dear Sybrand, I've used search button a lot. I'd be happier if you had pointed where the documentation has something useful for my situation. May be you meant
an alternate destination for archive logs, but as you know there is no exact upper bound for the "REOPEN" parameter. So switching can take undetermined time.
Seems to me I've made a full description of my situation.
I was not able to login to the database - this is a fact. My database is quite up to date (PSU 4) and OS environment is set correctly. So i don't believe that my case is something very uncommon.
Let me make several clarification:
1)I can not extend LUN on wich ASM disk group has been created, because this operation requires restart of the host - not an online operation.
2) I can not move archive logs from my ASM disk group to ether another one or to the file systems, because a) I can not login into the database neither through SQLPLUS nor through RMAN 2) in 10GR2 "asmcmd" doesn't have an ability to copy files ("cp")
3) Let's imagine (it can be) that connectivity with standby had been broken before my incident took place and none of the archive logs was shipped to the Standby site. This exclude an option of deleting some of archive logs through the "asmcmd" and "rm" because none of them were applied to a standby.
The only solution I used is to find some idle opened consoles in a SCREEN (linux) and use them. But this method is not quite reliable.
P.S. When I said "SQLPlus login hangs" I meant "/as sysdba", not just simple login.. It is clear that simple login will fail.

View logfile or view output hangs in 3 node environment

Hi All,
Briefing the environment:
Database -- ERPDB001
Conc + admin -- APPS001
Forms + web -- APPS002
Some times when we try to see view log or view output after running the Concurrent requests. The system seems like hanging and it will display following error after. If you check in the next minute, it will show the output/logfile. looks like a strange behavior
"An error occurred while attempting to establish an Application File Server connection with the node APPS001. There may be a network configuration problem, or the TNS listener on node APPS001 may not be running, Please contact your system administrator."
Is this the Oracle Apps issue or the network issue between two node.
Regards
Vasu

05-JUL-2008 09:51:56 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST
=<APPS002IP>)(PORT=56065)) * establish * FNDFS * 0
05-JUL-2008 09:51:59 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST
=<APPS002IP>)(PORT=56070)) * establish * FNDFS * 0
05-JUL-2008 09:52:16 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST
=<APPS002IP>)(PORT=56087)) * establish * FNDFS * 0
05-JUL-2008 10:20:28 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=<HostnameofAPPS001>)(US
ER=applprod))(COMMAND=status)(ARGUMENTS=64)(SERVICE=APPS_PROD)(VERSION=134243072
)) * status * 0
05-JUL-2008 10:20:28 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=<HostnameofAPPS001>)(US
ER=applprod))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=APPS_PROD)(VERSION=134243072))
* stop * 0
05-JUL-2008 09:51:56 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=56065)) * establish * FNDFS * 0
05-JUL-2008 09:51:59 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=56070)) * establish * FNDFS * 0
05-JUL-2008 09:52:16 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=56087)) * establish * FNDFS * 0
05-JUL-2008 10:20:28 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=<HostnameofAPPS001>)(USER=applprod))(COMMAND=status)(ARGUMENTS=64)(SERVICE=APPS_PROD)(VERSION=134243072)) * status * 0
05-JUL-2008 10:20:28 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=<HostnameofAPPS001>)(USER=applprod))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=APPS_PROD)(VERSION=134243072)) * stop * 0
TNSLSNR for Solaris: Version 8.0.6.3.0 - Production on 05-JUL-2008 10:25:03
(c) Copyright 1999 Oracle Corporation. All rights reserved.
Log messages written to /u02/applprod/prodora/8.0.6/network/admin/apps_prod.log
Listening on: (ADDRESS=(PROTOCOL=tcp)(DEV=10)(HOST=<IPofAPPS001>)(PORT=1676))
TIMESTAMP * CONNECT DATA [* PROTOCOL INFO] * EVENT [* SID] * RETURN CODE
05-JUL-2008 10:25:03 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=<HostnameofAPPS001>)(USER=applprod))(COMMAND=status)(ARGUMENTS=64)(SERVICE=APPS_PROD)(VERSION=134243072)) * status * 0
05-JUL-2008 10:25:34 * ping * 0
05-JUL-2008 10:25:35 * ping * 0
05-JUL-2008 10:25:35 * (CONNECT_DATA=(SID=FNDSM)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<IPofAPPS001>)(PORT=37849)) * establish * FNDSM * 0
05-JUL-2008 10:41:53 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=59076)) * establish * FNDFS * 0
05-JUL-2008 10:47:52 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=59435)) * establish * FNDFS * 0
05-JUL-2008 11:00:34 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=60186)) * establish * FNDFS * 0
05-JUL-2008 11:16:33 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=61139)) * establish * FNDFS * 0
05-JUL-2008 11:33:41 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=62210)) * establish * FNDFS * 0
05-JUL-2008 11:34:06 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=62236)) * establish * FNDFS * 0
05-JUL-2008 11:38:47 * 12502
TNS-12502: TNS:listener received no CONNECT_DATA from client
05-JUL-2008 11:46:32 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=62977)) * establish * FNDFS * 0
05-JUL-2008 12:12:37 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=64546)) * establish * FNDFS * 0
05-JUL-2008 12:12:39 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=64548)) * establish * FNDFS * 0
05-JUL-2008 12:13:37 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=64609)) * establish * FNDFS * 0
05-JUL-2008 12:26:42 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=65439)) * establish * FNDFS * 0
05-JUL-2008 12:26:44 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=65444)) * establish * FNDFS * 0
05-JUL-2008 12:27:22 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=65485)) * establish * FNDFS * 0
05-JUL-2008 12:28:38 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=32810)) * establish * FNDFS * 0
05-JUL-2008 12:30:14 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=32904)) * establish * FNDFS * 0
05-JUL-2008 12:30:27 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=32918)) * establish * FNDFS * 0
05-JUL-2008 12:51:17 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=34177)) * establish * FNDFS * 0
05-JUL-2008 13:47:59 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=37909)) * establish * FNDFS * 0
05-JUL-2008 13:48:02 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=37914)) * establish * FNDFS * 0
05-JUL-2008 14:15:10 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=39543)) * establish * FNDFS * 0
05-JUL-2008 14:15:12 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=39548)) * establish * FNDFS * 0
05-JUL-2008 14:25:03 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=40157)) * establish * FNDFS * 0
05-JUL-2008 14:25:07 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=40162)) * establish * FNDFS * 0
05-JUL-2008 14:25:17 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=39984)) * establish * FNDFS * 0
05-JUL-2008 14:26:19 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=40236)) * establish * FNDFS * 0
05-JUL-2008 16:32:50 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=47917)) * establish * FNDFS * 0
05-JUL-2008 16:32:55 * (CONNECT_DATA=(SID=FNDFS)) * (ADDRESS=(PROTOCOL=tcp)(HOST=<APPS002IP>)(PORT=47925)) * establish * FNDFS * 0
=========================
We found not much errors.
One time the error is
"05-JUL-2008 11:38:47 * 12502
TNS-12502: TNS:listener received no CONNECT_DATA from client"
But I used to get above Original error (Core issue) lot of times in the Application.
Even users are complaining about this.
Thanks and Regards
Vasu

What areas to look ,when the application causes the Oracel Sqlplus to Hang

we got an application that suddenly hangs and after that we are no longer able to connect through SQLplus.What should i look for.

My vote goes to, looking into the alert log file, any trace files generated, is the same with all the users and of course, my favorate v$session_wait.
V$SESSION_WAIT just reveals what is going on currently.
You can also try HANGANALYSIS, with the help of Oracle support.
For more details, check in Metalink for HANG ANALYSIS.
Jaffar

Sqlplus -s hangs

hi,
when using sqlplus with s option, sqlplus -s system/my_password
it hangs all the time,(sqlplus system/my_password is ok)
can anyone explain this for me, thanks.
(win2003,oracle 11g r1)
Edited by: roy_lau on 2010-1-2 上午11:04

If I remember correctly the '-s' command line switch stands for silent execution of a script.
In that case you would have to be sure that your script ends with an 'exit' (followed by a carriage return).
Also you would want to make sure that your script does not require any user-input, as it will just stop and wait for you to input that.
May be you can run the script without the silent option, and see if it then finishes without you having to enter anything with the keyboard.

Root.sh hangs in CRS install on first node

Hi all,
I am installing CRS 10.2.0 in RHEL 5 on VMWARE.when It is asking for two scripts(oraInstRoot.sh and root.sh) as a root user.First script completes with out any error on both nodes while root.sh script hangs in first node at line
Startup will be queued to init with in 90 seconds
I have checked for error in /tmp/crsctl. below is error
Failure at SlfAccess(/u01/crs/oracle/product/10.2..0/crs/bin) with code -1
Internal Error Information:
Category: -5
Operation: access failed
Location: SlfAccess
Other:
Dep: 2
Kindly share if any one has solved above error.
thanks
Latif

Hello,
Your problem is indeed Bug: 4679769. Our output of the tracefile explains this.
You need to download the patch, apply it.
But before you run root.sh again, clean the ocr disks. (both) using dd.
when done you can just reexecute the root.sh script again.

Service is not relocated to available instance when pref node hangs.

Hi,
I'm fighting strange problem with 4 node 10.2.0.3 RAC on Linux .
We have service defined as follows:
IT PREF: hp3 AVAIL: hp4 hp2 hp1
1* select name,service_id from dba_services where name = 'IT'
SQL> /
NAME                                                             SERVICE_ID
IT                                                                        8
1 select name, failover_method, failover_type, failover_retries,goal, clb_goal,aq_ha_notifications
2* from dba_services where service_id = 8
SQL> /
NAME            METHOD      TYPE        RETRIES GOAL       CLB_GOAL AQNOT
IT              BASIC       SELECT          180 NONE       LONG     YES
lsnrctl service gives
Service "IT" has 1 instance(s).
Instance "hp3", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:36495 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=_SOME_VIP_)(PORT=1521))When prefered instance hp3 goes down (aborts) ,all is working ok, service is relocated to hp4 and users can connect.
But when prefered instance hangs (local disk controller issue) users are not able to connect to that service .
My goal is to have that service on 1 node only, and in case on failure (hard or soft) redirect all conections to that
service on other node (only 1 not load balancing and so) .
How to configure that ?
Oracle pointed out :
How To Configure Server Side Transparent Application Failover [ID 460982.1]
but I dont want to loadbalance connection .
Please advice .
Regards
GregG

Hi,
actually your database should not hang. Because when it hangs on one node this error will impact all the other instances as well, since this node has database blocks in memory it exclusively accesses, and when these cannot be transferred, your other nodes will "wait". Hence it is crucially important that when your disk controller hangs, the database instance gets aborted, if not even the node should get evicted.
So if your described szenario really is the case, then the service definition will not help you fix the issue. You should instead have a look at the reason, why the node does not react (or does not get an error).
If the clusternode behaves correctly then the service will failover. Just make sure you configure your clients correctly to react to this:
http://www.oracle.com/technetwork/products/clustering/overview/awmrac11g-133673.pdf
Regards
Sebastian

Cancel or abort invoke node that hangs up

Hello,
i use an invoke node to grab an image from firewire camera. camera in trigger mode.
application hangs up when trigger signal fails. running VI in highlight execution mode shows that "IICImangingControl, MemorySnapImage" (upside left) hangs up.
VI cannot be cancel with abort execution button.
target: abort invoke node in case of timeout (error handling)
question: how can i abort hanged up invoke node?
timon
Attachments:
GrabPicture 02.tg.vi ‏97 KB

The camera code is an ActiveX/COM DLL that you are calling out to. Unfortunately, when a thread leaves LabVIEW and calls external code, we have no control over it anymore. When you abort the VI, we'll stop the code execution, but we have to wait for the COM DLL to release the thread.
Being able to do this is outside our control, unfortunately. You should look at the camera API, or talk to the vendor, about a method that either has a timeout or is asynchronous in nature.
Brian Tyler
http://detritus.blogs.com/lycangeek

Oracle Grid Infrastructure 11.2.0.2 root.sh hangs in the second node

Hi,
I am installing Oracle Grid Infrastructure 11.2.0.2.0 using VM's. Both the virtual machines are running RHEL 5.7 32 bit version.
Until the point where the installation suggests to run two scripts as root user, everything runs fine.
root.sh runs successfully on the first node.
On the second node it hangs at,
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

user10193468 wrote:
Hi,
Following is the last few lines of the $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_<node>.log file
2012-03-19 17:20:17: OLR successfully created or upgraded
2012-03-19 17:20:17: /u01/app/11.2.0/grid/bin/clscfg -localadd
2012-03-19 17:20:17: Executing /u01/app/11.2.0/grid/bin/clscfg -localadd
2012-03-19 17:20:17: Executing cmd: /u01/app/11.2.0/grid/bin/clscfg -localadd
I am using ASM for OCR and Voting Disks.
Private network is up and running.Please let me know what have you set in asm_diskstring parameter?
I have find following note from metalink which says that you should not set asm_diskstring to '/dev/' or '/dev/*'. This is not supported
root.sh Hangs or Reboots Node on Non First Node if ASM Discovery String asm_diskstring is Set to /dev/ or /dev/* [ID 1356567.1]

Sqlplus hangs up remotely after entering the credentials.

Hi Experts,
Sqlplus hangs remotely after giving the credentials. There are two homes in same Windows machine.Earlier one home was with Oracle 10g R1 and other with R2.
I upgraded the R1 release to R2.
Now the database which was earlier running on R2 has started giving this strange problem when trying to connect remotely, tnsping is successful to this database, sqlplus just hangs up no error nothing.
Has any one faced this kind of problem or knows how to resolve this, please help me out.
Thanks in Advance,
Bhavesh

Remotely :
H:\>sqlplus
SQL*Plus: Release 10.2.0.1.0 - Production on Fri Jun 13 19:04:51 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Enter user-name: system/oracle
ERROR:
ORA-12560: TNS:protocol adapter error
Locally when using @:
C:\Documents and Settings>sqlplus
SQL*Plus: Release 10.2.0.3.0 - Production on Fri Jun 13 18:20:41 2008
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Enter user-name: system@lbu1
Enter password:
Thts the hanged state..

Problem while running root.sh in RAC Node

I am currently configuring RAC on linux using VMWARE Server using the following link .My host Operating System is Windows XP and guest O/S Cent OS.Shared disks have been configured on my C drive for ocr,voting disk,asm.
http://www.oracle-base.com/articles/10g/OracleDB10gR2RACInstallationOnCentos4UsingVMware.php
I have performed all the steps succesfully but during the installation of Clusterware Software from node RAC1 it asks for the execution of root.sh from both the nodes.
Currently I am running root.sh from first node but it gets hanged at "Startup will be queued to init within 90 seconds" and doesn't move from there on..Also according to the link I am following I should be experiencing below output while running root.sh from Node RAC2 instead of RAC1 which I am experiencing....Please suggest on this ...
# cd /u01/crs/oracle/product/10.2.0/crs
# ./root.sh
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/crs/oracle/product' is not owned by root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/crs/oracle/product' is not owned by root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
---------------------------

Will someone update on this?

When we run $CRS_HOME/root.sh scripts-This hangs for a very long time

Hi,
At the time of oracle cluster ware installation, when we run $CRS_HOME/root.sh scripts…
bash-3.00# /export/home/oracle/product/10.2.0/crs/root.sh
WARNING: directory '/export/home/oracle/product/10.2.0' is not owned by root
WARNING: directory '/export/home/oracle/product' is not owned by root
WARNING: directory '/export/home/oracle' is not owned by root
WARNING: directory '/export/home' is not owned by root
WARNING: directory '/export' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/export/home/oracle/product/10.2.0' is not owned by root
WARNING: directory '/export/home/oracle/product' is not owned by root
WARNING: directory '/export/home/oracle' is not owned by root
WARNING: directory '/export/home' is not owned by root
WARNING: directory '/export' is not owned by root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/rdsk/c0d1s1
Format of 1 voting devices complete.
Startup will be queued to init within 30 seconds.
This hangs for a very long time; I controlled out of it and re-run it. The same result, it stops at the last line for hours now. Any idea?
Thanks
ANup
Edited by: user485641 on २५ अप्रैल, २००९ ७:०५ अपराह्न

what os ? what Oracle version?
What you find on cluster log.... ?

Unable to bring up ASM on 2nd node of a 2-node Cluster

Having a very wierd problem on a 2-node cluster. I can only bring up on ASM instance at a time. If i bring up the second, it hangs. This is what the second (hung) instance puts in the alert log:
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /ORAUTL/oraasm/product/ASM/dbs/arch
Autotune of undo retention is turned off.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.3.0.
System parameters with non-default values:
large_pool_size = 12582912
instance_type = asm
cluster_interconnects = 192.168.0.12
cluster_database = TRUE
instance_number = 2
remote_login_passwordfile= EXCLUSIVE
background_dump_dest = /ORAUTL/oraasm/admin/+ASM2/bdump
user_dump_dest = /ORAUTL/oraasm/admin/+ASM2/udump
core_dump_dest = /ORAUTL/oraasm/admin/+ASM2/cdump
pga_aggregate_target = 0
Cluster communication is configured to use the following interface(s) for this instance
192.168.0.12
Fri Nov 21 21:10:48 2008
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=5428
DIAG started with pid=3, OS id=5430
PSP0 started with pid=4, OS id=5432
LMON started with pid=5, OS id=5434
LMD0 started with pid=6, OS id=5436
LMS0 started with pid=7, OS id=5438
MMAN started with pid=8, OS id=5442
DBW0 started with pid=9, OS id=5444
LGWR started with pid=10, OS id=5446
CKPT started with pid=11, OS id=5448
SMON started with pid=12, OS id=5458
RBAL started with pid=13, OS id=5475
GMON started with pid=14, OS id=5487
Fri Nov 21 21:10:49 2008
lmon registered with NM - instance id 2 (internal mem no 1)
Fri Nov 21 21:10:49 2008
Reconfiguration started (old inc 0, new inc 2)
ASM instance
List of nodes:
0 1
Global Resource Directory frozen
Communication channels reestablished
After this it hangs. i've checked everything. CRS is fine.
I suspect its the kernel revision. This is a cluster of two v890's. Kernel rev is 127127-11. Anyone seen this issue ?
thanks

Responses in-line:
Have you got any issue reported from Lock Monitor's (LMON) ? (those messages are in the alert.log are summaries of the reconfiguration event.
No issues that I have seen. I see trc files on both nodes for lmon, but neither contain errors.Do you have any post issues on the date that issue began (something with Reconfiguration started) ?
This is a new build. Its going to be a DR environment (Dataguard Physical Standby), so we've never managed to get ASM up yet.Do you have any other errors on the second node on the date the issue appears (some ORA-27041 or other messages) errors?
No errors at all.What is the result of a crs_stat -t ?
HA Resource Target State
ora.vzdfwsdbp01.LISTENER_VZDFWSDBP01.lsnr ONLINE ONLINE on vzdfwsdbp01
ora.vzdfwsdbp01.gsd ONLINE ONLINE on vzdfwsdbp01
ora.vzdfwsdbp01.ons ONLINE ONLINE on vzdfwsdbp01
ora.vzdfwsdbp01.vip ONLINE ONLINE on vzdfwsdbp01
ora.vzdfwsdbp02.LISTENER_VZDFWSDBP02.lsnr ONLINE ONLINE on vzdfwsdbp02
ora.vzdfwsdbp02.gsd ONLINE ONLINE on vzdfwsdbp02
ora.vzdfwsdbp02.ons ONLINE ONLINE on vzdfwsdbp02
ora.vzdfwsdbp02.vip ONLINE ONLINE on vzdfwsdbp02
ASM isn't registered with CRS/OCR yet. I did add it at one time, but it didnt seem to make any difference.What is the release of your installation 10.2.0.4? Otherwise control if you can upgrade CRS, ASM and your RDBMS to that release.
CRS, ASM and Oracle will be 10.2.0.3Can't go to 10.2.0.4 yet as primary site is at 10.2.0.3 on a live system.
Can you please tell us what is the OS / Hardware in use?
Solaris 10, Sun v890$ uname -a
SunOS dbp02 5.10 Generic_127127-11 sun4u sparc SUNW,Sun-Fire-V890
What is the result of that on the second node:
even a startup nomount hangs on second node.connect sqlplus / as sysdba;
startup nomount
desc v$asmdiskgroup;
select name, mount from v$diskgroup;
In the case that no group is mounted do
alter database mount diskgroup 'your diskgroupname';
What is the result of that?
thanks
-toby

Primary Node Online but State=UNKNOWN ?

Thanks in advance.
Problem: When starting up "rac1" node the result is STATE=UNKNOWN
11g Clusterware and 11g database
rac1.respecti.com>crs_resources
HA Resource Target State
ora.RAC.RAC1.inst ONLINE UNKNOWN on rac1
ora.RAC.RAC2.inst ONLINE ONLINE on rac2
ora.RAC.db ONLINE ONLINE on rac2
ora.rac1.gsd ONLINE ONLINE on rac1
ora.rac1.ons ONLINE ONLINE on rac1
ora.rac1.rac1.lsnr ONLINE ONLINE on rac1
ora.rac1.vip ONLINE ONLINE on rac1
ora.rac2.gsd ONLINE ONLINE on rac2
ora.rac2.ons ONLINE ONLINE on rac2
ora.rac2.rac2.lsnr ONLINE ONLINE on rac2
ora.rac2.vip ONLINE ONLINE on rac2
rac1.respecti.com>
login as: rac
[email protected]'s password:
Last login: Sun Feb 15 13:53:58 2009 from 192.168.1.4
rac1.respecti.com>clear
rac1.respecti.com>crs_resources
HA Resource Target State
ora.RAC.RAC1.inst ONLINE UNKNOWN on rac1
ora.RAC.RAC2.inst ONLINE ONLINE on rac2
ora.RAC.db ONLINE ONLINE on rac2
ora.rac1.gsd ONLINE ONLINE on rac1
ora.rac1.ons ONLINE ONLINE on rac1
ora.rac1.rac1.lsnr ONLINE ONLINE on rac1
ora.rac1.vip ONLINE ONLINE on rac1
ora.rac2.gsd ONLINE ONLINE on rac2
ora.rac2.ons ONLINE ONLINE on rac2
ora.rac2.rac2.lsnr ONLINE ONLINE on rac2
ora.rac2.vip ONLINE ONLINE on rac2
rac1.respecti.com>pwd
/home/rac
rac1.respecti.com>cd /u01/
rac1.respecti.com>ls
app shared_conf shared_crs shared_data shared_home tmp
rac1.respecti.com>cd shared_crs
rac1.respecti.com>ls
bin diagnostics jlib nls perl slax tg4tera
cdata evm JRE OCR plsql sqlplus VOTE
cfgtoollogs has ldap OPatch precomp srvm xdk
clone install lib opmn racg sysman
crs install.platform lib32 oracore rdbms tg4ifmx
css inventory log oraInst.loc relnotes tg4ingr
cv jdk network oui root.sh tg4sybs
rac1.respecti.com>cd crs
rac1.respecti.com>ls
admin demo lib mesg public template
auth init log profile script trace
rac1.respecti.com>cd trace
rac1.respecti.com>ls
rac1.respecti.com>pwd
/u01/shared_crs/crs/trace
rac1.respecti.com>ls
rac1.respecti.com>ls -lt
total 0
rac1.respecti.com>pwd
/u01/shared_crs/crs/trace
rac1.respecti.com>cd ..
rac1.respecti.com>ls -lt
total 48
drwxr-xr-x 7 root dba 4096 Feb 15 20:31 auth
drwxr-xr-x 2 root dba 4096 Feb 14 20:38 init
drwxr-x--T 2 rac dba 4096 Feb 14 19:22 log
drwxrwxrwt 2 rac dba 4096 Jul 13 2008 public
drwxr-xr-x 2 root dba 4096 May 24 2008 profile
drwxr-x--T 2 rac dba 4096 May 24 2008 trace
drwxr-xr-x 2 root dba 4096 May 24 2008 script
drwxr-xr-x 2 rac dba 4096 May 24 2008 admin
drwxr-xr-x 2 rac dba 4096 May 24 2008 lib
drwxr-xr-x 2 rac dba 4096 May 24 2008 mesg
drwxr-xr-x 3 rac dba 4096 May 24 2008 demo
drwxr-xr-x 2 root dba 4096 May 24 2008 template
rac1.respecti.com>cd log
rac1.respecti.com>ls -lt
total 0
-rw-r----- 1 rac dba 0 Nov 30 13:37 startgmJZrY.stdout
-rw-r----- 1 rac dba 0 Nov 30 13:37 startzJeeIX.stderr
rac1.respecti.com>pwd
/u01/shared_crs/crs/log
rac1.respecti.com>cd ..
rac1.respecti.com>ls
admin demo lib mesg public template
auth init log profile script trace
rac1.respecti.com>pwd
/u01/shared_crs/crs
rac1.respecti.com>cd profile
rac1.respecti.com>ls
rac1.respecti.com>pwd
/u01/shared_crs/crs/profile
rac1.respecti.com>cd ..
rac1.respecti.com>pwd
/u01/shared_crs/crs
rac1.respecti.com>ls -alt
total 56
drwxr-xr-x 7 root dba 4096 Feb 15 20:31 auth
drwxr-xr-x 2 root dba 4096 Feb 14 20:38 init
drwxr-x--T 2 rac dba 4096 Feb 14 19:22 log
drwxrwxrwt 2 rac dba 4096 Jul 13 2008 public
drwxr-xr-x 43 root dba 4096 May 27 2008 ..
drwxr-xr-x 2 root dba 4096 May 24 2008 profile
drwxr-xr-x 14 root dba 4096 May 24 2008 .
drwxr-x--T 2 rac dba 4096 May 24 2008 trace
drwxr-xr-x 2 root dba 4096 May 24 2008 script
drwxr-xr-x 2 rac dba 4096 May 24 2008 admin
drwxr-xr-x 2 rac dba 4096 May 24 2008 lib
drwxr-xr-x 2 rac dba 4096 May 24 2008 mesg
drwxr-xr-x 3 rac dba 4096 May 24 2008 demo
drwxr-xr-x 2 root dba 4096 May 24 2008 template
rac1.respecti.com>pwd
/u01/shared_crs/crs
rac1.respecti.com>cd ..
rac1.respecti.com>ls
bin diagnostics jlib nls perl slax tg4tera
cdata evm JRE OCR plsql sqlplus VOTE
cfgtoollogs has ldap OPatch precomp srvm xdk
clone install lib opmn racg sysman
crs install.platform lib32 oracore rdbms tg4ifmx
css inventory log oraInst.loc relnotes tg4ingr
cv jdk network oui root.sh tg4sybs
rac1.respecti.com>pwd
/u01/shared_crs
rac1.respecti.com>cd log
rac1.respecti.com>ls
crs rac1 rac2
rac1.respecti.com>cd crs
rac1.respecti.com>ls
rac1.respecti.com>cd ../rac1
rac1.respecti.com>ls
admin alertrac1.log client crsd cssd evmd racg
rac1.respecti.com>cd crsd
rac1.respecti.com>ls
crsd.log crsdOUT.log
rac1.respecti.com>ls -lt
total 1424
-rw-r--r-- 1 root dba 1252906 Feb 14 21:29 crsd.log
-rw-r--r-- 1 root dba 193481 Feb 14 20:38 crsdOUT.log
rac1.respecti.com>vi /u01/shared_crs/log/rac1/crsd/crsd.log
2009-02-14 21:13:36.729: [ CRSRES][1484962112] Attempting to start `ora.RAC.RAC1.inst` on member `rac1`
2009-02-14 21:13:38.455: [ OCRUTL][1377945920]u_freem: mem passed is null
2009-02-14 21:29:21.741: [ CRSEVT][1484962112] CAAMonitorHandler :: 0:Could not join /u01/app/product/RAC/bin/racgwrap(start)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
2009-02-14 21:29:21.746: [ CRSEVT][1484962112] CAAMonitorHandler :: 0:Action Script /u01/app/product/RAC/bin/racgwrap(start) timed out for ora.RAC.RAC1.inst! (timeout=900)
2009-02-14 21:29:21.754: [ CRSAPP][1484962112] StartResource error for ora.RAC.RAC1.inst error code = -2
2009-02-14 21:29:24.407: [ CRSAPP][1484962112] StopResource error for ora.RAC.RAC1.inst error code = 1
2009-02-14 21:29:24.449: [ CRSRES][1484962112] X_OP_StopResourceFailed : Stop Resource failed
(File: rti.cpp, line: 1839
2009-02-14 21:29:24.451: [ CRSRES][1484962112][ALERT] `ora.RAC.RAC1.inst` on member `rac1` has experienced an unrecoverable failure.
2009-02-14 21:29:24.455: [ CRSRES][1484962112] Human intervention required to resume its availability.
14771,1 Bot
Edited by: yakub21 on Feb 15, 2009 8:54 PM
rac1 node database is accessible, though I would like to know how to resolve the clusterware problem.

Hi Yakub,
is this problem persistent? Does it happen every time you start node1 or is this the status after the last start?
What happened is that Oracle Clusterware got the impression that the RAC instance on node1 was not started according to some tests in the action script that is used to start the instance. Actually, some operations did not return in a timely fashion (hence, the time out in the crsd.log). According to what you said, the instance was however started - I presume a ps -ef would show that and connections against this instance are possible!?
Now, since Oracle Clusterware encountered some time outs, it cannot determine the state of the instance correctly and therefore declares the state of the resource (your rac instance) as UNKNOWN. This means: no monitoring of this resource and hence no corrective action in case of failures of this resource. In other words: for Clusterware, this instance is unmanaged.
What can you do to fix that? Well, "human intervention" is required. My suggestion would be: stop the instance explicitly (preferably using SRVCTL) and restart it using SRVCTL. After stopping the instance using that command, check the state from Clusterware perspective. It should be TARGET=OFFLINE. Make sure, your instance is down. Kill / shutdown abort the running RAC instance explicitly before attempting a restart.
Hope that helps. Thanks,
Markus

Sqlplus is hanging on RAC1 node

Similar Messages

Maybe you are looking for