RAC abnormal shutdown

Hello All,
We are facing an unusual scenario.
We are having a 3 nodes RAC on Oracle 10.2.0.2.0. 2 node were shutdown abnormally. Can you please help in finding the rootcause of the problem?
All the services has to be manually started
Find below the messages in Background Trace Files :
Thu Dec 02 10:38:36 2010
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
* domain 0 not valid according to instance 2
Thu Dec 02 10:38:36 2010
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu Dec 02 10:38:36 2010
LMS 1: 63 GCS shadows cancelled, 2 closed
Thu Dec 02 10:38:36 2010
LMS 0: 61 GCS shadows cancelled, 1 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Thu Dec 02 10:38:38 2010
Instance recovery: looking for dead threads
Thu Dec 02 10:38:38 2010
LMS 1: 23117 GCS shadows traversed, 4001 replayed
Thu Dec 02 10:38:38 2010
LMS 0: 23361 GCS shadows traversed, 4001 replayed
LMS 0: 23052 GCS shadows traversed, 4001 replayed
Thu Dec 02 10:38:39 2010
LMS 1: 23922 GCS shadows traversed, 4001 replayed
Thu Dec 02 10:38:39 2010
LMS 0: 23388 GCS shadows traversed, 4001 replayed
Thu Dec 02 10:38:39 2010
LMS 1: 23088 GCS shadows traversed, 4001 replayed
LMS 1: 23268 GCS shadows traversed, 4001 replayed
LMS 1: 23621 GCS shadows traversed, 4001 replayed
LMS 1: 22885 GCS shadows traversed, 4001 replayed
LMS 1: 23061 GCS shadows traversed, 4001 replayed
LMS 1: 23046 GCS shadows traversed, 4001 replayed
LMS 1: 24090 GCS shadows traversed, 4001 replayed
LMS 1: 23329 GCS shadows traversed, 4001 replayed
Thu Dec 02 10:38:39 2010
Beginning instance recovery of 1 threads

user3601721 wrote:
We are having a 3 nodes RAC on Oracle 10.2.0.2.0. 2 node were shutdown abnormally. Can you please help in finding the rootcause of the problem? This requires more that a snippet of part of the alert log of a single instance.
Why was the instances shutdown? Did they shutdown themselves, or were they simply killed? What does the kernel log say? What do the CRS and CSS logs say? Were there issues with the storage layer (what do you use as cluster storage layer)? Were there issues with the Interconnect (what do you use for the Interconnect)? Is ASM used? Etc. etc.
Why the manual start-up? Did this include restarting CRS or just the RAC instances? Or were the servers rebooted? What did the manual start-up entail?

Similar Messages

Cannot log in to Mobile Manager after abnormal shutdown (issue & solution)

After an abnormal shutdown of database, an attempt to log in to Mobile Manager as Administrator fails with the error "Please verify your username, password and try again!"
I had a power outage in our office and our development server shut down abruptly as a result. When power was restored, the database, listener and GlassFish Server started up automatically (init.d and rc.d), but when I attempt to log in as Administrator using Mobile Manager, I get an error "Please verify your username, password and try again!". I did some research and after some trial and error, figured out that the services must be started in this order (this may not be guaranteed in automatic startup scripts?):
1. Oracle Listener
2. Oracle Database
3. Oracle GlassFish Server with Domain
If you are seeing this error, please try the following:
1. Shutdown Glassfish domain
./asadmin stop-domain <domain>
2. Shutdown Oracle database
SQL> SHUTDOWN IMMEDIATE
3. Shutdown Listener
$ lsnrctl stop
and restart in this order
1. Oracle Listener
lsnrctl start
2. Oracle database
SQL> STARTUP
3. GlassFish domain
./asadmin start-domain <domain>
This should work!
My environment:
Redhat Enterprise Linux 5.4 with JDK 1.6
Oracle Database 11g Enterprise Edition 11.1.0.1.0
Oracle GlassFish Server 3.1.2
Oracle Database Mobile Server 11.1.0 for 64-bit Linux

Hi mario,
FYI
This issue can occurs when the primary and secondary Cisco ISE nodes' database are out of sync. For out of sync issues, which most likely are due to time changes or NTP sync issues, you must correct the system time and perform a manual sync up through the UI.
•For certificate expiry issues, you must install a valid certificate and perform a manual sync up through the UI.
•For a node that has been down for more than six hours, you must restart the node, check for connectivity issues, and perform a manual sync up through the UI.
For more information regarding this issue, please go through this link:
http://www.cisco.com/c/en/us/td/docs/security/ise/1-2/troubleshooting_guide/ise_tsg.html#wp192802

Hyperion Planning application got abnormal shutdown

Hi,
Would need your help on the issue I recently hit in my environment.
Product version: 9.3.1.1.00
Product: Hyperion Planning
Unix Platform: Sun Solaries 64bit
Issue:-
The application received the Abnormal Shutdown command then the application got shutdown and then not starting up by itself after that.
Query:-
Is there anyway to set it to be started automatically even after abnormal shutdown?
Thanks.
SuSin

Hi Sandeep,
The xcp files are in the application directory.
I've raised SR in Metalink but the support request me to change the Essbase.cfg on the NETRETRY and NETRETRYCOUNT from 5000 and 3000 respectively to 1000 each and also the SERVERTHREAD from 200 change to 100.
I've read from one of the reply in the forum saying that if the xcp files are in application directory then it does not really related to the Essbase.cfg unless it is from the server directory. So if in this case, should I still modify the config file?
Hi John,
Previously in Jan, it happended once where users are not able to retrieve the data from HyperionPlanning and also Workspace. So after that incident actually we did clear the data and reload the level-0 data. The system was OK (no more Abnormal Shutdown) for a while then came back with this issue again recently.
Anything can be apply in order to clear this problem?
Thanks.

Boot fsck from abnormal shutdown

Hi,
Since 2.6.31 my computer is unable to do a fsck when a abnormal shutdown occured (e.g. a powerout), and a second reboot is always needed because the system prompt me "give the root password or press CTRL-D to proceed" error. After pressing CTRL-D, the system reboots and do fsck at the second boot.
How can I restore to the mode that's been used before, which just do fsck automatically immediately once if an abnormal shutdown and a unclean filesystem is detected ?

This occured to me also, but I'm still using kernel 2.6.30.5
The problem seems to be elsewhere ...
At first I didn't thought this was a bug, but now that I read your post, maybe we should file a bug-report.
See also : http://bbs.archlinux.org/viewtopic.php?id=80564
Last edited by john_schaf (2009-10-02 08:14:09)

RECEIVED ABNORMAL SHUTDOWN COMMAND - APPLICATION TERMINATING

HI All,
We have an Planning application and when the user is trying to perform some activity and ti involves in running some calculation Scripts.
At point the application is going down and this is causing real mess.
There is no .xcp file created in the app folder. We are unable to find the root cause of the issue.
the following are the fixes we tried but gave us no result.
We tried to increase the Java Heep size
<variable id="ESS_CSS_JVM_OPTION8" value="-Xms512m"/>
<variable id="ESS_CSS_JVM_OPTION9" value="-Xmx1024m"/>
But the issue still exists.
[Mon Nov 4 11:24:32 2013]Local/SMARTPln///1127704896/Info(1042059)
Connected from [140.85.4.227]
[Mon Nov 4 11:24:32 2013]Local/SMARTPln/SMARTPln/gbeaton@CAL1/1127704896/Info(1020089)
Ignoring span Hybrid Analysis option. Spanning into Hybrid Analysis Relational Source has been disabled. See the essbase.cfg file
[Mon Nov 4 11:24:32 2013]Local/SMARTPln/SMARTPln/gbeaton@CAL1/1127704896/Info(1020055)
Spreadsheet Extractor Elapsed Time : [0.01] seconds
[Mon Nov 4 11:24:32 2013]Local/SMARTPln/SMARTPln/gbeaton@CAL1/1127704896/Info(1020082)
Spreadsheet Extractor Big Block Allocs -- Dyn.Calc.Cache : [204] non-Dyn.Calc.Cache : [0]
[Mon Nov 4 11:24:32 2013]Local/SMARTPln///1222445376/Info(1013210)
User [hypadmin@Native Directory] set active on database [Wrkforce]
[Mon Nov 4 11:24:32 2013]Local/SMARTPln///1130862912/Info(1042059)
Connected from [140.85.4.227]
[Mon Nov 4 11:24:32 2013]Local/SMARTPln/Wrkforce/hypadmin@Native Directory/1130862912/Info(1013091)
Received Command [MdxReport] from user [hypadmin@Native Directory]
[Mon Nov 4 11:24:32 2013]Local/SMARTPln/Wrkforce/hypadmin@Native Directory/1130862912/Info(1260039)
MaxL DML Execution Elapsed Time : [0] seconds
[Mon Nov 4 11:24:32 2013]Local/SMARTPln///1132968256/Info(1042059)
Connected from [140.85.4.227]
[Mon Nov 4 11:24:32 2013]Local/SMARTPln/Wrkforce/hypadmin@Native Directory/1132968256/Info(1013091)
Received Command [MdxReport] from user [hypadmin@Native Directory]
[Mon Nov 4 11:24:32 2013]Local/SMARTPln/Wrkforce/hypadmin@Native Directory/1132968256/Info(1260039)
MaxL DML Execution Elapsed Time : [0] seconds
[Mon Nov 4 11:24:32 2013]Local/SMARTPln///1129810240/Info(1042059)
Connected from [140.85.4.227]
[Mon Nov 4 11:24:32 2013]Local/SMARTPln/Wrkforce/hypadmin@Native Directory/1129810240/Info(1020055)
Spreadsheet Extractor Elapsed Time : [0] seconds
[Mon Nov 4 11:24:32 2013]Local/SMARTPln/Wrkforce/hypadmin@Native Directory/1129810240/Info(1020082)
Spreadsheet Extractor Big Block Allocs -- Dyn.Calc.Cache : [87] non-Dyn.Calc.Cache : [0]
[Mon Nov 4 11:24:35 2013]Local/SMARTPln///1134020928/Info(1042059)
Connected from [140.85.4.227]
[Mon Nov 4 11:24:35 2013]Local/SMARTPln/SMARTPln/gbeaton@CAL1/1134020928/Info(1013091)
Received Command [SetAlias] from user [gbeaton@CAL1]
[Mon Nov 4 11:24:35 2013]Local/SMARTPln///1135073600/Info(1042059)
Connected from [140.85.4.227]
[Mon Nov 4 11:24:35 2013]Local/SMARTPln/SMARTPln/gbeaton@CAL1/1135073600/Info(1020089)
Ignoring span Hybrid Analysis option. Spanning into Hybrid Analysis Relational Source has been disabled. See the essbase.cfg file
[Mon Nov 4 11:24:35 2013]Local/SMARTPln/SMARTPln/gbeaton@CAL1/1135073600/Info(1020055)
Spreadsheet Extractor Elapsed Time : [0] seconds
[Mon Nov 4 11:24:35 2013]Local/SMARTPln/SMARTPln/gbeaton@CAL1/1135073600/Info(1020082)
Spreadsheet Extractor Big Block Allocs -- Dyn.Calc.Cache : [7] non-Dyn.Calc.Cache : [0]
[Mon Nov 4 11:24:38 2013]Local/SMARTPln///1128757568/Info(1042059)
Connected from [140.85.4.227]
[Mon Nov 4 11:24:38 2013]Local/SMARTPln/SMARTPln/gbeaton@CAL1/1128757568/Info(1013091)
Received Command [SetAlias] from user [gbeaton@CAL1]
[Mon Nov 4 11:24:38 2013]Local/SMARTPln///1131915584/Info(1042059)
Connected from [140.85.4.227]
[Mon Nov 4 11:24:38 2013]Local/SMARTPln/SMARTPln/gbeaton@CAL1/1131915584/Info(1020089)
Ignoring span Hybrid Analysis option. Spanning into Hybrid Analysis Relational Source has been disabled. See the essbase.cfg file
[Mon Nov 4 11:24:38 2013]Local/SMARTPln///1131915584/Info(1002089)
RECEIVED ABNORMAL SHUTDOWN COMMAND - APPLICATION TERMINATING
[Mon Nov 4 11:24:52 2013]Local/SMARTPln///1218234688/Error(1013204)
Client Commands are Currently Not Being Accepted
Can you please help me out form this .
Regards,
Kiran.

Hi Mady,
Thanks for the reply.
We have increased the Heap size and restarted the services. But this issue still exists. And also restarted the entire system but the result is the same.
We are unable to find the root cause which is causing this issue.
Can any one suggest me....
Thanks in Advance.
Regards,
Kiran.

Finalizers + abnormal shutdown

I have read some notes about this, but I want to be sure: Some people in this forum have written that on an abnormal shutdown of the VM, not all finalizers are called. I have some questions about it:
1. Is this true?
2. Is it an implementation bug, or allowed by the VM spec / JLS? I have found this in the JLS:
The Java programming language does not specify how soon a finalizer will be invoked, except to say that it will happen before the storage for the object is reused.
Since storage for the object will certainly be re-used after termination of the VM, finalizers must be called before shutdown according to the JLS, right? Any pointer to contrary information would be nice.
3. If finalizers are not guaranteed to be called, what is their use? The typical action of a finalizer would be to de-allocate some system resource (close a file or network connection, release the graphics or sound hardware, etc). No sane programmer would leave such important tasks to an unreliable mechanism. So if finalizers are unreliable, where are they useful?
Thanks in advance.

What do you mean by "abnormal shutdown of the JVM"?I do not mean killing the VM process at the OS level. It's something that comes from "inside the VM". But I don't know what it is exactly, as I was only pointed in that direction by posts in this forum. System.exit? Exceptions? Don't know what.

Abnormal Shutdown in App when conducting training

Every time we conduct an Essbase Excel Add-In training class, the application that we are training in abnormally shuts down many times. The app does not appear to be corrupted and does not have problems outside of the class times. None of our other apps have problems during these times. Our classroom pc's are using network hubs and I'm guessing that these abnormal shutdowns have something to do with the fact that so many users are trying to touch the server/app at the same time going through the same network connection. The .xcp file appears to be almost identical every time. Below is an exerpt from the .xcp file:----- Exception Error Log Begin -----Current Date & Time: Thu Mar 20 13:11:41 2003Process Type: ApplicationApplication Name: SldfinDatabase Name: SldfinException Log File: D:\HYPERION\ESSBASE\app\Sldfin\Sldfin\log00012.xcpCurrent Thread Id: 514Exception Code: 0xC0000005=Access ViolationException Flags: 0x00000000=ContinuableException Address: 0x00523427Exception Parameters: 2Exception Parameter 0: 0x00000000=Read Violation Exception Parameter 1: 0x00000020 (Virtual Address)We have NETDELAY 5000 and NETRETRYCOUNT 1000 set in our essbase.cfg. Our server version is 6.2.1, the client versions are 6.0 and 6.2.1 (happens with either client version).Has anyone else experienced this? If so, is there any way to change the essbase.cfg or server o/s (Win NT) settings to avoid this problem?Thanks in advance.

When your sever is dead, there isn't anything there to run and all the connections are lost.
On the client side you will have to look for the disconnect, timeout, or some other exception so you know not to continue trying to connect to a dead server socket.

Abnormal Shutdown...

My Server turned off abnormally last night, We have started it later, Now, I opened and checked the Alert.Log file, but I unable to find any shutdown Information, that happened last Night.
Can't I know the reasons of Shutdown? Where I can find such Information.?
Regards!!

Server , Actually, System Admin Dept. called me and asked any information that has been stored in aler file regarding last night abnormal shutdown.
I can find Only starting Information.
THanks Adith for reply!!

Abnormal shutdown of database

Hai All,
This alert log content from my production database. what is the cause of the abnormal shutdown.After the shutdown, the database can startup normally.
Tue Aug 7 04:59:23 2007
Thread 1 advanced to log sequence 1376
Current log# 1 seq# 1376 mem# 0: /home/oracle/oracle/product/10.2.0/oradata/dsoft/redo01.log
Tue Aug 7 09:00:53 2007
Thread 1 advanced to log sequence 1377
Current log# 2 seq# 1377 mem# 0: /home/oracle/oracle/product/10.2.0/oradata/dsoft/redo02.log
Tue Aug 7 11:46:29 2007
Shutting down instance (abort)
License high water mark = 132
Instance terminated by USER, pid = 28548
Tue Aug 7 11:46:32 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_10 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Please help..

Hi Dear,
I feel this instance is terminated with shu abort command by
Instance terminated by USER, pid = 28548You should check who is this user who has sysdba privilages.........It can be really very dangerous.
Regards
Amit Raghuvanshi

2 Node RAC abnormal behaviour

Platform: "HP-UX 11.23 64-bit"
Database: "10.2.0.4 64-bit"
RAC: 2 Node RAC setup
Our RAC setup has been properly done and RAC is working fine with load balancing i.e clients are getting connection on both instances. BUT the issue I am facing with my RAC setup is High Availability testing. When I send reboot signal to "Node-2" and the "Node-1" is up what I observe and receive complain from clients that they have lost connection with database ALSO no new connections are being allowed. When I see the alert log of "Node-1" I see the following abnormal messages reported in it:
List of nodes:
0 1
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Tue Aug 9 04:02:15 2011
LMS 2: 0 GCS shadows cancelled, 0 closed
Tue Aug 9 04:02:15 2011
LMS 0: 0 GCS shadows cancelled, 0 closed
Tue Aug 9 04:02:15 2011
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Tue Aug 9 04:02:15 2011
LMS 1: 1908 GCS shadows traversed, 1076 replayed
Tue Aug 9 04:02:15 2011
LMS 2: 1911 GCS shadows traversed, 1086 replayed
Tue Aug 9 04:02:15 2011
LMS 0: 1899 GCS shadows traversed, 1164 replayed
Tue Aug 9 04:02:15 2011
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Tue Aug 9 04:02:16 2011
ARCH shutting down
ARC2: Archival stopped
Tue Aug 9 04:02:21 2011
Redo thread 2 internally enabled
Tue Aug 9 04:02:35 2011
Reconfiguration started (old inc 4, new inc 6)
List of nodes:
0
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Tue Aug 9 04:02:35 2011
LMS 1: 0 GCS shadows cancelled, 0 closed
Tue Aug 9 04:02:35 2011
LMS 2: 0 GCS shadows cancelled, 0 closed
Tue Aug 9 04:02:35 2011
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Tue Aug 9 04:02:35 2011
Instance recovery: looking for dead threads
Tue Aug 9 04:02:35 2011
Beginning instance recovery of 1 threads
Tue Aug 9 04:02:35 2011
LMS 1: 1908 GCS shadows traversed, 0 replayed
Tue Aug 9 04:02:35 2011
LMS 2: 1907 GCS shadows traversed, 0 replayed
Tue Aug 9 04:02:35 2011
LMS 0: 1899 GCS shadows traversed, 0 replayed
Tue Aug 9 04:02:35 2011
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Tue Aug 9 04:02:37 2011
parallel recovery started with 11 processes
Tue Aug 9 04:02:37 2011
Started redo application at
Thread 2: logseq 6, block 2, scn 1837672332
Tue Aug 9 04:02:37 2011
Errors in file /u01/app/oracle/product/10.2.0/db/admin/BAF/bdump/baf1_smon_10253.trc:
ORA-00600: internal error code, arguments: [kcratr2_onepass], [], [], [], [], [], [], []
Tue Aug 9 04:02:38 2011
Errors in file /u01/app/oracle/product/10.2.0/db/admin/BAF/bdump/baf1_smon_10253.trc:
ORA-00600: internal error code, arguments: [kcratr2_onepass], [], [], [], [], [], [], []
Tue Aug 9 04:02:38 2011
Errors in file /u01/app/oracle/product/10.2.0/db/admin/BAF/bdump/baf1_smon_10253.trc:
ORA-00600: internal error code, arguments: [kcratr2_onepass], [], [], [], [], [], [], []
SMON: terminating instance due to error 600
Tue Aug 9 04:02:38 2011
Dump system state for local instance only
System State dumped to trace file /u01/app/oracle/product/10.2.0/db/admin/BAF/bdump/baf1_diag_10229.trc
Tue Aug 9 04:02:38 2011
Instance terminated by SMON, pid = 10253
Tue Aug 9 04:04:09 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 lan3 192.168.1.0 configured from OCR for use as a cluster interconnect
Interface type 1 lan2 172.20.21.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned off.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 300
sessions = 335
timed_statistics = TRUE
Kindly help me to get rid out of this issue. Waiting for the quick and helpful response from the gurus in the forum. Thanks in advance.
Regards,

if above were really 100% correct, you would not be here posting about errors!Definitely but these situations could become the cause for new BUGS, isn't it?
I don't know what is real & what is unnecessary obfuscation.What part of the thread you didn't understand.
It is not a good idea to have subtraction sign/dash character as part of object/host name; i.e. "Node-1"."Node-1" is not the hostname it is just to make clear understanding. the hostname is "sdupn101" for node-1 and "sdupn102" for node-2.
ORA-00600/ORA-07445/ORA-03113 = Oracle bug => search on Metalink and/or call Oracle supportNewbie is my status on this forum but I have little bit ethics of using forums and suppot blogs. I searched but unfortunately didn't find any matching solution.
Anyway will update you once find any solution so that you can assist someone else in future.

Need to shutdown GG process if 1 node of the RAC is shutdown?

if only 1 node of the 2-node RAC cluster is shutdown, Does GG have to come down as well or can it run against the 1 node that is up.
Thanks

Apologize for wrong answer to this post, due to site network issues wrong page was opened, hence wrong answer was posted earlier.
Hi,
If one of the node is down, OGG Extract also breaks down as it is unable to process that particular redo thread. As you are aware, in a RAC environment, the Extract process uses threads, one for each redo thread in the RAC cluster.
Hence the only way to restart the Extract is to exclude the node(redo thread) by using THREADOPTIONS PROCESSTHREADS EXCEPT <thread spec>.
Example: THREADOPTIONS PROCESSTHREADS EXCEPT 2
Extract threads are mapped to redo threads, so while excluding the thread please verify the mapping correctly and exclude the right thread.
SQL> select distinct THREAD# from gv$log;
Caution: Excluding any of the Extract threads from being processed excludes that data from being synchronized with the target tables.
Once the node is back online, you could comment out the above parameter in the Extract parameter file and restart the Extract to process all the redo threads in RAC.
Hope this information helps.
Thanks & Regards
SK
Edited by: Santhosh on Aug 2, 2011 9:51 AM

TAF Failover issue when RAC node shutdown

Dear all,
We have a two-node RAC database. We use sqlplus from a client laptop to test RAC TAF failover when one node is being shutdown. And there's a tnsnames.ora file with TAF settings in the client laptop.
First we connect to RAC database via sqlplus, when we are under the "SQL>" command prompt, we type " select instance_name from v$instance; " and we can see what instance we truely connect to. Then we shutdown the node we truely connect to; At the meanwhile, if we type "select instance_name from v$instance;" again right away, sometimes the sqlplus hangs and with no response; but if we wait utill the VIP failover to another node then type "select instance_name from v$instance;" we can see it always show the other node's instance name and we know the session is successfully failover to the healthy node.
My question is :
Does RAC TAF failover can always and "no down time" failover the session to another healthy node? Or there are some circumstances that the session would hang and need to connect again?
Any help would be appreciated.

Hi, thanks for your help.
There are many things you have to do but if you don't have the knowledge will be difficult.Right. The cluster was setup by consultants but we're still trying to pick up basic Oracle knowledge by self study...
Found some messages about eviction in old cssd logs in $ORA_CRS_HOME/log/cssd/. Will further dig into it.
Yes, we tried rebooting different nodes many times in the clusters before, without any problem.
Thanks a lot.
/ST Wong

Abnormal Shutdown and roles missing

Yesterday my database shutdown abnormally.. after opening the connect,dba ansd some other roles missing . and users cannot able to connect .later I give the create session privilage to all users then they can able to connect How it happened? please help
How it happened?
alert log file
Thu Feb 15 09:02:13 2007
ARC1: Completed archiving log 1 thread 1 sequence 4726
Thu Feb 15 09:20:53 2007
Thread 1 advanced to log sequence 4728
Current log# 3 seq# 4728 mem# 0: /data/oradata/wg92/redo05.log
Thu Feb 15 09:20:53 2007
ARC0: Evaluating archive log 4 thread 1 sequence 4727
ARC0: Beginning to archive log 4 thread 1 sequence 4727
Creating archive destination LOG_ARCHIVE_DEST_1: '/oraarc/wg92/arch/1_4727.dbf'
Thu Feb 15 09:21:37 2007
ARC0: Completed archiving log 4 thread 1 sequence 4727
Thu Feb 15 10:13:20 2007
Thread 1 advanced to log sequence 4729
Current log# 1 seq# 4729 mem# 0: /data/oradata/wg92/redo06.log
Thu Feb 15 10:13:20 2007
ARC1: Evaluating archive log 3 thread 1 sequence 4728
ARC1: Beginning to archive log 3 thread 1 sequence 4728
Creating archive destination LOG_ARCHIVE_DEST_1: '/oraarc/wg92/arch/1_4728.dbf'
ARC1: Completed archiving log 3 thread 1 sequence 4728
Thu Feb 15 13:51:38 2007
Shutting down instance (abort)
License high water mark = 135
Instance terminated by USER, pid = 7740
Thu Feb 15 13:48:50 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 3
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.1.0.
System parameters with non-default values:
processes = 150
timed_statistics = TRUE
shared_pool_size = 117440512
large_pool_size = 16777216
java_pool_size = 117440512
control_files = /data1/oradata/wg92/control01.ctl, /data2/oradata/wg92/control02.ctl, /data3/oradata/wg92/control03.ctl
db_block_size = 8192
db_cache_size = 318767104
compatible = 9.2.0.0.0
log_archive_start = TRUE
log_archive_dest = /oraarc/wg92/arch
db_file_multiblock_read_count= 16
fast_start_mttr_target = 300
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 10800
max_enabled_roles = 100
remote_login_passwordfile= NONE
db_domain =
instance_name = wg92
dispatchers = (PROTOCOL=TCP) (SERVICE=wg92XDB)
job_queue_processes = 10
hash_join_enabled = TRUE
background_dump_dest = /oracle/app/product/9.2.0/oracle9/admin/wg92/bdump
user_dump_dest = /oracle/app/product/9.2.0/oracle9/admin/wg92/udump
core_dump_dest = /oracle/app/product/9.2.0/oracle9/admin/wg92/cdump
sort_area_size = 524288
db_name = wg92
open_cursors = 300
star_transformation_enabled= FALSE
query_rewrite_enabled = FALSE
pga_aggregate_target = 25165824
aq_tm_processes = 1
PMON started with pid=2
DBW0 started with pid=3
LGWR started with pid=4
Please help

later I give the create session privilage to all users With what privilge do you logged on yourself to grant privileges?
users cannot able to connect Which users cannot connect. Do you mean all users excluding sys?
If you start your database in restricted mode then users with restricted session privilege can only connect. Is this your case?
Message was edited by:
Vishal V.

Abnormal shutdown of JVM cause server unreachable

Hi all,
My application is based on client-server and work fine in normal condition.When my server module shutdown abnormally (cause system shutdown or JVM abnormal termination)so my all clients hangs up due to server unreachable.
I have a method for normal shutdown of my server module that cause all client module logout properly.And it work only when I shutdown my server module by given control (Like stop server button).If there is any event that fire at the time when JVM is shutdown by using it I will run my properly shutdown method.
Thanks in advance

When your sever is dead, there isn't anything there to run and all the connections are lost.
On the client side you will have to look for the disconnect, timeout, or some other exception so you know not to continue trying to connect to a dead server socket.

What startup command should be writen after abnormal shutdown?

Hi all
I have a 10g DB instance on Unix server.
The server was shutdown without first shutting down the DB and Listener.
Now when I started the server, should I type:
sqlplus: sys as sysdba
startup
in normal mode or nomount mode?
Is there anything else to be done after startup?
Please mention the reason!
Thank you

Well, yes all db files are autoextend
When I typed the command you gave, it showed AUTO YESGood.
But i'll have to work on making the size unlimited.Use this command:
SQL> ALTER DATABASE DATAFILE '<FULL Path to Datafile>' AUTOEXTEND ON MAXSIZE UNLIMITED;Note: 468096.1 - How To Check For Autoextensible Datafiles Set To Maxsize Unlimited
https://metalink2.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=468096.1

RAC abnormal shutdown

Similar Messages

Maybe you are looking for