Node 2 in RAC 11gr2 went down and hasn't been able to restart. Help pls.

Hello all,
I'm working on this RAC configuration 11gR2 updated to the last patchset. During the night everything on node 2 went down and hasn't been able to come back, not even after a reboot. I've checked through the logfiles and I think the problem might be traceable to this:
vi ocssd.log2013-01-15 02:42:47.629: [    CSSD][1104660800]clssnmvDHBValidateNcopy: node 1, dbacc1, has a disk HB, but no network HB, DHB has rcfg 247316625, wrtcnt, 18953467, LATS 165822984, lastSeqNo 18949942, uniqueness 1357546658, timestamp 1358214167/666569324
… after some time...
2013-01-15 02:43:01.929: [    CSSD][1115699520]###################################
2013-01-15 02:43:01.929: [    CSSD][1115699520]clssscExit: CSSD aborting from thread clssnmRcfgMgrThread
2013-01-15 02:43:01.929: [    CSSD][1115699520]###################################
2013-01-15 02:43:01.929: [    CSSD][1115699520](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally.
crsctl check cluster -all**************************************************************
dbacc1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
dbacc2:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
I also tried to ping the vip from one machine to the other and vice versa and they all answered fine, also with huge size ping (i've found that this could have spotted a possibile bug).
Any help would be really really appreciated. Thanks.
Edited by: Klawd on 16-gen-2013 9.29

crsctl stat res -t -init on node 1
NAME           TARGET  STATE        SERVER                 STATE_DETAILS
Cluster Resources
ora.asm
      1        ONLINE  ONLINE       dbacc1                 Started
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       dbacc1
ora.crf
      1        ONLINE  ONLINE       dbacc1
ora.crsd
      1        ONLINE  ONLINE       dbacc1
ora.cssd
      1        ONLINE  ONLINE       dbacc1
ora.cssdmonitor
      1        ONLINE  ONLINE       dbacc1
ora.ctssd
      1        ONLINE  ONLINE       dbacc1                 OBSERVER
ora.diskmon
      1        OFFLINE OFFLINE
ora.drivers.acfs
      1        ONLINE  ONLINE       dbacc1
ora.evmd
      1        ONLINE  ONLINE       dbacc1
ora.gipcd
      1        ONLINE  ONLINE       dbacc1
ora.gpnpd
      1        ONLINE  ONLINE       dbacc1
ora.mdnsd
      1        ONLINE  ONLINE       dbacc1crsctl stat res -t -init on node 2
NAME           TARGET  STATE        SERVER                 STATE_DETAILS
Cluster Resources
ora.asm
      1        ONLINE  OFFLINE
ora.cluster_interconnect.haip
      1        ONLINE  OFFLINE
ora.crf
      1        ONLINE  ONLINE       dbacc2
ora.crsd
      1        ONLINE  OFFLINE
ora.cssd
      1        ONLINE  OFFLINE
ora.cssdmonitor
      1        ONLINE  ONLINE       dbacc2
ora.ctssd
      1        ONLINE  OFFLINE
ora.diskmon
      1        OFFLINE OFFLINE
ora.drivers.acfs
      1        ONLINE  ONLINE       dbacc2
ora.evmd
      1        ONLINE  OFFLINE
ora.gipcd
      1        ONLINE  ONLINE       dbacc2
ora.gpnpd
      1        ONLINE  ONLINE       dbacc2
ora.mdnsd
      1        ONLINE  ONLINE       dbacc2Edited by: Klawd on 16-gen-2013 11.49
Edited by: Klawd on 16-gen-2013 12.18

Similar Messages

  • During BIOS update Satellite Pro A60 went down and hasn't started

    Hello everyone,
    I have a Satellite Pro A60 and I updated the bios. Halfway the update, my laptop went down and hasn´t started since. All it does is give a couple of beeps (trying to tell me the bios is corrupted, I suppose) and shuts itself off again.
    Tried to start with the (~) key pressed down (a tip I got from a friend), but no succes.
    I currently live in a country where there is no Toshiba dealer available and I rather not ship the laptop to have it repaired.
    Does anyone know of a way to fix this? Any help would be greatly appreciated.
    Thanks

    This sounds not good.
    If during the POST the BIOS detects a problem it will normally display a visual error message on the monitor explaining what the problem is. However, if a problem is detected before the BIOS initializes the video card or a video card is not present or not detected then the BIOS will play several BEEPS through the speaker to let you know there is a problem.
    Depending on the type of the BIOS you have the BIOS may play beeps in a specific pattern to indicate what the problem is, or play the same beep a number of times indicating the problem.
    In your case its possible that you have used a wrong BIOS or a wrong BIOS procedure. Both can damage the motherboard.
    To prevent such damages and any mistakes Toshiba has published the FAQ document.
    How to update the BIOS on most units!
    This you should read before doing any BIOS updates!
    Unfortunately I dont think that you can solve it yourself. I think you should ask the ASP for help.

  • Tried to restart computer and hasn't been able to turn on since

    My computer was working just fine. I shut down it down for about an hour and when I tried to turn it back on nothing happened, no noise, no lights, nothing. I left it unplugged for a while and tried the SMU restart thing but nothing helped. Does anyone have any information that might help me? Thanks in advance.

    hello
    you are right, it is very likely a PSU problem. i have just repaired my iMac G5 which had a similar problem. Please have a look here at this thread:
    http://discussions.apple.com/message.jspa?messageID=9285919#9285919
    What you can do: open the iMac (see apple website) and look at the LEDs (see the link to the Apple downloads in the thread mentioned). Have a look at the website of Jimwarolic (mentioned in the same thread). This might indicate to you whether it's the PSU or not.
    Also: the link in the thread gives you an idea how you can repair this.
    Hope this helps. please report back
    Thomas

  • I've attempted to set up iCloud on my iPhone - everytime I try to log on to the mail I get the message"incorrect username or password" I have tried deleting iCloud 3 times switching the ooh one on and off and still (and never have) been able to email help

    Please help!

    I was getting the same error too for a long time. It seems to have stopped....and my calendar and contacts synching has also stopped. Not sure what's going on.

  • How to migrate DB from single node 10gR2 to RAC 11gR2 on diff platform?

    How to migrate DB from single node 10gR2 to RAC 11gR2 on different platform with possible minimum downtime? We have a situation of upgrade/migrate oracle 10gR2 single instance DB to 2-node RAC 11gR. The source OS is Solaris 10 on SPARC and the target OS is Linux (the target servers could be changed to Solaris 11 x86 if needed). What is the best solution on that?
    Thanks,

    Technically, can do the following for upgrading and migration?
    1. Create 11gR2 oracle home on the same server and upgrade the database from 10gR2 to 11gR2 by running conversion (2 hour down time?)
    2. Set up Heterogeneous Primary and Physical Standbys by RMAN. The standby is the RAC with ASM. No need down time. (from Solaris Space to Linux - this may be a problem)
    3. At the cutoff time, activate the standby DB from the RAC ASM.
    If feasible, do we have some detail guild line for each step?

  • I had a free trial of Lightroom 5 and tried to purchase it, but page errors blocked me from being able to do this. I then got a free trial of Adobe CC.  At first I saw my old photos, then the whole thing went down, and now I have a new Lightroom 5 trial,

    I had a free trial of Lightroom 5 and tried to purchase it, but page errors blocked me from being able to do this. I then got a free trial of Adobe CC.  At first I saw my old photos, then the whole thing went down, and now I have a new Lightroom 5 trial, and none of my edited photos.  Can I retrieve all these photos from my first trial?

    Yes, you have to find the catalog file from the CC trial (use your operating system to search for it, the file name ends with .LRCAT), and then once you find it, open the LR5 trial and open the catalog you just found using File->Open Catalog

  • I synced my phone to my computer, went down and upgraded my phone, and found that before I could sync my new phone my computer crashed. No, I didn't have it backed up properly. Is there anything I can do???

    I synced my old iphone4, went down and upgraded to the iPhone 5c. When I came back my itunes wouldn't open properly and is now no longer functioning. Is there any way that I can sync my new phone to my old itunes???

    Follow these instructions EXACTLY: Remove and reinstall iTunes and related software components for Windows Vista, Windows 7, or Windows 8 - Apple Support

  • I just bought a macbook pro and the mountain lion free update applied correctly but the internet went down and now the download won't appear/ continue, and when i open the app store it requests me to pay for the update? what can i do?

    i just bought a macbook pro and the mountain lion free update applied correctly but the internet went down and now the download won't appear/ continue, and when i open the app store it requests me to pay for the update? what can i do?

    Welcome to Apple Communities
    http://www.apple.com/support/mac/app-store/contact

  • TS3276 my server went down and now my email is not working, can anyone help please?

    My website server went down and now my email is not working (though the server is back up)
    Does anyone know what I need to do?
    I'm told by my website hosts that I need to 'reconfigure authentication" but I have no idea how to do this, they are now closed for the weekend and I need it fixed before Monday morning. Could anyone tell me where to find step by step instructions. I've tried but have made matters worse and the couple of mail accounts that WERE still receiving email are now also not.
    Help please!

    Hi Eyen80,
    If you are having issues with your iPhone’s camera not functioning, you may want to troubleshoot using the steps in this article -
    iPhone: Hardware troubleshooting
    http://support.apple.com/kb/TS2802
    Specifically -
    Camera isn't functioning or has undesired image quality
    If the screen shows a closed lens or black image, force quit the Camera app.
    If you do not see the Camera app on the Home screen, try searching for it in Spotlight. If the camera does not show up in the search, check to make sure that Restrictions are not turned on by tappingSettings > General > Restrictions.
    Ensure the camera lens is clean and free from any obstructions. Use a microfiber polishing cloth to clean the lens.
    Cases can interfere with the camera and the flash. Try gently cleaning the lens with a clean dry cloth or removing the case if you see image or color-quality issues with photos.
    Try turning iPhone off and then back on.
    Tap to focus the camera on the subject. The image may pulse or briefly go in and out of focus as it adjusts.
    Try to remain steady while focusing:
    Still images: Remain steady while taking the picture. If you move too far in any direction, the camera automatically refocuses to the center.
    Note: If you take a picture with iPhone turned sideways, it is automatically saved in landscape orientation.
    Video: Adjust focus before you begin recording. You can also tap to readjust focus while recording. Exiting the Camera application while recording will stop recording and will save the video to the Camera Roll.
    Note: Video-recording features are not available on original iPhone or iPhone 3G.
    If your iPhone has a front and rear camera, try switching between them to verify if the issue persists on both.
    Thanks for using Apple Support Communities.
    Best,
    Brett L

  • Downloaded songs frome itunes and internet went down and they will not play and it also caused other songs not to but most will what do i need to do wiyhout erasing everything

    downloaded songs from itunes and the internet went down and they will not play and also caused other songs already downloaded not to play what can i do

    You can try a couple of different things. First - restart the iPod and see if the songs will play.
    Restart the iPod by holding down on the sleep button until the red slider appears and then slide to shut off. To power up hold the sleep button until the Apple logo appears and let go of the button.
    If that doesn't help - delete the songs and then download them again. You can download purchased music again at no charge as long as you use the same Apple ID that you bought them with originally and as long as you live in one of these countries.
    Previously purchased music is only available through iTunes in the Cloud in  Argentina, Australia, Belgium, Bolivia, Brazil, Canada, Chile, Colombia, Costa Rica, Cyprus, Czech Republic, Dominican Republic, Ecuador, El Salvador, France, Germany, Guatemala, Honduras, Ireland, Luxembourg, Malta, Mexico, New Zealand, Nicaragua, Panama, Paraguay, Peru, Slovakia, Spain, Switzerland, the United Kingdom, United States, and Venezuela.
    If you are running iOS 5 on the iPod - You can delete songs by swiping across the song in list view to bring up a red delete button. Tap the button to delete the songs. Download the songs again in the purchased tab in iTunes.
    Downloading Past Purchases

  • TS1717 Slotomania went down and will not reload and all my tokens are lost.

    Slotomania went down and will not reload and all tokens lost.  What can I do?

    Have you tried to restart Slotomania or your device?
    Contacting Slotomania ma be your best bet.
    You can contact them here -> http://www.playtika.com/contact.html

  • My company runs Adobe Acrobat XI on a PC.  Our PC went down and we replaced it with a new one.  We need Acrobat XI on it and have bought it already.  How can I get it put onto our new computer?

    My company runs Adobe Acrobat XI on a PC.  Our PC went down and we replaced it with a new one.  We need Acrobat XI on it and have bought it already.  How can I get it put onto our new computer without having to buy it again?

    Hi chrisb75729216,
    I checked with your account but could not find any order history so you might be using another email address on forums.
    You will first need to deactivate Acrobat from old machine and then install Acrobat on the new machine by activating with the same serial number.
    Please download Acrobat XI pro from here: Other downloads
    If you have single user license, then remember that the software can be activated on maximum two computers, say your desktop at work and laptop at home.
    Regards,
    Anubha

  • My server went down and now that it is back up my mail will not import back in help what steps do I do to rectify this?

    My server went down and now that it is back up it will not load my aol and gmail accounts mail help.  What steps do I do to make this happen?

    Hi there MeganElizabeth02061999,
    You may want to check your past purchases and see if you can find the tracks there. Take a look at the article below for more information.
    Downloading past purchases from the iTunes Store, App Store, and iBooks Store
    http://support.apple.com/kb/ht2519
    -Griff W.

  • My 2002 iMac G4 went down during a power out. won't restart. tried PMU button. still won't start. fan tries for 2 sec, but nothing

    my 2002 iMac G4 went down during a power out. won't restart. tried PMU button. still won't start. fan tries for 2 sec, but nothing

    Sometimes, a replacement 1/2AA 3.6v Lithium PRAM battery is needed
    to get these back into running form again, because these are one-way
    and without a good battery, settings are lost or easily corrupted. And this
    can contribute to a failure on startup. But that is a troubleshooting measure.
    There could be another issue, although these batteries are only good for a
    few years at best. A power outage can revoke any remainder of life out of it.
    The take-apart is tedious and detail oriented; to include new thermal paste
    and some special tools, good instructions, and some skills also help. An
    ifixit.com repair guide may help; as would an original Apple Service guide.
    Not sure if other good online guides or youtube video are available.
    If the power supply is bad, the test ports near the PMU button are where
    to check; with multi-meter. Also power supplies may develop issues with
    bad capacitors. They have been said to be repairable.
    Here's an older site with a take-apart from some time ago.
    http://www.xlr8yourmac.com/systems/imac_g4/imacg4_takeapart.html
    This also would be a good time to consider, if battery is all that's failed, to
    upgrade the internal RAM chip, or upgrade the hard disk drive, etc.
    Good luck & happy computing!
    edited

  • Why does my ipod sometimes flashes white then shuts down and then the only way to restart is to restore?

    why does my ipod sometimes flashes white then shuts down and then the only way to restart is to restore?

    If you restored to factory settings/new iPod and still have the problem that indicates a hardware problem.
    Make an appointment at the Genius Bar of an Apple store..
    Apple Retail Store - Genius Bar

Maybe you are looking for