ASM instance getting down

While production database is operational, ASM instance gets down out of sudden.
The error reflecting in alert log is :
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Also in trace files the same error is reflecting. I am unable to find the root cause why the ASM instance getting down again n again .
Kindly help !
Thanks !!

I thought about more input. But well: Check for errors in the clusterware logs next. And check your system logs.
Any tracefiles created by the ASM instance?
A note: You´re on 10.2.0.2.0. Thats way too old. The current version is 10.2.0.4.0. I STRONGLY recommend to upgrade your clusterware and ASM to that version. If you need for some reason you can leave your database on 10.2.0.2.0.
Ronny Egner
My Blog: http://blog.ronnyegner-consulting.de

Similar Messages

Whether instance gets down due to insufficient memory ?

hi,
11g R2 (Linux)
Other day one of the 3 nodes got evicted due to insufficient memory
Err message is
ORA-04031: unable to allocate 4136 bytes of shared memory ("shared pool","unknown object","sga heap(3,0)","gcs dynamic r")"
"LMS0 (ospid: 8489): terminating the instance due to error 4031"
Thanks

Hi, Have you configured memory_target or memory_max_target? If so then it may supersede depending on values set. Please post all memory parameters set and based on that we can answer. Also let us know the OS since in Linux there is a concept of tempfs which also limits the maximum memory to be allocated. While in solaris there is concept of project, which limits max memory per oracle instance depending on OS user.
Regards
Tushar

Automatic instance getting down with ORA-600

ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [ktspfredo-setstate-1], [0], [1], [16], [], [], [], [], [], [], [], []
help me start the instance to stable state@

These errors indicate a detected corruption when trying to apply undo/redo information, probably during instance recovery. [6006] is for an index, [ktspfredo-setstate-1] for a LOB (related bug: #11790175). The trace file should include more information though in the cryptic form of a diagnostic block/undo/redo dump.
Back up the current state of the database (to be able to restart the whole recovery process). Then, if you have a valid older backup, try restoring from that backup. If you have no backup, it may be difficult to recover. If you have a support contract, Oracle Support will probably be able to help. It is not possible to effectively help with recovery of a corrupted database in a community forum.
Thanks,
Sergiusz

ASM instance crash due to error ORA-27506: IPC error connecting to a port

Hi All,
Today the ASM instance goes down.
When i checked the alert log I found the below error.
ORA-27506: IPC error connecting to a port
ORA-27300: OS system dependent operation:sendmsg failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: sskgxpsnd1
Please find the environment details.
OS : RHEL-5
DB: 11.1.0.7 2-node RAC
I want to know the root cause of this issue.
Please suggest.
Thanks and Regards,

Hi,
Could you please upload cluster alert log and cssd.log?
regards,
Kishore

ASM instance removal not happening: showing ora.asm still runing

I am trying to remove the ASM instance, though the ASM instance is down, still it shows resouce ora.asm is running.
Could you please help me to remove the ASM instance.
oracle@tbardbb2:/usr/home/rcummins> srvctl remove ASM -n tbardbb2
-n option has been deprecated and will be ignored.
PRCR-1025 : Resource ora.asm is still running
Now I stopped the resource: crsctl stop resource ora.asm -n tbardbb2 -f
now while deleting it shows:
oracle@tbardbb2:/usr/home/rcummins> srvctl remove asm -n tbardbb2
-n option has been deprecated and will be ignored.
PRCR-1001 : Resource ora.asm does not exist
How will I check if my ASM instance is removed..
Edited by: 784786 on Jul 29, 2011 5:26 PM

784786 wrote:
I am trying to remove the ASM instance, though the ASM instance is down, still it shows resouce ora.asm is running.
Could you please help me to remove the ASM instance.
oracle@tbardbb2:/usr/home/rcummins> srvctl remove ASM -n tbardbb2
-n option has been deprecated and will be ignored.
PRCR-1025 : Resource ora.asm is still running
Now I stopped the resource: crsctl stop resource ora.asm -n tbardbb2 -f
now while deleting it shows:
oracle@tbardbb2:/usr/home/rcummins> srvctl remove asm -n tbardbb2
-n option has been deprecated and will be ignored.
PRCR-1001 : Resource ora.asm does not exist
How will I check if my ASM instance is removed..
Edited by: 784786 on Jul 29, 2011 5:26 PM
HI,
Please read note How to remove an ASM instance and its corresponding database(s) on UNIX [ID 357261.1] for the details
CHeers

Getting error 'could not locate ASM instance' when make copy of ASM VM

Hi everyone,
I make virtual machine of OEL 5.7 with oracle 10.2.0.1 and successfully configured ASM instance. I am using vmware workstation.
Now when I make copy of this virtual machine and startup database, then I get the below error:
[oracle@oelinux ~]$ export ORACLE_SID=asmdb
[oracle@oelinux ~]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.1.0 - Production on Sat Sep 29 01:10:26 2012
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+ASMGROUP/asmdb/spfileasmdb.ora'
*ORA-17503: ksfdopn:2 Failed to open file +ASMGROUP/asmdb/spfileasmdb.ora*
ORA-15077: could not locate ASM instance serving a required diskgroup
So how can i solve this problem??
Note: I copy this VM using both copy/paste and clone option, but get this error both times.
Regards,
Khurram Bhatti

EdStevens wrote:
Khurram Shehzad wrote:
I add 3 SCSI vmware disks. and then use them as a raw device. no ASMLib is used.
I refer below tutorial
[http://www.oracle-base.com/articles/10g/asm-using-asmlib-and-raw-devices.php]
That tutorial picks up after you created the disks.
How did you create the virtual disks attatched to your vm? What are its properties - from VMWorksation's perspective.
How did you create the "copy" of the vm?
It depends on how you actually created the disks, at the vm level, not the partitioning and formatting at the vm's OS level. I've used Tim's tutorials myself on some other ASM related projects, and he typically has you create the ASM disks as 'sharable', which would mean they would not copy necessarily copy with the vm. I've not played around the with variants much, but I'm pretty sure what you are seeing is a result of either the way the disk was created in the original vm, or the way you made the copy of the vm, or both.
Take a look at the machine properties of the two vms in the VMWorkstation control panel, looking at the HDD definitions. Compare and contrast.Hi EdStevens,
My problem resolved with below steps. Thanks fro your replies.
export ORACLE_SID=+ASM
sqlplus sys as sysdba
Connected to an idle instance.
SQL> startup mount
SQL> alter system register;
SQL> exit
[oracle@oelinux ~]$ export ORACLE_SID=asmdb
[oracle@oelinux ~]$ sqlplus sys as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Regards,
Khurram

Shutting down ASM instance in 11.2 RAC

DB version : 11.2.0.2
Platform : Solaris 10 / AIX 6.1
2-node RAC
Since OCR and Voting Disks are stored in ASM Diskgroups (11.2) , is it OK to shutdown ASM instance in any or both of the nodes ? ie.
Will CRS in a node go down if you shutdown ASM instance in that Node ?

greeny wrote:
DB version : 11.2.0.2
Platform : Solaris 10 / AIX 6.1
2-node RAC
Since OCR and Voting Disks are stored in ASM Diskgroups (11.2) , is it OK to shutdown ASM instance in any or both of the nodes ? ie.
Will CRS in a node go down if you shutdown ASM instance in that Node ?Yes. You will have to first stop the Oracle Clusterware stack on the node on which you need to stop the Oracle ASM instance.
Either use "crsctl stop cluster -n node_name" or "crsctl stop crs" for this purpose.
Since you are using OCR and VOTING DISK on ASM, it's not possible stop only ASM and keep cluster running.
Regards,
Levi Pereira

Getting errors while starting ASM instance in 11gR2

Hi All,
While starting ASM instance, i'm getting the following errors.
[grid@xxxxx ~]$ . oraenv
ORACLE_SID = [+ASM2] ?
The Oracle base has been set to /u01/app/oracle
[grid@xxxxx ~]$ sqlplus "/ as sysasm"
SQL*Plus: Release 11.2.0.3.0 Production on Thu May 3 05:23:52 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected.
SQL> cl scr
SQL> shut immediate
ORA-00604: error occurred at recursive SQL level 1
ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","unknown object","KGLH0^37a71565","kglHeapInitialize:temp")
Please give me any suggestions?
--Thanks,

http://ora-04031.ora-code.com/
ORA-04031:
unable to allocate string bytes of shared memory ("string","string","string","string")
Cause:     More shared memory is needed than was allocated in the shared pool.
Action:     If the shared pool is out of memory, either use the dbms_shared_pool package to pin large packages, reduce your use of shared memory, or increase the amount of available shared memory by increasing the value of the INIT.ORA parameters "shared_pool_reserved_size" and "shared_pool_size". If the large pool is out of memory, increase the INIT.ORA parameter "large_pool_size".

ASM Instance down in Solaris due to missing ocr.loc file

We had to restart our solaris machine due to a hardware failure.
When i tried to start to ASM instance, i got the error.
SQL> startup mount;
ORA-29701: unable to connect to Cluster Manager
The css daemons were not started. I checked it using
$ ps -ef | grep css
After logging in as ROOT and setting Oracle's bin directory in its path, i tried to run
#./localconfig . But i didn't succeed.
Then I came to know that you can start css by executing init.cssd script in ($ORACLE_HOME/css/admin)
/u01/oracle/10g/css/admin
$ ./init.cssd start
./init.cssd: /var/opt/oracle/ocr.loc: cannot open
There is no such file called ocr.loc in this location. I confirmed it by
So I looked for /var/opt/oracle/ocr.loc
$ file /var/opt/oracle/ocr.loc
/var/opt/oracle/ocr.loc: cannot open: No such file or directory
What do i do?

Boutet,
Before you go ahead and restore the OCR, for that matter even to restore it successfully, you would need to restore/recreate the file /var/opt/oracle/ocr.loc. This file has the location of the OCR file, without this file, you will not be able to restore the OCR.
The contents of the ocr.loc file would be something like this (I am including this information so that you may like to use it to recreate the file, in case you lost the original one).
cat /etc/oracle/ocr.loc
ocrconfig_loc=/<path>/cdata/localhost/local.ocr
local_only=TRUE
Hope this help.
Thanks
Chandra Pabba

Proble shutting down ASM instance on Windows 2003 server.

Hi,
Before booting the windows 2003 server, I am trying to shutdown the ASM instance and it is just siiting there without any response. This is a 10g database and a 2003 server.
C:\Documents and Settings\Administrator>set oracle_sid=+ASM
C:\Documents and Settings\Administrator>sqlplus " / as sysdba"
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
SQL> shutdown immediate;
Thanks.

Thanks for your response. Yes I did stopped the database. I was having the same problem with the database ( as it was sitting there after I issued the shutdown immediate ). Finally I had to do shutdown abort, start up in restrict mode and do a shutdown again to stop the database gracefully

How does 1 ASM instance in a 10 node RAC cause all 10 ASM instance to hang

Linux RHEL4
11.2.0.1.0 GI
11.2.0.1.0 RDBMS
11.1.0.7.0 RDBMS
10.2.0.4.4 RDBMS
Yesterday we had one of the ASM instance on our 10-node cluster hang with a latch "ASM file allocation latch". Oracle support has stated we are hitting a bug with is fixed in the latest PSU.
What I am trying to understand is how a latch on one ASM instance caused all 10 ASM instances to hang?
Oracle Supports explanation still does not answer how all 10 ASM instance were affected other then the faulty code that the patch fixes. Bellow is info out of the SAR.
Data Guard Site:
================
ARC1 PID 21508 reports ORA-240 'control file enqueue held for more than 120 seconds' at Thu Oct 28 11:35:54 2010. This message means ARC1 PID 21508 was holding the enqueue past the maximum tine limit of 120 seconds.
Next ARC0 PID 17398 reports ORA-16146: 'control file enqueue unavailable' at Thu Oct 28 11:36:52 2010. This means ARC0 PID 17398 tried to access the controlfile and it could get the lock.
ARC1 PID 21508 is then killed at Thu Oct 28 11:49:57 2010.
Killing enqueue blocker (pid=21508) on resource CF-00000000-00000000 by (pid=23078) by killing session 5.3
We see that RSM has been restarted by Broker. This matches to the alert.log and messages showing RSM being restarted several time. i.e. RSM0 started with pid=84, OS id=27915
Since this started happening around 11:14, the problem may have really started then instead of w/ the CF enqueue block.
The primary appears to be shipping redo to the standby using asynchronous mode. Unless there was a gap, I do no know if ARC1 should have been doing any remote network I/O. Because a log switch to thread 1 sequence 5235 had just happened at 11:33am, I suspect it was a local disk I/O that was started at least. The ORA-240 is reported at 11:35, exactly 120 sec (or 2min) from this log switch. Notice also the entry to show the log thread 1 sequence 5234 was registered to the controlfile didn't happen yet.
ASM Side:
============
From the systemstate dump on ASM, multiple processes are waiting on 'ASM file allocation latch' and we are kind of stuck on stack:
ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1782<-sspuser()+112<-0000003218E0C5B0<-kfuhInsert()+175<-kffilCreate()+601<-kfnsUFG()+3691<-kfnsBackground()+4382<-kfnDispatch()+527<-opiodr()+1149<-ttcpip()+1251<-opitsk()+1633<-opiino()+958<-opiodr()+1149<-opidrv()+570<-sou2o()+103<-opimai_real()+133<-ssthrdmain()+214<-main()+201<-__libc_start_main()+219
This caused the hang on the ASM side and hence ASM was not responding to the database. The CF Enqueue issue on the database was a side effect of this.
This is a direct <BUG:9232266> which is a duplicate of <BUG:8974548>

Thanks Murali,
I was little reluctent to open a tar since my experince says that its better to troubleshoot the issue yourself instead of spending time with the Oracle support :)
Well, I got some more information on this : the sql for the process id of ASM instance is :
Alter Diskgroup mount all;
It looks like it is unable to mount the diskgroup and get hunged at that point , is it due to the new disk added to the system or do we missed something after adding the new disk.
Also, the ASM process can not be killed from unix box while the ASM is down.
Will appreciate your time if we can move in any directions from here ..
Thanks ,
Ankur

Possible to create new ASM instance and "import" existing diskgroup?

This is my worst nightmare. Two days before a server was about to be shut down, it crashed on me, corrupting the filesystem where Oracle was installed. Setup was an HP-UX 11.11 server running Oracle 10g with ASM storage on a disk array. Both instances were in the filesystem that got corrupted, and amazingly, we don't have a backup, so I'm not able to restore the instance configs and restart the database. Presumably, my data is still sitting there on the disks that belong to the ASM diskgroup, but I don't know how to get at it. I started by creating a new ASM instance, but I can't figure out if there's a way to create a diskgroup out of those 16 disks without wiping out what's on them.
V$ASM_DISK shows all the disks as MEMBERs, with the CREATE_DATE that they were originally created, so I know at least the headers are intact. So is there any way I can manually recreate that diskgroup in my new instance so it will mount the disks and let me at the data?
I realize that not having backups is the height of stupidity, but that boat has sailed. My company also chose not to pay for Oracle Support this year, so I'm kind of on my own. If anybody can at least point me in the right direction or tell me that it's a lost cause, I'd appreciate it.

Well, a little progress. I was able to finally mount the disks in my new ASM instance, and using asmcmd, I can actually see my datafiles sitting there. So now I'm thinking I should try manually creating a new database instance pointing to the spfile that's in there and try starting that. I'm also going to try using RMAN and the XMLDB FTP trick to copy those datafiles somewhere else so I know I can play with them.

Problem starting ASM instance after installing Oracle Grid standalone

Hi!
After installing Oracle Grid 11gr2 and rebooting my machine:
Linux localhost.localdomain 2.6.32-300.10.1.el5uek #1 SMP Wed Feb 22 17:37:40 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
I run asmca command and it failed to startup ASM instance, I tried to paste the log but it's too long!
Could someone help me please! thank you very much!
José
I paste just the last piece:
+[main] [ 2012-03-15 15:01:56.853 CET ] [SQLEngine.initialize:354] m_bReaderStarted: false+
+[main] [ 2012-03-15 15:01:56.857 CET ] [SQLEngine.initialize:358] Starting Reader Thread...+
+[main] [ 2012-03-15 15:01:56.919 CET ] [UsmcaLogger.logExit:122] Exiting oracle.sysman.assistants.usmca.backend.USMInstance Method : createSQLEngine+
+[main] [ 2012-03-15 15:01:56.919 CET ] [OracleHome.getVersion:877] OracleHome.getVersion called. Current Version: null+
+[main] [ 2012-03-15 15:01:56.927 CET ] [InventoryUtil.getOUIInvSession:347] setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS+
+[main] [ 2012-03-15 15:01:56.927 CET ] [OracleHome.getVersion:896] Homeinfo /u01/app/grid/product/11.2.0/grid,1+
+[main] [ 2012-03-15 15:01:56.928 CET ] [Version.isPre:274] version to be checked 11.2.0.1.0 major version to check against10+
+[main] [ 2012-03-15 15:01:56.928 CET ] [Version.isPre:285] isPre.java: Returning FALSE+
+[main] [ 2012-03-15 15:01:56.929 CET ] [OCR.loadLibrary:308]+
Inside constructor of OCR
+[main] [ 2012-03-15 15:01:56.929 CET ] [OCR.isCluster:939] Calling OCRNative for isCluster()+
+[main] [ 2012-03-15 15:01:56.930 CET ] [OCR.isCluster:943] OCR Result status = true+
+[main] [ 2012-03-15 15:01:56.930 CET ] [OCR.isCluster:955] Bolean result = false+
+[main] [ 2012-03-15 15:01:56.930 CET ] [Version.isPre:274] version to be checked 11.2.0.1.0 major version to check against10+
+[main] [ 2012-03-15 15:01:56.930 CET ] [Version.isPre:285] isPre.java: Returning FALSE+
+[main] [ 2012-03-15 15:01:56.931 CET ] [MessageBundle.getMessageFields:338] MessageBundle.getMessage() failed to load the string PRKC-PRKC-1036 : CRS_HOME name passed to the method was null Exception: java.util.MissingResourceException: Can't find resource for bundle oracle.ops.mgmt.resources.PrkcMsg, key PRKC-1036 : CRS_HOME name passed to the method was null+
+[main] [ 2012-03-15 15:01:56.931 CET ] [ClusterUtils.getCRSActiveVersion:1797] Error getting CRS active version: PRKC-PRKC-1036 : CRS_HOME name passed to the method was null : <null>+
+[main] [ 2012-03-15 15:01:56.931 CET ] [OracleHome.getVersion:903] OracleHome.CRSActiveVersion: null+
+[main] [ 2012-03-15 15:01:57.084 CET ] [OracleHome.getVersion:922] OracleHome.server.getVersion Version: 11.2.0.1.0+
+[main] [ 2012-03-15 15:01:57.084 CET ] [OracleHome.getVersion:943] Current Version From Inventory: 11.2.0.1.0+
+[main] [ 2012-03-15 15:01:57.084 CET ] [OracleHome.getVersion:948] using sqlplus: /u01/app/grid/product/11.2.0/grid/bin/sqlplus+
+[main] [ 2012-03-15 15:01:57.084 CET ] [UsmcaLogger.logInfo:141] Role SYSASM+
+[main] [ 2012-03-15 15:01:57.085 CET ] [UsmcaLogger.logInfo:141] OS Auth true+
+[main] [ 2012-03-15 15:01:57.182 CET ] [SQLEngine.done:2148] Done called+
+[main] [ 2012-03-15 15:01:57.183 CET ] [UsmcaLogger.logException:171] SEVERE:method oracle.sysman.assistants.usmca.backend.USMInstance:checkAndStartupInstance+
+[main] [ 2012-03-15 15:01:57.183 CET ] [UsmcaLogger.logException:172] ORA-01034: ORACLE not available+
+[main] [ 2012-03-15 15:01:57.183 CET ] [UsmcaLogger.logException:173] oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-01034: ORACLE not available+
oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1650)
oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeQuery(SQLEngine.java:823)
oracle.sysman.assistants.usmca.backend.USMInstance.checkAndStartupInstance(USMInstance.java:573)
oracle.sysman.assistants.usmca.backend.USMInstance.discoverASMInstance(USMInstance.java:340)
oracle.sysman.assistants.usmca.service.UsmcaService.discoverASMInstance(UsmcaService.java:146)
oracle.sysman.assistants.usmca.Usmca.execute(Usmca.java:230)
oracle.sysman.assistants.usmca.Usmca.main(Usmca.java:333)
+[main] [ 2012-03-15 15:01:57.184 CET ] [UsmcaLogger.logExit:122] Exiting oracle.sysman.assistants.usmca.backend.USMInstance Method : checkAndStartupInstance+
+[main] [ 2012-03-15 15:01:57.184 CET ] [UsmcaLogger.logInfo:141] ASM failed to start. Check /u01/app/grid/cfgtoollogs/asmca/asmca-1203153PM0138.log for details.+

Hi,
i've got similar problem but my asm is not just down. After installing grid home with infrastructure I've restarted os and there is no crs services and the only result from this command:
# ps -ef | grep grid
grid      2075     1 0 13:58 ?        00:00:00 /u01/app/grid/product/11.2.0/grid/bin/ohasd.bin reboot
root      2463 2439 0 14:06 pts/0    00:00:00 grep gridgive me opposite to this one before restart:
$ ps -ef | grep grid
grid      2076     1 0 13:01 ?        00:00:00 /u01/app/grid/product/11.2.0/grid/bin/ohasd.bin reboot
grid      3646     1 0 13:44 ?        00:00:04 /u01/app/grid/product/11.2.0/grid/bin/ohasd.bin reboot
grid      3806     1 0 13:45 ?        00:00:02 /u01/app/grid/product/11.2.0/grid/bin/oraagent.bin
grid      3820     1 0 13:45 ?        00:00:00 /u01/app/grid/product/11.2.0/grid/bin/evmd.bin
grid      3853 3820 0 13:45 ?        00:00:00 /u01/app/grid/product/11.2.0/grid/bin/evmlogger.bin -o /u01/app/grid/product/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/grid/product/11.2.0/grid/evm/log/evmlogger.log
root      3879 2307 0 13:46 pts/0    00:00:00 su - grid
grid      3881 3879 0 13:46 pts/0    00:00:00 -bash
grid      4172     1 0 13:48 ?        00:00:00 /u01/app/grid/product/11.2.0/grid/bin/cssdagent
grid      4197     1 0 13:48 ?        00:00:00 /u01/app/grid/product/11.2.0/grid/bin/ocssd.bin
grid      4550     1 0 13:49 ?        00:00:00 asm_pmon_+ASM
grid      4552     1 0 13:49 ?        00:00:00 asm_psp0_+ASM
grid      4554     1 0 13:49 ?        00:00:02 asm_vktm_+ASM
grid      4558     1 0 13:49 ?        00:00:00 asm_gen0_+ASM
grid      4560     1 0 13:49 ?        00:00:00 asm_diag_+ASM
grid      4562     1 0 13:49 ?        00:00:00 asm_dia0_+ASM
grid      4564     1 0 13:49 ?        00:00:00 asm_mman_+ASM
grid      4566     1 0 13:49 ?        00:00:00 asm_dbw0_+ASM
grid      4568     1 0 13:49 ?        00:00:00 asm_lgwr_+ASM
grid      4570     1 0 13:49 ?        00:00:00 asm_ckpt_+ASM
grid      4572     1 0 13:49 ?        00:00:00 asm_smon_+ASM
grid      4574     1 0 13:49 ?        00:00:00 asm_rbal_+ASM
grid      4576     1 0 13:49 ?        00:00:00 asm_gmon_+ASM
grid      4578     1 0 13:49 ?        00:00:00 asm_mmon_+ASM
grid      4580     1 0 13:49 ?        00:00:00 asm_mmnl_+ASM
grid      4582     1 0 13:49 ?        00:00:00 oracle+ASM (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
grid      6128     1 0 13:52 ?        00:00:00 /u01/app/grid/product/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid      6142 3881 0 13:53 pts/0    00:00:00 ps -ef
grid      6143 3881 0 13:53 pts/0    00:00:00 grep grid
$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): pc
$ srvctl status asm
ASM is running on pcI'm using 2x 1TB hdisk as 2 pv in one vg. I thought that lvolume permissions are incorrect but now I had:
# ls -l /dev | grep dm
crw-rw---- 1 root root      10, 61 May 16 13:57 cpu_dma_latency
brw-rw---- 1 root disk     253,   0 May 16 13:57 dm-0
brw-rw---- 1 root disk     253,   1 May 16 13:57 dm-1
brw-rw---- 1 grid asmadmin 253, 10 May 16 13:57 dm-10
brw-rw---- 1 grid asmadmin 253, 11 May 16 13:57 dm-11
brw-rw---- 1 grid asmadmin 253, 12 May 16 13:57 dm-12
brw-rw---- 1 grid asmadmin 253, 13 May 16 13:57 dm-13
brw-rw---- 1 grid asmadmin 253,   2 May 16 13:57 dm-2
brw-rw---- 1 grid asmadmin 253,   3 May 16 13:57 dm-3
brw-rw---- 1 grid asmadmin 253,   4 May 16 13:57 dm-4
brw-rw---- 1 grid asmadmin 253,   5 May 16 13:57 dm-5
brw-rw---- 1 grid asmadmin 253,   6 May 16 13:57 dm-6
brw-rw---- 1 grid asmadmin 253,   7 May 16 13:57 dm-7
brw-rw---- 1 grid asmadmin 253,   8 May 16 13:57 dm-8
brw-rw---- 1 grid asmadmin 253,   9 May 16 13:57 dm-9and:
# ls -l /dev/vg_main/
total 0
lrwxrwxrwx 1 root root 7 May 16 13:57 asmlvol1 -> ../dm-2
lrwxrwxrwx 1 root root 8 May 16 13:57 asmlvol10 -> ../dm-11
lrwxrwxrwx 1 root root 8 May 16 13:57 asmlvol11 -> ../dm-12
lrwxrwxrwx 1 root root 8 May 16 13:57 asmlvol12 -> ../dm-13
lrwxrwxrwx 1 root root 7 May 16 13:57 asmlvol2 -> ../dm-3
lrwxrwxrwx 1 root root 7 May 16 13:57 asmlvol3 -> ../dm-4
lrwxrwxrwx 1 root root 7 May 16 13:57 asmlvol4 -> ../dm-5
lrwxrwxrwx 1 root root 7 May 16 13:57 asmlvol5 -> ../dm-6
lrwxrwxrwx 1 root root 7 May 16 13:57 asmlvol6 -> ../dm-7
lrwxrwxrwx 1 root root 7 May 16 13:57 asmlvol7 -> ../dm-8
lrwxrwxrwx 1 root root 7 May 16 13:57 asmlvol8 -> ../dm-9
lrwxrwxrwx 1 root root 8 May 16 13:57 asmlvol9 -> ../dm-10
lrwxrwxrwx 1 root root 7 May 16 13:57 lv01_swap -> ../dm-1
lrwxrwxrwx 1 root root 7 May 16 13:57 lv02_root -> ../dm-0I've used search path for ams disks as /dev/dm* and /dev/vg_main/asm* and seen all of them in both configurations.
I've tried to reconfigure HAS through (as root):
$GRID_HOME/crs/install/roothas.pl -deconfig -force -verbose
$GRID_HOME/root.shthis gave me working asm instance but after restart I'm in base position with no crs services.
And... I've tried to startup crs after restart with no luck but before restart I can shutdown everything and startup ASM instance too (so everything is fine till the reboot). I'm on CentOS 6:
$ uname -a
Linux pc 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST 2012 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/issue
CentOS release 6.2 (Final)
Kernel \r on an \mAny hints about how to resolve this problem would be most welcome.
Regards.

Listener getting down in every 5-6 hrs

Hi Frends,
Our database listener is getting down in 5-6 hrs from last 3 days. On this database, we run batch files which continueosly upload data using SQL loader. Our environment detail is as follos:
Database : Version 10.2.0.2.0
OS : Linux
In listener log it throws below errors :
TNS-12518: TNS:listener could not hand off client connection
Regards,
Umesh

Hi Pavan, please find the details as below:
1.Try to check whether are you using dedicated or shared server. ?
--> Its a dedicated server
Try to check the listener services - you can check number of refused connections.-- >
oracle@btc:/btc> lsnrctl services
LSNRCTL for Linux: Version 10.2.0.2.0 - Production on 12-APR-2011 13:33:31
Copyright (c) 1991, 2005, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM", status BLOCKED, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         LOCAL SERVER
Service "+ASM_XPT" has 1 instance(s).
Instance "+ASM", status BLOCKED, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         LOCAL SERVER
Service "btc.ind.com" has 1 instance(s).
Instance "btc", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:64665 refused:13 state:ready
         LOCAL SERVER
Service "btcXDB.ind.com" has 1 instance(s).
Instance "btc", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER <machine: btc, pid: 7683>
         (ADDRESS=(PROTOCOL=tcp)(HOST=btc.ind.com)(PORT=19662))
Service "btc_XPT.ind.com" has 1 instance(s).
Instance "btc", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:64665 refused:13 state:ready
         LOCAL SERVER
The command completed successfully
If you are working with dedicated server then try to increase the process paramter value since it might not be sufficient/could not able to handle the connections and monitor listener.log for issues.--> Process parameter has been set to 2000 and running processes and session is as follows:
SQL> select count(*) from v$session;
COUNT(*)
37
SQL> select count(*) from v$process;
COUNT(*)
40
Regards,
Umesh

Change Node association of ASM instance

Hi,
I have a 4-node cluster running RHEL 5 and GI - 11.2.0.2. After a successful installation we discovered that the asm instance association got switched between Node 2 & Node 3.
e.g., - Node 2 is running ASM3 instance
Node 3 is running ASM2 instance.
Is it possible to correct this, or do I need to rebuild the whole cluster again!?
Thanks,
AbbY

Hi AbbY,
this is the result, that the root.sh on node 3 was faster than the one on node 2 during install.
If you would like to have the ASM instance in a specific order after installation, you have to run them sequentially (and nor in parallel).
But one thing you should also be aware of, is that the ASM number in 11.2 is not fix anymore:
=> The ASM instance number is dependant on the node number (olsnodes -n -i -t -s).
=> Each nodes gets a lease for the node number for approx. 1 week. It's like a DHCP lease, which can expire if not started.
So if you would shut down node 2 and 3 for one week and then restart the nodes in a different order, lease would change (and therefor the ASM instance number).
Which leads to the further question (because this is really flexible): Why do you want the +ASM instances to have fixed names?
If your nodes would be "Mars" and "Pluto", would it matter where ASM1 and ASM2 runs on?
If you say: We need to know for sqlplus to connect to ASM instance. Why not simply connect this way to ASM:
sqlplus sys/pwd@hostname:port/+ASM
You will always end up on the ASM instance on the node (no matter what instance number the ASM instance has on that node).
O.k. I have to admit there are still a few tools who need the instance number (like asmcmd), but I know this necessity will change, since cluster gets more flexible in the future.
Sebastian

ASM instance getting down

Similar Messages

Maybe you are looking for