Can't mount diskgroup on second RAC node (ss3)

I have a newly created 11g (Oracle 11.1.0.6) RAC database consisting of two nodes (SS1 and SS3).
Installed CRS and everything is working normal: crs_stat -t -v shows:
Name Type R/RA F/FT Target State Host
ora....Y1.inst application 0/5 0/0 ONLINE ONLINE ss1
ora....Y2.inst application 0/5 0/0 ONLINE OFFLINE
ora.SY.db application 0/0 0/1 ONLINE ONLINE ss3
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE ss1
ora....S1.lsnr application 0/5 0/0 ONLINE ONLINE ss1
ora.ss1.gsd application 0/5 0/0 ONLINE ONLINE ss1
ora.ss1.ons application 0/3 0/0 ONLINE ONLINE ss1
ora.ss1.vip application 0/0 0/0 ONLINE ONLINE ss1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE ss3
ora....S3.lsnr application 0/5 0/0 ONLINE ONLINE ss3
ora.ss3.gsd application 0/5 0/0 ONLINE ONLINE ss3
ora.ss3.ons application 0/3 0/0 ONLINE ONLINE ss3
ora.ss3.vip application 0/0 0/0 ONLINE ONLINE ss3
Installed ASM in it's own ORACLE_HOME. From ss1 I see:
SELECT group_number, disk_number, mount_status, header_status, state, path FROM v$asm_disk;
GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU STATE PATH
1 0 CACHED MEMBER NORMAL /dev/rdsk/c4t600C0FF0000000000B5DB04B603F9500d0s6
2 0 CACHED MEMBER NORMAL /dev/rdsk/c4t600C0FF0000000000B5DB04B603F9500d0s7
SELECT name, label FROM v$asm_disk;
NAME LABEL
DATA_0000
RECOVERY_0000
But, from ss3 I see:
SELECT group_number, disk_number, mount_status, header_status, state, path FROM v$asm_disk;
no rows selected
SELECT name, label FROM v$asm_disk;
NAME LABEL
DATA_0000
RECOVERY_0000
ALTER DISKGROUP DATA MOUNT;
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
I am using multipathed I/O ~ which I believe is significant from the info on multiplexed I/O in the Storage Administrator's Guide here:
[http://download.oracle.com/docs/cd/B28359_01/server.111/b31107/asmprepare.htm#BHCJBJFD]
/u01/app/product/11.1.0/asm/dbs/init.ora on both nodes has:
asm_diskstring='/dev/rdsk/*'
Any troubleshooting help is much appreciated.

Just for information:
make sure the owner of the database can read/write in the ASM devices.
i.o.w. If you have installed ASM under a different user than your database (for example - asm is installed with oracle and your database with oradb), you must make sure user oradb can read/write in the devices as well. Otherwise, you won't be able to startup your database either. Solve this by putting both users in the dba group and give the dba group read/write permission on the devices.
Arnoud

Similar Messages

Can we have Multiple Instance on same Node in Oracle 10g RAC

Hi All,
I am planning to implement the RAC in Oracle 10g.Before that i have one doubt regarding RAC.
My question is "Can we create multiple Instance on Same node(Server) ?"
is it possible.
Any ideas or thoughts would be apperciable.
Thanks in Advance.
Anwar

This is where it is important to keep the separation between 'database' and 'instance'.
A database is the set of files that contains the data (and the redo, control files, etc). A database does nothing by itself, other than take up lots of disk space.
An instance is theCPU cycles (running software) and the memory to control the database.
In Oracle RAC, you can have as many instances controlling one database [at the same time] as you want (within reason). Each instance must be able to access the disk(s) that contains the database.
These multiple instances can be on the same computer (effectively taking up a lot of server memory and CPU for nothing) or they can be on separate computers.
If they are on separate computers, the disk subsystems must be able to be shared across computers - this is occasionally done using operating system clusterware and is the main reason why clusterware is required at all. (This is also the toughest part of the pre-requisites in setting up a RAC and is very vendor dependent unless you use ASM.)
These instances need a communication connection to coordinate their work (usually a separate network card for each computer) so they do not corrupt the disk when they are trying to access the same file, and possibly the same block, at the same time.
In a RAC configuration, instances can be added, started, running, stopped and removed independent of each other (allowing a lot of high availability) or can be started and stopped as a group.
Each instance gets it's own SID, which is in no way any different than a non-RAC SID. It's just the name of a service that can be invoked. The neat thing is that the SID
a) helps the DBA keep things straight by lettiung us talk about 'instance A' (the Oracle software to be running over on computer A) vs 'instance B' when starting, stopping and managing;
b) helps the application by providing targets that can be listed in the TNSNAMES.ORA [against one service alias], which is used by ORacle Networking to provide automated load balance or failover (instance/SID a is not available, I guess I'll try the next in the list)
Hope that helps the concept level a bit.

Can't mount second internal drive

After two weeks off, my Mac can't spin up the second drive, which I installed two years ago and which has been no problem until now. I've done a clean system install and tried all kinds of boots with extensions off, etc. Drive Setup, TechTool Pro, and Disk First Aid do not see the drive. Retrospect sees it: "Bus 1, SCSI ID 2, IBM DDYS-T36958M, version S96H--no driver." SCSIProbe sees same thing, doesn't mount it: "Trying Device 1.2.0--FAILED: SCSI Error: -7932 (2): Sense 4(0). Trying Device 1.2.1--FAILED: Logical Units not supported." Norton Disk Doctor shows location after "Add Custom Disks" command, no volume name (the drive has--or had--three partitions.
Disk First Aid (v. 8.6.1) and Disk Doctor (1.0.5) both say everything is copacetic on the startup disk. TechTool (3.0.4) says all's fine except the system is missing a file called Enabler.
The drive came with an adapter which went between the SCSI cable and the drive. Directions say don't use jumpers with this adapter, which has worked fine up until now. The vendor appears to be out of business--phones disconnected, etc.
Any suggestions appreciated. Thanks, Fletcher

When drives won't come ready, they may be dead, or there may be termination problems. No termination often causes a Mac to hang during startup. Too much termination can cause drive not ready, or data errors, or occasionally work fine.
The usual method of providing termination with a 68-pin controller and 68-pin cable is with a large discrete terminator at the end of the cable. If your SCSI card is original, that is how I would expect it to be set up. In mine, the terminator was stuck to the top of the last drive with double-stick tape.
Recent 80-pin drives often do not have any way of providing termination. They take their address from the jumpers on the 68-to-80 adapter. Some older (generally SE-only) drives can still provide SE termination, and having a drive set for termination while using the end-of-cable termination can sometimes cause problems.

Rac node restart

Hello everyone,
I have met an error,that is our RAC node auto restart with below messages.
#/u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/alert_odsdb1.log
Fri Jun 07 12:23:42 2013
Thread 1 cannot allocate new log, sequence 58363
Checkpoint not complete
Current log# 2 seq# 58362 mem# 0: +DATA/odsdb/onlinelog/group_2.265.812288839
Current log# 2 seq# 58362 mem# 1: +DATA/odsdb/onlinelog/group_2.266.812288839
Fri Jun 07 12:23:42 2013
NOTE: ASMB terminating
Errors in file /u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/odsdb1_asmb_32641.trc:
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID:
?? ID: 2047 ???: 5
Errors in file /u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/odsdb1_asmb_32641.trc:
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID:
?? ID: 2047 ???: 5
ASMB (ospid: 32641): terminating the instance due to error 15064
Fri Jun 07 12:23:44 2013
ORA-1092 : opitsk aborting process
Fri Jun 07 12:23:46 2013
ORA-1092 : opitsk aborting process
Instance terminated by ASMB, pid = 32641
Fri Jun 07 12:25:02 2013
Starting ORACLE instance (normal)
Fri Jun 07 12:25:23 2013
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.37.103, mac=00-26-55-eb-61-89, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=135.33.2.8, mac=00-26-55-eb-61-88, net=135.33.2.0/27, mask=255.255.255.224, use=public/1]
Public Interface 'eth0:1' configured from GPnP for use as a public interface.
[name='eth0:1', type=1, ip=135.33.2.13, mac=00-26-55-eb-61-88, net=135.33.2.0/27, mask=255.255.255.224, use=public/1]
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.2.0/dbhome_2/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_2
System name:     Linux
Node name:     odsdb1
Release:     2.6.18-308.el5
Version:     #1 SMP Fri Jan 27 17:17:51 EST 2012
Machine:     x86_64
Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0/dbhome_2/dbs/initodsdb1.ora
System parameters with non-default values:
processes = 4500
sessions = 6784
event = ""
spfile = "+DATA/odsdb/spfileodsdb.ora"
nls_language = "SIMPLIFIED CHINESE"
nls_territory = "CHINA"
memory_target = 170G
control_files = "+DATA/odsdb/controlfile/current.262.812288837"
control_files = "+DATA/odsdb/controlfile/current.261.812288837"
db_block_size = 8192
compatible = "11.2.0.0.0"
db_files = 4096
cluster_database = TRUE
db_create_file_dest = "+DATA"
db_recovery_file_dest = ""
db_recovery_file_dest_size= 38820M
thread = 1
undo_tablespace = "UNDOTBS1"
instance_number = 1
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=odsdbXDB)"
remote_listener = "odsdb-cluster-scan:1521"
job_queue_processes = 1000
audit_file_dest = "/u01/app/oracle/admin/odsdb/adump"
audit_trail = "DB"
db_name = "odsdb"
open_cursors = 300
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
169.254.37.103
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Fri Jun 07 12:25:33 2013
PMON started with pid=2, OS id=22959
Fri Jun 07 12:25:33 2013
PSP0 started with pid=3, OS id=22962
Fri Jun 07 12:25:34 2013
VKTM started with pid=4, OS id=22971 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Fri Jun 07 12:25:34 2013
GEN0 started with pid=5, OS id=22977
Fri Jun 07 12:25:34 2013
DIAG started with pid=6, OS id=22979
Fri Jun 07 12:25:35 2013
DBRM started with pid=7, OS id=22981
Fri Jun 07 12:25:35 2013
PING started with pid=8, OS id=22983
Fri Jun 07 12:25:35 2013
ACMS started with pid=9, OS id=22985
Fri Jun 07 12:25:35 2013
DIA0 started with pid=10, OS id=22987
Fri Jun 07 12:25:35 2013
LMON started with pid=11, OS id=22989
Fri Jun 07 12:25:35 2013
LMD0 started with pid=12, OS id=22991
* Load Monitor used for high load check
* New Low - High Load Threshold Range = [61440 - 81920]
Fri Jun 07 12:25:35 2013
LMS0 started with pid=13, OS id=22994 at elevated priority
Fri Jun 07 12:25:35 2013
LMS1 started with pid=14, OS id=22998 at elevated priority
Fri Jun 07 12:25:35 2013
LMS2 started with pid=15, OS id=23002 at elevated priority
Fri Jun 07 12:25:35 2013
LMS3 started with pid=16, OS id=23006 at elevated priority
Fri Jun 07 12:25:35 2013
RMS0 started with pid=17, OS id=23010
Fri Jun 07 12:25:35 2013
LMHB started with pid=18, OS id=23013
Fri Jun 07 12:25:35 2013
MMAN started with pid=19, OS id=23015
Fri Jun 07 12:25:35 2013
DBW0 started with pid=20, OS id=23017
Fri Jun 07 12:25:35 2013
DBW1 started with pid=21, OS id=23019
Fri Jun 07 12:25:35 2013
DBW2 started with pid=22, OS id=23022
Fri Jun 07 12:25:35 2013
DBW3 started with pid=23, OS id=23024
Fri Jun 07 12:25:35 2013
DBW4 started with pid=24, OS id=23026
Fri Jun 07 12:25:35 2013
DBW5 started with pid=25, OS id=23028
Fri Jun 07 12:25:35 2013
DBW6 started with pid=26, OS id=23031
Fri Jun 07 12:25:35 2013
DBW7 started with pid=27, OS id=23033
Fri Jun 07 12:25:35 2013
LGWR started with pid=28, OS id=23035
Fri Jun 07 12:25:35 2013
CKPT started with pid=29, OS id=23037
Fri Jun 07 12:25:35 2013
SMON started with pid=30, OS id=23039
Fri Jun 07 12:25:35 2013
RECO started with pid=31, OS id=23041
Fri Jun 07 12:25:35 2013
RBAL started with pid=32, OS id=23043
Fri Jun 07 12:25:35 2013
ASMB started with pid=33, OS id=23045
Fri Jun 07 12:25:35 2013
MMON started with pid=34, OS id=23048
Fri Jun 07 12:25:35 2013
MMNL started with pid=35, OS id=23052
Fri Jun 07 12:25:35 2013
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
NOTE: initiating MARK startup
starting up 1 shared server(s) ...
Starting background process MARK
Fri Jun 07 12:25:35 2013
MARK started with pid=37, OS id=23056
NOTE: MARK has subscribed
lmon registered with NM - instance number 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 119)
List of instances:
1 2 (myinst: 1)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid according to instance 2
* domain 0 valid = 1 according to instance 2
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration started (old inc 119, new inc 121)
List of instances:
1 2 (myinst: 1)
Nested reconfiguration detected.
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Fri Jun 07 12:25:45 2013
Submitted all GCS remote-cache requests
Fri Jun 07 12:26:08 2013
Fix write in gcs resources
Reconfiguration complete
Fri Jun 07 12:26:10 2013
LCK0 started with pid=40, OS id=23632
Fri Jun 07 12:26:10 2013
Starting background process RSMN
Fri Jun 07 12:26:10 2013
RSMN started with pid=41, OS id=23646
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /u01/app/oracle
Fri Jun 07 12:26:11 2013
ALTER SYSTEM SET local_listener=' (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=135.33.2.13)(PORT=1521))))' SCOPE=MEMORY SID='odsdb1';
ALTER DATABASE MOUNT /* db agent *//* {1:9971:2} */
Fri Jun 07 12:26:11 2013
NOTE: Loaded library: System
Fri Jun 07 12:26:11 2013
SUCCESS: diskgroup DATA was mounted
Fri Jun 07 12:26:11 2013
NOTE: dependency between database odsdb and diskgroup resource ora.DATA.dg is established
Fri Jun 07 12:26:16 2013
Successful mount of redo thread 1, with mount id 3452000551
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Lost write protection disabled
Completed: ALTER DATABASE MOUNT /* db agent *//* {1:9971:2} */
ALTER DATABASE OPEN /* db agent *//* {1:9971:2} */
Picked broadcast on commit scheme to generate SCNs
Thread 1 advanced to log sequence 58364 (thread open)
Thread 1 opened at log sequence 58364
Current log# 2 seq# 58364 mem# 0: +DATA/odsdb/onlinelog/group_2.265.812288839
Current log# 2 seq# 58364 mem# 1: +DATA/odsdb/onlinelog/group_2.266.812288839
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jun 07 12:26:21 2013
SMON: enabling cache recovery
Fri Jun 07 12:26:23 2013
minact-scn: Inst 1 is a slave inc#:121 mmon proc-id:23048 status:0x2
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000
Fri Jun 07 12:26:34 2013
[23651] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:2061372614 end:2061384964 diff:12350 (123 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Fri Jun 07 12:26:34 2013
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
Starting background process GTX0
Fri Jun 07 12:26:35 2013
GTX0 started with pid=45, OS id=23931
Starting background process RCBG
Fri Jun 07 12:26:35 2013
RCBG started with pid=46, OS id=23933
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Fri Jun 07 12:26:35 2013
QMNC started with pid=48, OS id=23940
Completed: ALTER DATABASE OPEN /* db agent *//* {1:9971:2} */
Fri Jun 07 12:26:38 2013
Starting background process CJQ0
Fri Jun 07 12:26:38 2013
CJQ0 started with pid=55, OS id=23977
Fri Jun 07 12:27:56 2013
Thread 1 advanced to log sequence 58365 (LGWR switch)
Current log# 1 seq# 58365 mem# 0: +DATA/odsdb/onlinelog/group_1.263.812288839
Current log# 1 seq# 58365 mem# 1: +DATA/odsdb/onlinelog/group_1.264.812288839
Fri Jun 07 12:28:18 2013
Starting background process SMCO
Fri Jun 07 12:28:18 2013
SMCO started with pid=70, OS id=25166
Fri Jun 07 12:29:01 2013
Thread 1 cannot allocate new log, sequence 58366
Trace file /u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/odsdb1_asmb_32641.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_2
System name: Linux
Node name: odsdb1
Release: 2.6.18-308.el5
Version: #1 SMP Fri Jan 27 17:17:51 EST 2012
Machine: x86_64
Instance name: odsdb1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 33
Unix process pid: 32641, image: oracle@odsdb1 (ASMB)
*** 2013-05-14 15:37:08.705
*** SESSION ID:(3499.1) 2013-05-14 15:37:08.705
*** CLIENT ID:() 2013-05-14 15:37:08.705
*** SERVICE NAME:() 2013-05-14 15:37:08.705
*** MODULE NAME:() 2013-05-14 15:37:08.705
*** ACTION NAME:() 2013-05-14 15:37:08.705
NOTE: initiating MARK startup
*** 2013-05-14 15:37:16.835
instance health monitoring reports instance shutting down
*** 2013-06-07 12:23:42.700
NOTE: ASMB terminating
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID:
?? ID: 2047 ???: 5
error 15064 detected in background process
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID:
?? ID: 2047 ???: 5
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+461<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdicrshnfy()+53<-ksuitm()+1332<-ksbrdp()+3344<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201<-__libc_start_main()+244<-_start()+36
----- End of Abridged Call Stack Trace -----
*** 2013-06-07 12:23:42.783
ASMB (ospid: 32641): terminating the instance due to error 15064
/u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
NOTE: ASMB process exiting, either shutdown is in progress
NOTE: or foreground connected to ASMB was killed.
Fri Jun 07 12:23:42 2013
NOTE: client exited [14808]
Fri Jun 07 12:23:44 2013
Received an instance abort message from instance 2
Please check instance 2 alert and LMON trace files for detail.
Fri Jun 07 12:23:44 2013
Received an instance abort message from instance 2
Please check instance 2 alert and LMON trace files for detail.
LMD0 (ospid: 31201): terminating the instance due to error 481
Instance terminated by LMD0, pid = 31201
Fri Jun 07 12:24:30 2013
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.37.103, mac=00-26-55-eb-61-89, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=135.33.2.8, mac=00-26-55-eb-61-88, net=135.33.2.0/27, mask=255.255.255.224, use=public/1]
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/11.2.0.2/grid/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
[grid@odsdb1 cssd]$ file core.30481
core.30481: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'ocssd.bin'
[grid@odsdb1 cssd]$ gdb
gdb gdbserver gdbtui
[grid@odsdb1 cssd]$ gdb ocssd.bin core.30481
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-42.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /u01/app/11.2.0.2/grid/bin/ocssd.bin...(no debugging symbols found)...done.
[New Thread 30486]
[New Thread 30530]
[New Thread 30526]
[New Thread 30525]
[New Thread 30523]
[New Thread 30522]
[New Thread 30521]
[New Thread 30520]
[New Thread 30519]
[New Thread 30504]
[New Thread 30503]
[New Thread 30495]
[New Thread 30485]
[New Thread 30484]
[New Thread 30483]
[New Thread 30481]
Reading symbols from /u01/app/11.2.0.2/grid/lib/libhasgen11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libhasgen11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libocr11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libocr11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libocrb11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libocrb11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libocrutl11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libocrutl11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libclntsh.so.11.1...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libclntsh.so.11.1
Reading symbols from /u01/app/11.2.0.2/grid/lib/libskgxn2.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libskgxn2.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /u01/app/11.2.0.2/grid/lib/libasmclntsh11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libasmclntsh11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libcell11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libcell11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libskgxp11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libskgxp11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libnnz11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libnnz11.so
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libaio.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libaio.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /u01/app/11.2.0.2/grid/lib/libnque11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libnque11.so
Reading symbols from /opt/oracle/extapi/64/asm/orcl/1/libasm.so...(no debugging symbols found)...done.
Loaded symbols for /opt/oracle/extapi/64/asm/orcl/1/libasm.so
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff505fd000
Core was generated by `/u01/app/11.2.0.2/grid/bin/ocssd.bin '.
Program terminated with signal 6, Aborted.
#0 0x000000369ea30265 in raise () from /lib64/libc.so.6
(gdb) where
#0 0x000000369ea30265 in raise () from /lib64/libc.so.6
#1 0x000000369ea31d10 in abort () from /lib64/libc.so.6
#2 0x00002afc67f9aeda in scls_abort (flags=0) at scls.c:7088
#3 0x000000000040babd in clssscExit (thrd=0x10d325a0, status=clssscreasonSHUTNORM) at clsssc.c:2155
#4 0x0000000000446221 in clssgmClientShutdown (thrd=0x10d325a0, cmInfo=0x10b40090) at clssgmc.c:6415
#5 0x0000000000436707 in clssgmProcClientReqs (thrd=0x10d325a0, clctx=0x10b40630) at clssgmc.c:704
#6 0x0000000000436405 in clssgmclientlsnr (thrd=0x10d325a0) at clssgmc.c:644
#7 0x000000000040ac2f in clssscthrdmain (thrd=0x10d325a0) at clsssc.c:1716
#8 0x000000369fa0677d in start_thread () from /lib64/libpthread.so.0
#9 0x000000369ead49ad in clone () from /lib64/libc.so.6
(gdb)
2013-06-07 12:19:37.377: [    CSSD][1085888832]clssscSelect: cookie accept request 0x10b40630
2013-06-07 12:19:37.377: [    CSSD][1085888832]clssgmAllocProc: (0x2aaab0133ea0) allocated
2013-06-07 12:19:37.379: [    CSSD][1085888832]clssgmClientConnectMsg: properties of cmProc 0x2aaab0133ea0 - 1,2,3,4,5
2013-06-07 12:19:37.379: [    CSSD][1085888832]clssgmClientConnectMsg: Connect from con(0x6ae44fa) proc(0x2aaab0133ea0) pid(14139/14139) version 11:2:1:4, properties: 1,2,3,4,5
2013-06-07 12:19:37.379: [    CSSD][1085888832]clssgmClientConnectMsg: msg flags 0x0000
2013-06-07 12:19:37.384: [    CSSD][1085888832]clssscSelect: cookie accept request 0x2aaab0133ea0
2013-06-07 12:19:37.384: [    CSSD][1085888832]clssscevtypSHRCON: getting client with cmproc 0x2aaab0133ea0
2013-06-07 12:19:37.384: [    CSSD][1085888832]clssgmRegisterClient: proc(69/0x2aaab0133ea0), client(1/0x2aaab010c5c0)
2013-06-07 12:19:37.385: [    CSSD][1085888832]clssgmRegisterShared: grp DBODSDB, mbr 0, type 1
2013-06-07 12:19:37.385: [    CSSD][1085888832]clssgmQueueShare: (0x2aaab0085790) target global grock DBODSDB member 0 type 1 queued from client (0x2aaab010c5c0), global grock DBODSDB, refcount 23
2013-06-07 12:19:37.385: [    CSSD][1085888832]clssgmRegisterShared: global grock DBODSDB member 0 share type 1, refcount 23
2013-06-07 12:19:37.391: [    CSSD][1085888832]clssscSelect: cookie accept request 0x2aaab0133ea0
2013-06-07 12:19:37.391: [    CSSD][1085888832]clssscevtypSHRCON: getting client with cmproc 0x2aaab0133ea0
2013-06-07 12:19:37.391: [    CSSD][1085888832]clssgmRegisterClient: proc(69/0x2aaab0133ea0), client(2/0x2aaab0061f10)
what is the problem
Edited by: 徐振富 on 2013-6-7 下午6:38
Edited by: 徐振富 on 2013-6-7 下午6:45

is your ASM instance up?
If not, trying bring up ASM instance up just by itself and see if it throws any error?
Post status of crsctl status cluster -all

Can I mount another instance of standby database

1. If I have a RAC physical standby database and instance 1 is mounted and is applying redo, can i mount second instance if I wish for example to perform a backup using rman?
2. If instance 1 is mounted and applying redo can I stop it and start instance 2 and continue to apply redo??
Thank you.
Regards.
Cristina.

1. If I have a RAC physical standby database and instance 1 is mounted and is applying redo, can i mount second instance if I wish for example to perform a backup using rman?Yes as Emre mentioned, you can mount all of the instances in cluster.
2. If instance 1 is mounted and applying redo can I stop it and start instance 2 and continue to apply redo??Redo apply will active only on one node, remaining standby instances will be idle
So you can start recovery any instance where you wish.

Mounted.ocfs2 -f shows current node only

I am trying to configure two-node RAC (10g R2) on 32-bit Linux on VMWare (Vincent Chan's guide).
when I run this command, it only shows the current (where the command is issued) node:
on rac1
[root@rac1 init.d]# mounted.ocfs2 -f
Device                FS     Nodes
/dev/sdb1             ocfs2 rac1It shows "rac1" only. "rac2" is missing.
on rac2
[root@rac2 init.d]# mounted.ocfs2 -f
Device                FS     Nodes
/dev/sdb1             ocfs2 rac2It shows "rac2" only. "rac1" is missing.
Here is some info:
*1st Node - rac1*
[root@rac1 init.d]# mounted.ocfs2 -f
Device                FS     Nodes
/dev/sdb1             ocfs2 rac1
[root@rac1 init.d]# mounted.ocfs2 -d
Device                FS     UUID                                  Label
/dev/sdb1             ocfs2 18f13142-7c34-43a1-92d3-668f666dbacb oracle
[root@rac1 init.d]# ./ocfs2 status
Configured OCFS2 mountpoints: /ocfs
Active OCFS2 mountpoints: /ocfs
[root@rac1 init.d]# ./o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold: 61
Network idle timeout: 10000
Network keepalive delay: 5000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active*2nd Node - rac2*
[root@rac2 init.d]# mounted.ocfs2 -f
Device                FS     Nodes
/dev/sdb1             ocfs2 rac2
[root@rac2 init.d]# mounted.ocfs2 -d
Device                FS     UUID                                  Label
/dev/sdb1             ocfs2 b65abba9-5988-4ee0-9e10-ad5533dff807 oracle
[root@rac2 init.d]# ./ocfs2 status
Configured OCFS2 mountpoints: /ocfs
Active OCFS2 mountpoints: /ocfs
[root@rac2 init.d]# ./o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold: 61
Network idle timeout: 10000
Network keepalive delay: 5000
Network reconnect delay: 2000
Checking O2CB heartbeat: ActiveAlso, if I create a file on "/ocfs" from rac1, it is not visible on rac2 and vice versa.
Could someone please let me know what should I check further? Or what the problem maybe?
TIA
Edited by: user130038 on Jul 21, 2011 10:10 AM

I don't have access to the virtual machines at the moment so I'll send you the "fdisk -l" output later but here is the excerpt from Vincent Chan's Guide that I am following to configure ocfs2 oartition - my apologies that it is not formatted/readable because I have copied it from the PDF and pasted it:
6. Configure Oracle Cluster File System (OCFS2)
OCFS2 is a general-purpose cluster file system developed by Oracle and integrated with the
Enterprise Linux kernel. It enables all nodes to share files concurrently on the cluster file system and
thus eliminates the need to manage raw devices. Here you will house the OCR and Voting Disk in
the OCFS2 file system. Additional information on OCFS2 can be obtained from OCFS2 User’s
Guide.
You should already have the OCFS2 RPMs installed during the Enterprise Linux installation. Verify
that the RPMs have been installed on both nodes.
rac1-> rpm -qa | grep ocfs
ocfs2-tools-1.2.2-2
ocfs2console-1.2.2-2
ocfs2-2.6.9-42.0.0.0.1.ELsmp-1.2.3-2
Create the OCFS2 configuration file. As the root user on rac1, execute
# ocfs2console
OCFS2 Console: Select Cluster, 1. Configure Nodes.
2. “The cluster stack has been started”: Click on Close.
3. Node Configuration: Click on Add.
Add Node: Add the following nodes and then click on Apply.
Name: rac1
IP Address: 192.168.2.131
IP Port: 7777
Name: rac2
IP Address: 192.168.2.132
4.
Install Oracle RAC 10g on Oracle Enterprise Linux Using VMware Server http://www.oracle.com/technology/pub/articles/chan-ubl-vmware.html?...
20 of 42 11-1-2007 7:40
IP Port: 7777
Verify the generated configuration file.
# more /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 192.168.2.131
number = 0
name = rac1
cluster = ocfs2
node:
ip_port = 7777
ip_address = 192.168.2.132
number = 1
name = rac2
cluster = ocfs2
cluster:
node_count = 2
name = ocfs2
5.
Propagate the configuration file to rac2. You can rerun the steps above on rac2 to generate the
configuration file or select Cluster, Propagate Configuration on the OCFS2 Console on rac1
to propagate the configuration file to rac2.
6.
Configure the O2CB driver. O2CB is a set of clustering services that manages the communication
between the nodes and the cluster file system. Below is a description of the individual services:
NM: Node Manager that keep track of all the nodes in the cluster.conf
HB: Heartbeat service that issues up/down notifications when nodes join or leave the cluster
TCP: Handles communication between the nodes
DLM: Distributed lock manager that keeps track of all locks, its owners, and status
CONFIGFS: User space driven configuration file system mounted at /config
DLMFS: User space interface to the kernel space DLM
Perform the procedure below on both nodes to configure O2CB to start on boot.
When prompted for a value for the heartbeat dead threshold, you have to specify a value higher than
7 to prevent the nodes from crashing due to the slow IDE disk drive. The heartbeat dead threshold is
a variable used to calculate the fence time.
Fence time (seconds) = (heartbeat dead threshold -1) * 2
A fence time of 120 seconds works well in our environment. The value of heartbeat dead threshold
should be the same on both nodes.
As the root user, execute
# /etc/init.d/o2cb unload
Stopping O2CB cluster ocfs2: OK
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK
# /etc/init.d/o2cb configure
Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
without typing an answer will keep that current value. Ctrl-C
Install Oracle RAC 10g on Oracle Enterprise Linux Using VMware Server http://www.oracle.com/technology/pub/articles/chan-ubl-vmware.html?...
21 of 42 11-1-2007 7:40
will abort.
Load O2CB driver on boot (y/n) [y]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [7]: 61
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK
Format the file system. Before proceeding with formatting and mounting the file system, verify that
O2CB is online on both nodes; O2CB heartbeat is currently inactive because the file system is not
mounted.
# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Checking O2CB heartbeat: Not active
You are only required to format the file system on one node. As the root user on rac1, execute
# ocfs2console
OCFS2 Console: Select 1. Tasks, Format.
Format:
Available devices: /dev/sdb1
Volume label: oracle
Cluster size: Auto
Number of node slots: 4
Block size: Auto
2.
3. OCFS2 Console: CTRL-Q to quit.
Mount the file system. To mount the file system, execute the command below on both nodes.
# mount -t ocfs2 -o datavolume,nointr /dev/sdb1 /ocfs
To mount the file system on boot, add the following line in /etc/fstab on both nodes.
/etc/fstab
/dev/sdb1 /ocfs ocfs2 _netdev,datavolume,nointr 0 0
Create Oracle Clusterware directory. Create the directory in OCFS2 file system where the OCR
and Voting Disk will reside.
On rac1,
# mkdir /ocfs/clusterware
# chown -R oracle:dba /ocfs
You have completed the set up of OCFS2. Verify that you can read and write files on the shared
cluster file system from both nodes.

Private Interconnect: Should any nodes other than RAC nodes have one?

The contractors that set up our four-node production 10g RAC (and a standalone development server) also assigned private interconnect addresses to 2 Apache/ApEx servers and a standalone development database server.
There are service names in the tnsnames.ora on all servers in our infrastructure referencing these private interconnects- even the non-rac member servers. The nics on these servers are not bound for failover with the nics bound to the public/VIP addresses. These nics are isolated on their own switch.
Could this configuration be related to lost heartbeats or voting disk errors? We experience rac node expulsions and even arbitrary bounces (reboots!) of all the rac nodes.

I do not have access to the contractors. . . .can only look at what they have left behind and try to figure out their intention. . .
I am reading the Ault/Tumha book Oracle 10g Grid and Real Application Clusters and looking through our own settings and config files and learning srvctl and crsctl commands from their examples. Also googling and OTN searching through the library full of documentation. . .
I still have yet to figure out if the private interconnect spoken about so frequently in cluster configuration documents are the binding to the set of node.vip address specifications in the tnsnames.ora (bound the the first eth adaptor along with the public ip addresses for the nodes) or the binding on the second eth adaptor to the node.prv addresses not found in the local pfile, in the tnsnames.ora, or the listener.ora (but found at the operating system level in the ifconfig). If the node.prv addresses are not the private interconnect then can anyone tell me that they are for?

RAC node restarting!

hi
one of our RAC environment keep restarting.
i've disable the init.cssd, init.crs, init.evmd in the /etc/inittab in order to check the logs.
this is the situation:
crsd.log:
2009-02-04 00:09:00.118: [ COMMCRS][9]clsc_connect: (8000000100318640) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_node1_loud))
2009-02-04 00:09:00.132: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
2009-02-04 00:09:00.134: [ CRSRTI][1]32CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2009-02-04 00:09:08.016: [    CRSD][1]32Daemon Version: 10.2.0.2.0 Active Version: 10.2.0.2.0
2009-02-04 00:09:08.016: [    CRSD][1]32Active Version and Software Version are same
2009-02-04 00:09:08.017: [ CRSMAIN][1]32Initializing OCR
2009-02-04 00:09:08.037: [ OCRRAW][1]proprioo: for disk 0 (/dev/rdsk/ora_ocr_raw), id match (1), my id set (752560621,1028247821) total id sets (1), 1st set
(752560621,1028247821), 2nd set (0,0) my votes (2), total votes (2)
2009-02-04 00:09:08.140: [ CSSCLNT][24]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying
ocssd.log:
[    CSSD]2009-02-03 21:52:08.651 [9] >USER: clssnmHandleUpdate: NODE 1 (node1l) IS ACTIVE MEMBER OF CLUSTER
[    CSSD]2009-02-03 21:52:08.651 [9] >TRACE: clssnmHandleUpdate: diskTimeout set to (200000)ms
[    CSSD]2009-02-03 21:52:08.651 [16] >TRACE: clssnmWaitForAcks: done, msg type(15)
[    CSSD]2009-02-03 21:52:08.651 [16] >TRACE: clssnmDoSyncUpdate: Sync Complete!
[    CSSD]2009-02-03 21:52:08.722 [1] >USER: NMEVENT_SUSPEND [00][00][00][00]
[    CSSD]2009-02-03 21:52:08.724 [17] >TRACE: clssgmReconfigThread: started for reconfig (1)
[    CSSD]2009-02-03 21:52:08.749 [17] >USER: NMEVENT_RECONFIG [00][00][00][02]
[    CSSD]2009-02-03 21:52:08.749 [17] >TRACE: clssgmEstablishConnections: 1 nodes in cluster incarn 1
[    CSSD]2009-02-03 21:52:08.751 [13] >TRACE: clssgmPeerListener: connects done (1/1)
[    CSSD]2009-02-03 21:52:08.752 [17] >TRACE: clssgmEstablishMasterNode: MASTER for 1 is node(1) birth(1)
[    CSSD]2009-02-03 21:52:08.752 [17] >TRACE: clssgmChangeMasterNode: requeued 0 RPCs
[    CSSD]2009-02-03 21:52:08.752 [17] >TRACE: clssgmMasterCMSync: Synchronizing group/lock status
[    CSSD]2009-02-03 21:52:08.752 [17] >TRACE: clssgmMasterSendDBDone: group/lock status synchronization complete
[    CSSD]CLSS-3000: reconfiguration successful, incarnation 1 with 1 nodes
[    CSSD]CLSS-3001: local node number 1, master node number 1
[    CSSD]2009-02-03 21:52:08.753 [17] >TRACE: clssgmReconfigThread: completed for reconfig(1), with status(1)
[    CSSD]2009-02-03 21:52:08.863 [10] >TRACE: clssgmClientConnectMsg: Connect from con(80000001008fd2a0) proc(8000000100ae26a8) pid() proto(10:2:1:1)
[    CSSD]2009-02-03 21:52:08.864 [10] >TRACE: clssgmClientConnectMsg: Connect from con(8000000100ae0128) proc(8000000100ae2a10) pid() proto(10:2:1:1) from con(8000000100aa32c0) proc(8000000100aa5b90) pid() proto(10:2:1:1)
alertlog:
[cssd(2535)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 .
2009-02-03 23:55:20.821
[cssd(2575)]CRS-1605:CSSD voting file is online: /dev/rdsk/ora_voting_raw. Detai ls in /work/crs/product/10.2/crs/log/lourmel/cssd/ocssd.log.
2009-02-03 23:55:28.376
evmd.log:
Oracle Database 10g CRS Release 10.2.0.2.0 Production Copyright 1996, 2004, Oracle. All rights reserved
2009-02-04 00:08:58.331: [    EVMD][1]32EVMD waiting for CSS to be ready err = 3
2009-02-04 00:08:59.939: [ COMMCRS][9]clsc_connect: (800000010007d658) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_node1_loud))
2009-02-04 00:08:59.946: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
2009-02-04 00:08:59.948: [    EVMD][1]32EVMD waiting for CSS to be ready err = 3
2009-02-04 00:09:07.596: [ CSSCLNT][1]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying
syslog:
Feb 4 00:08:41 lourmel syslog: Oracle Cluster Ready Services starting up automatically.
Feb 4 00:08:45 lourmel sfd[2153]: starting the daemon.
Feb 4 00:08:45 lourmel su: + tty?? root-orac
Feb 4 00:08:45 lourmel krsd[2152]: Delay time is 300 seconds
Feb 4 00:08:43 lourmel syslog: Oracle Cluster Ready Services starting up automatically.
Feb 4 00:08:52 lourmel above message repeats 2 times
Feb 4 00:08:52 lourmel syslog: Cluster Ready Services completed waiting on dependencies.
Feb 4 00:08:53 lourmel syslog: Running CRSD with TZ =
when i checked(befor the restart) the command crs_stat i got the message:
ORA-0184: Cannot communicate wirh CRS
crsctl check crs gives us:
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
as i said befor, the machine always restarting
anyone have an idea?? please

Dear All,
I recently upgrade the Few RAC setups with Oracle 10g Patchset 3 (10.2.0.4) on Linux Servers
In one of the RAC setup, found servers are rebooting daily. The same setup was working fine and problem started only after applying the Patchset. Checked all the logs and Found nothing relevant.
Then i checked the things which added with this Patchset.
The Most interesting found , Oracle Added a New Daemon- oprocd.
# ps -efl | grep oprocd
4 S root 6440 6063 0 -40 - - 2114 - Mar03 ? 00:00:00 /opt/oracle/product/10.2.0/crs/bin/oprocd.bin run -t 1000 -m 500 -hsi 5:10:50:75:90 -f
These are Interesting Points about above line
1.This Process is running by root user
2. With Highest Priority -40
3. Probing every Seconds (t 1000)
4. waiting CPU response for 500 Milliseconds ( -m 500 means margin time is 500 Milli Seconds)
5. Process status is Fatal (-f)
Now I am concluding these points- This daemon will probe cpu every second and wait for response within 500 Mill seconds. If in the 500 Milli second not getting any response from the cpu, will assume the CPU is hang and try to Reboot the Machine. The OPERATING SYSTEM will not get enough time to write the system logs and server reboots.
So the solution is increase the Margin time for 500 Milli second to 10 seconds.
These are following steps to increase the Margin time.
Please Remember- The Modification process need Downtime and You need to stop cluster service in all member nodes.
1. Stop The CRS Process
#crsctl stop crs
#<CRS_HOME>/bin/oprocd stop
2. Ensure that Clusterware stack is down and not running
#ps -ef |egrep "crsd.bin|ocssd.bin|evmd.bin|oprocd"
This should return no processes.
3. From one node of the cluster, change the value of the "diagwait" parameter to 13 by issuing the command as root:
#crsctl set css diagwait 13 -force
4. Check if diagwait is successfully set.
#crsctl get css diagwait
5. Restart the Oracle Clusterware on all the nodes by executing:
#crsctl start crs
(Note- If facing any problem to restarting the CRS services, ASM and Database, You can reboot the Nodes.The Cluster and Database will come automatically due to init startup scripts.)
6. The oprocd daemon process will show with -m 10000
# ps -efl| grep oprocd
# 4 S root 6440 6063 0 -40 - - 2114 - Feb02 ? 00:00:00 /opt/oracle/product/10.2.0/crs/bin/oprocd.bin run -t 1000 -m 10000 -hsi 5:10:50:75:90 -f
Rollback Procedure-
If You need to unset oprocd value due any reason
#crsctl unset css diagwait
I am confident, The abnormal RAC Node restart problem will solve with this workaround.
Regards,
Sumit
Bangalore,India

Solaris RAC nodes re-booting

I have a pre-production 2-node cluster running on Solaris 10, Oracle 10.2.0.3 with the Oracle CRS, and using a NetApp filer as the shared storage.
I also have a separate Solaris server running Grid Control 10.2.0.3, with the repository as one of the databases on the RAC (don't know if this is relevant to my problem).
Periodically both RAC nodes reboot, with no trace of why (the GC server is fine). There is nothing logged in the Solaris logs (messages file), CRS logs, Oracle logs or the NetApp logs.
All that is shown is the relevant service starting up following the shutdown.
Has anyone any experience of this, or any thoughts on which component may cause such an issue?
Thanks in advance
Bob

What type of Sun hardware are you using?
Below is the Action Plan Oracle support sent me on my SR on this issue, not sure if any of this was provided to you or would be of help.
ACTION PLAN
============
1. there is nothing on the files at all that sheds any light on the issue
agian 3 sperate sets of clusters all losing all nodes at the same tiem is a very strange occurance. Please be sure to have the admin look for
anything in common wiht all custers.
2. advice placing oswatcher on the systems Note.301137.1 Ext/Pub OS Watcher User Guide
if we should have another occurances we will want the oswatcher logs for 1 hr before issue thru issue
also see if the unix admin perhaps has any os stats from this occurance
3. advice settign ntpd to run with -x option I do see that you are having negative time changes
at times
-x will give us a skew rather then an abbrupt time change
4. advice setting this when you can
Please do the following
set the diagwait parameter:
crsctl set css diagwait N [-force]
Where N is the number of seconds to wait for a filesystem sync to
complete (after this wait the node will reboot regardless of whether the
sync has completed). This change must be made with the clusterware
down, which will require the '-force', or with the stack up on just 1
node, after which the stack on that node must be restarted before the
stack starts up on any of the other nodes.
N should be set to 25 (25 seconds)
5. advice that you have with pcw mlr#6 Patch 5980915 on the systems as well
but I do not believe that this was an oracle bug the reason for placing the patch on is for advanced diagnostics that is in that patchset
6. the two issues sun is workking on
Sun is working to resolve a time skew issue and a Solaris 10 kernel SIGALRM Sun#6292092 in addition to Sun#6595936.
7. we do have a diagnostic oprocd that soem sites have used but on thier test systems. It stops reboots adn dumps information but I have
been hesitant to place it on production boxes if you continue to have issues we may consider download the oprocd_skewfix_noreboot fro
m Bug 6279879 but at this time I do not belvve that is warrented

What is best use of 1400 gb SGA (2 rac nodes 768gb each)

currently using 11.2.0.3.0 on unix sun sever with 2 RAC nodes each 8 UltraSPARC-T1 cpus (came out in 2005) four threads each so oracle sees 32 CPUS very slow(1.2 gb). Database is 4TB in size on regular SAN (10k speed).
8gb SGA.
New boss wants to update system to the max to get best performance possible Money is a concern of course but budget is pretty high, Our use case is 12-16 users at same time, running reports some small others very large (return single row or 10000s or rows). reports take 5 sec to 5 minutes, Our job is get the fastest system possible, We have total of 8 licenses available so we can have 16 cores. We are also getting a 6tb all flash SSD array for database. we can get any CPU we want but we cant use parallel query server due to all kinds of issues we have experienced (too many slaves, RAC interconnect saturation etc, whack-a-mole). sparc has too many threads and without PS oracle runs query in single thread.
we have speced out the following system for each RAC node
HP ProLiant DL380p Gen8 8 SFF server
2 Intel Xeon E5-2637v2 3.5GHz/4-core cpus
768 gb ram
2 HP 300GB 6G SAS 15K drives for database software
this will give us total of 4 Xeon E5-2637v2 cpus 16 cores total (,5 factor for 8 licenses) and 1536 ram (leaving ~1400 for sga). this will guarantee an available core for each user. we intend to create very very large keep pool around 300 gb for each node that will hold all our dimension tables. this we hope will reduce reads from the SSD to just data from fact tables.,
Are we doing a massive overkill here? the budget for this was way less than what our boss expected. will that big an sga be wasted will say a 256gb be fine. or will oracle take advantage of it and be able to keep most blocks in there.
will an sga that big cause oracle problems due to overhead of handling that much ram?

Current System:
===========
a. Version : 11.2.0.3
b. Unix Sun
c. CPU - 8 cpus with 4 threads => 32 logical cpus or cores
d. database 4TB
e. SAN - 10k speed disk drives
f. 8gb SGA
g. 1.2 gb ??
h. Users --> 12-16 concurrent and run reports varying size
i. reports elasped time 5 sec to 5 mins
j. cpu license -->8
Target System
===========
a. Version: 11.2.0.3
b. HP ProLiant DL380p Gen8 8 SFF server
c. RAM --> 768 GB
d. 2 HP 300GB 6G SAS 15K drives for database software
e. large keep pool -->90 gb to hold all dimension tables.
f. SSD to just data from fact tables
g. SGA -->256gb
Reassessment of the performance issues of current system appears to be required.Good performance tuning expert is required to look into tuning issues of current application by analyzing awr performance metrics . If 8GB SGA is not enough,then reason behind so is that queries running in the system are not having good access path to select lesser data to avoid flushing out of recent buffers from different tables involved in the query. Until those issues are identified , wherever you go, performance issue wont be going away as table size increase in future , problem will reappear.Even if the queries are running with more FULL Scan , then re-platforming to Exadata might be right decision as Exadata has smart scan , cell offloading feature which works faster and might be right direction for best performance and best investment for future.Compression (compress for OLTP) could be one of the other feature to exploit to improve further efficiency while reading the lesser block in lesser read time.
Investment in infrastructure will solve a few issue in short term but long term issue will again arise.
Investment in identifying the performance issues of current system would be best investment in current scenario.

Can't mount volume

i've got a second hard drive and in doing some copying and the program coming to some error and crashing (no error reported, but it was while using Carbon Copy Cloner and after the copying was finished.
The disk was showing issues and Disk Repair in Disk Utility said it had a Volume Header error that it couldn't fix. when I have the drive in, I can't even boot up (even though it's not my startup drive) It just sits at a blank, gray screen (as if it's about to show the dark gray apple logo and spinning busy thing at the bottom, but it doesn't.
Sounds like this volume header error is causing the problem, but I dont have a clue on hwo to fix it, since I can't even mount the thing.
What if I was able to boot up with the drive's IDE and power plugged in, but the computer won't try to mount it. Instead, I'll mount it using diskutil at the command line. Maybe if I can mount the correct partition, it will 'ignore' the boot partition and I can get it mounted as a temp fix? I just don't know the /dev/xyz path
EDIT: this kind of makes sense to me... if i can go into single user mode on the good drive, plug the second drive in and then manually mount it. if it works, i can run fsck. thoughts?

I tried your advice and some other things. Here's what I've come up with:
- both hard drives work
- both can be mounted, but NOT AT THE SAME TIME
- having both connected (physically) at startup causes the computer to hang at the gray screen and never displaying anything more (no apple logo either). there's a little pin beside where the IDE cable goes in, and both drives are properly connected as Master w/ Slave and Slave, respectively.
- i can boot into both. in order to make the volume on the 2nd drive, i just had it archive the OS on there and replace it off my OS X cd. I told it to keep the user data for the new installation, but when i got to the login screen, no usernames worked. by default, the clue option wasn't on, so i couldn't even see if it somehow had an old copy of the password file (although i would think that's a stretch)
The only other info I think I've left out is that the second drive's volume still has its original name, so it's not like it somehow got to become literally an exact duplicate of the original, first hard drive's volume (initially, i wondered if that was the case and that the computer thought it was trying to mount the same volume from two locations)

Question on Rebooting RAC Nodes

Hi, I heard that when rebooting all RAC nodes, one has to wait at least 5 minutes between each node reboot. So you would reboot node 1 at 0 time, node 2 5 minutes later etc.
However, I could not find any documentation on this, can someone please point me to the right place to look? Thanks.

I have not heard that before. I generally use srvctl to stop/start the databases, I do not think it waits 5 minutes between starting nodes as it does not take 15 minutes to stop/start the databases. As far as rebooting the hosts, the only time interval between machines was the time it took to send the reboot command to the host.

Multiple Standby Databases on same RAC nodes.

We have a 3 node Oracle 10gR2 RAC production environment on site A and a 3 node Oracle 10g RAC standby environment on site B. Both use like HW and OS - HP BL45p with RHEL AS 4.x.
Can we have heterogeneous standby databases(logical and physical) running on the same RAC nodes?
Can the 2 apply processes (MRP-manageed recovery process & LSP-logical standby process) coexists on the same set of nodes in a cluster at the same time. Are there any conflicts or limitations?
Is there any documentation that supports this?

Would Active Data Guard give you the best of both worlds?
The caveat might be your SID_LIST_LISTENER setup.
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /u01/app/oracle/product/11.2.0)
(PROGRAM = extproc)
(SID_DESC =
(global_dbname = <database1>_DGMGRL.yourdomain)
(ORACLE_HOME = /u01/app/oracle/product/11.2.0)
(sid_name = <database1>)
(SID_DESC =
(global_dbname = <database2>_DGMGRL.yourdomain)
(ORACLE_HOME = /u01/app/oracle/product/11.2.0)
(sid_name = <database2>)
I have a server with ten standbys on it currently and no issues.
As long as the version is the same you should be good.

Updater still "can't mount" while trying to restore

I posted the following in March and got nowhere:
"A few days ago, my iPod nano started to malfunction. It would act if it was continually resetting - it would turn on, display the Apple logo, then turn on, and start over again. I read a post on here that advised to hold down the center button and the play button simultaneously to switch to disk mode, then restore using the updater. This worked - or so I thought.
After doing these steps, it seemed that functionality had been restored. However, I had only loaded two songs on it before my computer acted as if the iPod had disappeared. I turn on the iPod and see the folder icon with the URL to the support site. I can switch it to disk mode, but I can't restore it.
I open the updater and wait for it to prompt me to plug in the iPod. I do this, then wait for it to acknowledge the iPod. Then I press the restore button. It starts to restore, but after a few moments it says "can't mount iPod". I have tried every USB port on my computer, I have tried rebooting my computer, I have tried resetting the iPod. Nothing seems to correct this problem. Please help!"
Someone suggested trying to change the drive letter, but the disc manager says the drive is unreadable. Then he said he would get back to me, but never did. Please, somebody, help me with this.

Hello,
If a sad iPod icon or an exclamation point and folder icon appears on your iPod’s screen, or with sounds of clicking or HD whirring, it is usually the sign of a hard drive problem and you have the power to do something about it now. Your silver bullet of resolving your iPod issue – is to restore your iPod to factory settings.
http://docs.info.apple.com/article.html?artnum=60983
If you're having trouble, try these steps at different levels one at a time until the issue is resolved. These steps will often whip your iPod back into shape.
Make sure you do all the following “TRYs”
A. Try to wait 30 minutes while iPod is charging.
B. Try another FireWire or USB through Dock Connector cable.
C. Try another FireWire or USB port on your computer .
D. Try to disconnect all devices from your computer's FireWire and USB ports.
E. Try to download and install the latest version of iPod software and iTunes
http://www.apple.com/ipod/download/
http://www.apple.com/itunes/download/
F. Try these five steps (known as the five Rs) and it would conquer most iPod issues.
http://www.apple.com/support/ipod/five_rs/
G. Try to put the iPod into Disk Mode if it fails to appear on the desktop
http://docs.info.apple.com/article.html?artnum=93651
If none of these steps address the issue, you may need to go to Intermediate level listed below in logical order. Check from the top of the lists to see if that is what keeping iPod from appearing on your computer in order for doing the Restore.
Intermediate Level
A. Try to connect your iPod with another computer with the iPod updater pre-installed.
B. Still can’t see your iPod, put it in Disk Mode and connect with a computer, instead of doing a Restore on iPod Updater. Go and format the iPod instead.
For Mac computer
1. Open the disk utility, hope your iPod appears there (left hand side), highlight it
2. Go to Tab “Partition”, click either “Delete” or “Partition”, if fails, skip this step and go to 3
3. Go to Tab “Erase” , choose Volume Format as “MAC OS Extended (Journaled), and click Erase, again if fails, skip it and go to 4
4. Same as step 3, but open the “Security Options....” and choose “Zero Out Data” before click Erase. It will take 1 to 2 hours to complete.
5. Eject your iPod and do a Reset
6. Open the iPod Updater and click “Restore”
For Window computer
Go to folder “My Computer”
Hope you can see your iPod there and right click on the iPod
Choose “Format”. Ensure the settings are at “Default” and that “Quick Format” is not checked
Now select “Format”
Eject your iPod and do a Reset
Open the iPod Updater and click “Restore”
In case you do not manage to do a “Format” on a window computer, try to use some 3rd party disk utility software, e.g.“Partition Magic”.
C. Windows users having trouble with their iPods should locate a Mac user. In many cases when an iPod won't show up on a PC that it will show up on the Mac. Then it can be restored. When the PC user returns to his computer the iPod will be recognized by the PC, reformatted for the PC, and usable again. By the way, it works in reverse too. A Mac user often can get his iPod back by connecting it to a PC and restoring it.
Tips
a. It does not matter whether the format is completed or not, the key is to erase (or partly) the corrupted firmware files on the Hard Drive of the iPod. After that, when the iPod re-connected with a computer, it will be recognized as an fresh external hard drive, it will show up on the iPod updater.
b. It is not a difficult issue for a Mac user to find a window base computer, for a PC user, if they can’t find any Mac user, they can go to a nearest Apple Shop for a favor.
c. You may need to switch around the PC and Mac, try to do several attempts between “Format” and “Restore”
Advance Level
A. Diagnostic mode solution
If you have tried trouble shooting your iPod to no avail after all the steps above, chances are your iPod has a hardware problem. The iPod's built-in Diagnostic Mode is a quick and easy way to determine if you have a "bad" iPod.
You need to restart your iPod before putting it into Diagnostic Mode. Check that your hold switch is off by sliding the switch away from the headphone jack. Toggle it on and off to be safe.
Press and hold the following combination of buttons simultaneously for approximately 10 seconds to reset the iPod.
iPod 1G to 3G: "Menu" and "Play/Pause"
iPod 4G+ (includes Photo, Nano, Video, and Mini): "Menu" and "Select"
The Apple logo will appear and you should feel the hard drive spinning up. Press and hold the following sequence of buttons:
iPod 1G to 3G: "REW", "FFW" and "Select"
iPod 4G+ (includes Photo, Nano, Video, and Mini): "Back" and "Select"
You will hear an audible chirp sound (3G models and higher) and the Apple logo should appear backwards. You are now in Diagnostic Mode. Navigate the list of tests using "REW" and "FFW". The scroll wheel will not function while in diagnostic mode. For further details on Diagnostic mode can be found at http://www.methodshop.com/mp3/ipodsupport/diagnosticmode/
Try to do the 5in1, HDD R/W and HDD scan tests. Some successful cases have been reported after the running the few tests under the Diagnostic mode. In case it does not work in your case, and the scan tests reports show some errors then it proves your iPod has a hardware problem and it needs a repairing service.
B. Format your iPod with a start disk
I have not tried this solution myself, I heard that there were few successful cases that the users managed to get their iPod (you must put your iPod in disk mode before connecting with a computer) mounted by the computer, which was booted by a system startup disk. For Mac, you can use the Disk Utility (on the Tiger OS system disk), for PC user, you can use the window OS system disk. Try to find a way to reformat your iPod, again it does not matter which format (FAT32, NTFS or HFS+) you choose, the key is to erase the corrupted system files on the iPod. Then eject your iPod and do a Reset to switch out from Disk Mode. Reboot your computer at the normal way, connect your iPod back with it, open the iPod updater, and hopefully your iPod will appear there for the Restore.
If none of these steps address the issue, your iPod may need to be repaired.
Consider setting up a mail-in repair for your iPod http://depot.info.apple.com/ipod/
Or visit your local Apple Retail Store http://www.apple.com/retail/
In case your iPod is no longer covered by the warranty and you want to find a second repairing company, you can try iPodResQ at your own risk
http://www.ipodresq.com/index.php
Just in case that you are at the following situation
Your iPod warranty is expired
You don’t want to pay any service charges
You are prepared to buy a new one
You can’t accept the re-sell value of your broken iPod
Rather than leave your iPod as paper-weight or throw it away.
You can try the following, but again, only do it as your last resort and at your own risk.
Last Resort Level
1. . Disconnecting the Hard Drive and battery inside the iPod – Warning !! Your iPod warranty will be waived once you open the iPod.
In Hong Kong there are some electronic shops offering an iPod service for Sad iPod, the first thing they do is to open up the iPod’s case and disconnecting the battery and the Hard Drive from the main board of the iPod. Wait for 5-10 minutes and reconnecting them back. The reason behind which I can think of is to do a fully reset of a processor of the iPod. In case you want do it itself and you believe that you are good on fixing the electronics devices and have experience to deal with small bits of electronic parts, then you can read the following of how to open the iPod case for battery replacement (with pictures)
http://www.kokopellimusic.us/KM_instructions.htm
Have I tried this myself? No, I am afraid to do it myself as I am squeamish about tinkering inside electronic devices, I have few experiences that either I broke the parts (which are normally tiny or fragile) or failed to put the parts back to the main case. Therefore, I agree with suggestion to have it fixed by a Pro.
2. Do a search on Google and some topics on this discussion forum about “Sad iPod”
Strange error on iPod (probably death)
http://discussions.apple.com/thread.jspa?threadID=435160&start=0&tstart=0
Sad Face on iPod for no apparent reason
http://discussions.apple.com/thread.jspa?threadID=336342&start=0&tstart=0
Meeting the Sad iPod icon
http://askpang.typepad.com/relevanthistory/2004/11/meeting_thesad.html#comment-10519524
I am not suggesting that you should follow as well, but just read them as your reference. You are the person to make the call.

Pointing existing RAC nodes to a fresh Shared Storage discarding old one

Hi,
I have a RAC Setup with the Primary Database on Oracle 10gR2.
For this setup, there is a Physical Standby Database Setup (using DataGuard configuration) also with 30min delay.
Assume that the "Shared Storage" of the Primary DB fails completely.
In the above scenario, my plan is to refresh a "fresh" shared storage device using Physical Standby Database Setup and then "point" the RAC nodes to the new "Shared Storage".
Is this possible?
Simply put, how can I refresh the Primary database using the Standby Database?
Please help with the utilities (RMAN, DatGuard, other non-Oracle products etc.) that can be used to do this.
Regards

Does following Shared Device configuration is fine for 10g RAC on Windows 2003?
. 1 SCSI drive
• Two PCI network adapters on each node in the cluster.
• Storage cables to attach the shared storage device to all computers.
regard.

Can't mount diskgroup on second RAC node (ss3)

Similar Messages

Maybe you are looking for