ORA-29702

Today suddenly our one UNIX server containing the one node of Oracle 10g database cluster
Got reboot, after that hard reboot when the server started and I try to start that node it is showing the following error
SQL> startup
ORA-29702: error occurred in Cluster Group Service operation
SQL>
While cluster is up:
# cmviewcl
CLUSTER STATUS
kumaon up
NODE STATUS STATE
upcld1 up running
PACKAGE STATUS STATE AUTO_RUN NODE
prod1 up running enabled upcld1
NODE STATUS STATE
upcld2 up running
PACKAGE STATUS STATE AUTO_RUN NODE
prod2 up running enabled upcld2
Please suggest, what to do?
With warm regards!
Deep

Trace File
==============
/u01/app/oracle/product/10.2.0/db_1/admin/kumaondb/bdump/kumaondb2_lmon_20386.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1
System name:     HP-UX
Node name:     upcld2
Release:     B.11.23
Version:     U
Machine:     9000/800
Instance name: kumaondb2
Redo thread mounted by this instance: 0 <none>
Oracle process number: 0
Unix process pid: 20386, image: oracle@upcld2
Ioctl ASYNC_CONFIG error, errno = 1
*** SERVICE NAME:() 2006-08-07 21:16:45.865
*** SESSION ID:(167.1) 2006-08-07 21:16:45.862
GES resources 4161 pool 3
GES enqueues 6044
GES IPC: Receivers 2 Senders 2
GES IPC: Buffers Receive 1000 Send (i:1050 b:483) Reserve 301
GES IPC: Msg Size Regular 408 Batch 8192
Batching factor: enqueue replay 201, ack 224
Batching factor: cache replay 126 size per lock 64
kjxggin: receive buffer size = 32768
clsc_connect: (80000001004601a0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_upcld2_crs))
2006-08-07 21:16:48.810: [ CSSCLNT]clsssInitNative: connect failed, rc 9
kgxgncin: CLSS init failed with status 3
kjxgmin: kgxgncin fails - (2)
kjxggin: generic group layer init fails
*** 2006-08-07 21:16:48.810
Global Enqueue Service Shutdown
*** 2006-08-07 21:16:48.849
global enqueue service detaching from CM:pid=20386

Similar Messages

ORA-29702: error occurred in Cluster Group Service operation

Hi,
After shutting down one node I'm trying but without success to start this node up:
SQL> startup
ORA-29702: error occurred in Cluster Group Service operation
I got this message.
Here is a log from file: orcl2_lmon_13886.trc
/oracle/admin/orcl/bdump/orcl2_lmon_13886.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters and Data Mining options
ORACLE_HOME = /oracle/DB
System name:     Linux
Node name:     DB2
Release:     2.6.18-53.el5
Version:     #1 SMP Wed Oct 10 16:34:19 EDT 2007
Machine:     x86_64
Instance name: orcl2
Redo thread mounted by this instance: 0 <none>
Oracle process number: 5
Unix process pid: 13886, image: oracle@DB2 (LMON)
*** SERVICE NAME:() 2008-12-19 10:08:26.214
*** SESSION ID:(1102.1) 2008-12-19 10:08:26.214
GES resources 21522 pool 8
GES enqueues 33158
GES IPC: Receivers 3 Senders 3
GES IPC: Buffers Receive 1000 Send (i:2750 b:1934) Reserve 1000
GES IPC: Msg Size Regular 408 Batch 8192
Batching factor: enqueue replay 201, ack 224
Batching factor: cache replay 126 size per lock 64
kjxggin: receive buffer size = 32768
clsc_connect: (0x1d19a030) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_db2_crs))
2008-12-19 10:08:29.819: [ CSSCLNT]clsssInitNative: connect failed, rc 9
kgxgncin: CLSS init failed with status 3
kjxgmin: kgxgncin fails - (2)
kjxggin: generic group layer init fails
*** 2008-12-19 10:08:29.819
Global Enqueue Service Shutdown
*** 2008-12-19 10:08:29.820
global enqueue service detaching from CM:pid=13886
I did check listener, it did not work but I start it up:
LSNRCTL> status
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
Alias LISTENER
Version TNSLSNR for Linux: Version 10.2.0.3.0 - Production
Start Date 19-DEC-2008 10:49:42
Uptime 0 days 0 hr. 3 min. 55 sec
Trace Level off
Security ON: Local OS Authentication
SNMP ON
Listener Parameter File /oracle/DB/network/admin/listener.ora
Listener Log File /oracle/DB/network/log/listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost.localdomain)(PORT=1521)))
The listener supports no services
The command completed successfully
LSNRCTL>
Could you give me a guide how to start up this node? or where is a problem?

A quick search of the Oracle documentation on the error number comes up with this
http://download-west.oracle.com/docs/cd/B10501_01/server.920/a96525/e29250.htm#1000969
ORA-29702 error occurred in Cluster Group Service operation
Cause: An unexpected error occurred while performing a CGS operation.
Action: Verify that the LMON process is still active. Also, check the Oracle LMON trace files for errors.
Justin
Distributed Database Consulting, Inc.
http://www.ddbcinc.com/askDDBC

ORA-29701과 ORA-29702 ERROR의 원인과 조치방법(OGMS관련 ORACLE ERROR)

제품 : ORACLE SERVER
작성날짜 : 2004-08-13
SCOPE
Standard Edition 에서는 Real Application Clusters 기능이 10g(10.1.0) 이상 부터 지원이 됩니다.
ORA-29701과 ORA-29702 ERROR의 원인과 조치 방법(OGMS 관련 Oracle ERROR)
======================================================================
Explanation
ORA-29701과 ORA-29702는 OGMS(Oracle Group Membership Services)에 문제가
발생했을 때 Oracle에서 로그를 남기는 에러이다.
1. OGMS란?
OPS 운영 환경에서 노드 간의 통신을 위해 IDLM(Integrated Distributed Lock Manager)과
다른 오라클 컴포넌트들은 OS의 Cluster Manager와 통신을 해야 한다.
이 때에 인터페이스를 제공하여 주는 것이 바로 OGMS이다.
GMS는 일련의 Application Programming Interface (API)들로 구성되어 있으며,
OPS 컴포넌트들은 이 API들을 통해 Cluster Manager와 통신을 하게 된다.
GMS는 클러스터의 운영 상태를 모니터하고, 클러스터와의 instance connection을
관장하며, 메시지를 전달하는 역할을 담당한다.
platform에 따라 GMS는 ogmsctl 프로그램을 사용해서 따로 기동을 시켜 주어야만
하거나, opsctl 프로그램을 사용할 때 자동적으로 기동된다.
2. Error 분석
1) ORA-29701
a. 오라클에서 GMS에 접속할 수 없을 경우 발생한다.
이것은 Oracle이 startup 될 때, GMS가 실행되지 않고 있을 때 발생한다. 이와
같은 경우에는 'ogmsctl status' 명령을 상용하여 GMS가 startup되었는지 확인
하여야 한다.
b. lmon( GMS client )이 communication file의 위치를 찾지 못할 경우 발생한다.
기본적으로 사용하는 디렉토리인 /tmp/.ogms를 사용하지 않을 경우 GMS home이
지정되어야 한다. OGMS home directory를 별도로 지정하여 사용할 경우에는
init.ora 파라미터 파일에서 ogms_home 파라미터 값을 지정해 주어야 한다.
c. 기타 발생 원인은 GMS에 틀린 internal function(skgxn)이 사용되거나
GMS가 local request에 대한 서비스를 할 수 없거나 CM subsystem에 문제가 있을
경우 등등이 있다.
2) ORA-29702
GMS 서비스에 예상치 못한 에러가 발생했을 경우 로그에 남게 된다. GMS가 실행
중인지를 확인해 보고, 내부 에러로 인해 GMS가 스스로 shutdown 되었다면
daemon file에 기록된 트레이스 정보를 살펴보아야 한다.
Reference Documents
none

ORA-15001: diskgroup "DATA" does not exist or is not mounted in 11gR2

Dear all,
the system is HP-UX 11.31
the new RAC infrastructure is 11gR2 11.2.0.2 (2nodes using ASM)
i am restoring the 10G 10.2.0.4 database to this infrastructure
copy binary files over to new oracle home,
relink all binaries with option rac_off
--startup nomount
--restore controlfile
--mount database
--catalog the restore files
then run restore
run {
restore database ;
recover database;
i always get this error, i did not see this error in 11gR1( restore few times to 11gR1)
verified the diskgroup DATA and i was mounted
ORA-19504: failed to create file "+DATA/OCP/data1.dbf"
ORA-17502: ksfdcre:3 Failed to create file +DATA/OCP/data1.dbf
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Manager
Please give me an advice.
Thanks,

found this note in Metalink and it had solved my issue.
Unable To Create 10.1 or 10.2 or 11.1(< 11gR2) ASM RAC Databases (ORA-29702) Using Brand New 11gR2 Grid Infrastructure Installation . [ID 946332.1]

ORA-00313: open failed for members of log group 4 of thread 2

Hi.
i have found the below error in one of RAC instance.Could you please advise on this since i am new to RAC env
ORA-00313: open failed for members of log group 4 of thread 2
ORA-00312: online log 4 thread 2: '+NEWDISK_GROUP/DEMO/onlinelog/group_4.1453.1029312423'
ORA-17503: ksfdopn:2 Failed to open file +NEWDISK_GROUP/DEMO/onlinelog/group_4.1453.1029312423
ORA-15032: not all alterations performed
ORA-29702: error occurred in Cluster Group Service operation
ORA-29702: error occurred in Cluster Group Service operation
Thanks
Sai

any update on this?

10.1.0.4 on SLES 9 SP1 ora-03113

Hello,
SLES9 SP1 is on PIII with two 800MHz procesors and 2GB RAM,
on four SATA 156GBhard disks are created two RAID1 arays,
on one RAID is OS with 1GB swap and Oracle 10.1.0.4 enterprise edition, on second RAID are two raw partitions each 70GB ....
I start DBCA custom database and configure one ASM disk group by using
two raw partitions (/dev/sdb1, /dev/sdb2),then I add
4 new tablespaces with datafiles of 2GB each autoextend on 10MB, change character set,
deselect spatial & data mining....
Instalation starts but stops betwen 13&14% while creating data dictionary with error ORA-03113
end-of-file on comunication chanell
Any suggestion is extra welcome.....
Thanks 4 Your time

HERE ARE MY ALERT LOGS AND TRACE FILES:
/admin/+ASM/bdump>cat alert_+ASM.log
Tue Jun 28 16:22:50 2005
SUCCESS: diskgroup DISKOVI was mounted
Tue Jun 28 16:22:51 2005
NOTE: recovering COD for group 1/0xac774a8b (DISKOVI)
SUCCESS: completed COD recovery for group 1/0xac774a8b (DISKOVI)
Tue Jun 28 16:29:19 2005
Errors in file /opt/oracle/admin/+ASM/bdump/+asm_smon_5880.trc:
ORA-29702: error occurred in Cluster Group Service operation
ORA-29702: error occurred in Cluster Group Service operation
Tue Jun 28 16:29:19 2005
SMON: terminating instance due to error 29702
Instance terminated by SMON, pid = 5880
Tue Jun 28 16:34:18 2005
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
/admin/+ASM/bdump> cat +asm_smon_5880.trc
/opt/oracle/admin/+ASM/bdump/+asm_smon_5880.trc
Oracle Database 10g Enterprise Edition Release 10.1.0.4.0 - Production
With the Partitioning, OLAP and Data Mining options
ORACLE_HOME = /opt/oracle/product/10g
System name: Linux
Node name: zeus
Release: 2.6.5-7.139-smp
Version: #1 SMP Fri Jan 14 15:41:33 UTC 2005
Machine: i686
Instance name: +ASM
Redo thread mounted by this instance: 0 <none>
Oracle process number: 7
Unix process pid: 5880, image: oracle@zeus (SMON)
*** 2005-06-28 16:29:19.298
*** SERVICE NAME:() 2005-06-28 16:29:19.298
*** SESSION ID:(33.1) 2005-06-28 16:29:19.298
clssgsGroupGetStatus: clsssRecvMsg failed 3 0)
clssgsGroupGetStatus: returning 8
kgxgnpstat: received ABORT event from CLSS
Group services Error [NM abort event ] @ 28019:715
error 29702 detected in background process
ORA-29702: error occurred in Cluster Group Service operation
ORA-29702: error occurred in Cluster Group Service operation
/admin/twrh/bdump>cat alert_twrh.log
ORA-15064: Message 15064 not found; No message file for product=RDBMS, facility=ORA
ORA-01089: Message 1089 not found; No message file for product=RDBMS, facility=ORA
ORA-07445: exception encountered: core dump [kksParseCursor()+9] [SIGSEGV] [Address not mapped to object] [0x0] [] []

Database is not starting after RAC installation

Hi All,
I have installed 10g R2 2-node RAC on RHEL4 without any problem. But after the installation the database is not starting and i'm not able to connect. It is showing the below error..
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux Error: 2: No such file or directory
Can anybody help what might be the problem?
Thanks,
Praveen

Hi,
I'm giving the contents of alter log file from the first node...
Mon Feb 19 16:45:59 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =36
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
sga_target = 1073741824
db_block_size = 8192
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database_instances= 1
db_create_file_dest = +ORCL_DATA1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 16:46:00 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=31282
DIAG started with pid=3, OS id=31284
PSP0 started with pid=4, OS id=31286
LMON started with pid=5, OS id=31300
LMD0 started with pid=6, OS id=31315
MMAN started with pid=7, OS id=31317
DBW0 started with pid=8, OS id=31319
LGWR started with pid=9, OS id=31321
CKPT started with pid=10, OS id=31323
SMON started with pid=11, OS id=31325
RECO started with pid=12, OS id=31327
CJQ0 started with pid=13, OS id=31329
MMON started with pid=14, OS id=31331
Mon Feb 19 16:46:00 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=15, OS id=31333
Mon Feb 19 16:46:00 2007
starting up 1 shared server(s) ...
Mon Feb 19 16:46:00 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 16:46:01 2007
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Reconfiguration complete
Mon Feb 19 16:46:01 2007
CREATE DATABASE "bsaprod"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
DATAFILE SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
CHARACTER SET WE8ISO8859P1
NATIONAL CHARACTER SET AL16UTF16
LOGFILE GROUP 1 SIZE 51200K,
GROUP 2 SIZE 51200K
USER SYS IDENTIFIED BY *USER SYSTEM IDENTIFIED BY
Mon Feb 19 16:46:01 2007
Starting background process ASMB
ASMB started with pid=19, OS id=31365
Starting background process RBAL
RBAL started with pid=20, OS id=31369
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 16:46:05 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 16:46:06 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 16:46:06 2007
Database mounted in Exclusive Mode
Mon Feb 19 16:46:09 2007
Successful mount of redo thread 1, with mount id 3309011657
Assigning activation ID 3309011657 (0xc53b82c9)
Thread 1 opened at log sequence 1
Current log# 1 seq# 1 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 16:46:09 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 16:46:09 2007
SMON: enabling cache recovery
Mon Feb 19 16:46:09 2007
create tablespace SYSTEM datafile SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL online
Mon Feb 19 16:46:20 2007
Completed: create tablespace SYSTEM datafile SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL online
Mon Feb 19 16:46:20 2007
create rollback segment SYSTEM tablespace SYSTEM
storage (initial 50K next 50K)
Completed: create rollback segment SYSTEM tablespace SYSTEM
storage (initial 50K next 50K)
Mon Feb 19 16:46:24 2007
CREATE SMALLFILE UNDO TABLESPACE UNDOTBS1 DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Mon Feb 19 16:46:31 2007
Successfully onlined Undo Tablespace 1.
Completed: CREATE SMALLFILE UNDO TABLESPACE UNDOTBS1 DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Mon Feb 19 16:46:31 2007
create tablespace SYSAUX datafile SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO online
Completed: create tablespace SYSAUX datafile SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO online
Mon Feb 19 16:46:36 2007
CREATE SMALLFILE TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
Completed: CREATE SMALLFILE TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
Mon Feb 19 16:46:36 2007
ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP
Completed: ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP
Mon Feb 19 16:46:36 2007
ALTER DATABASE DEFAULT TABLESPACE SYSTEM
Completed: ALTER DATABASE DEFAULT TABLESPACE SYSTEM
Mon Feb 19 16:46:38 2007
SMON: enabling tx recovery
Mon Feb 19 16:46:39 2007
Threshold validation cannot be done before catproc is loaded.
Threshold validation cannot be done before catproc is loaded.
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=22, OS id=32303
Mon Feb 19 16:46:39 2007
Completed: CREATE DATABASE "bsaprod"
MAXINSTANCES 32
MAXLOGHISTORY 1
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
DATAFILE SIZE 300M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE SIZE 120M AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED
SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE SIZE 20M AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
CHARACTER SET WE8ISO8859P1
NATIONAL CHARACTER SET AL16UTF16
LOGFILE GROUP 1 SIZE 51200K,
GROUP 2 SIZE 51200K
USER SYS IDENTIFIED BY *USER SYSTEM IDENTIFIED BY
Mon Feb 19 16:46:39 2007
CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Completed: CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 200M AUTOEXTEND ON NEXT 5120K MAXSIZE UNLIMITED
Mon Feb 19 16:46:46 2007
CREATE SMALLFILE TABLESPACE "USERS" LOGGING DATAFILE SIZE 5M AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
Completed: CREATE SMALLFILE TABLESPACE "USERS" LOGGING DATAFILE SIZE 5M AUTOEXTEND ON NEXT 1280K MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
Mon Feb 19 16:46:46 2007
ALTER DATABASE DEFAULT TABLESPACE "USERS"
Completed: ALTER DATABASE DEFAULT TABLESPACE "USERS"
Mon Feb 19 16:47:05 2007
Thread 1 advanced to log sequence 2
Current log# 2 seq# 2 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:47:34 2007
Thread 1 cannot allocate new log, sequence 3
Checkpoint not complete
Current log# 2 seq# 2 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 3
Current log# 1 seq# 3 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:48:10 2007
Thread 1 cannot allocate new log, sequence 4
Checkpoint not complete
Current log# 1 seq# 3 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 4
Current log# 2 seq# 4 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:49:05 2007
Thread 1 cannot allocate new log, sequence 5
Checkpoint not complete
Current log# 2 seq# 4 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 5
Current log# 1 seq# 5 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:50:19 2007
Thread 1 cannot allocate new log, sequence 6
Checkpoint not complete
Current log# 1 seq# 5 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 6
Current log# 2 seq# 6 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:51:00 2007
Thread 1 cannot allocate new log, sequence 7
Checkpoint not complete
Current log# 2 seq# 6 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 7
Current log# 1 seq# 7 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:51:54 2007
Thread 1 cannot allocate new log, sequence 8
Checkpoint not complete
Current log# 1 seq# 7 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 8
Current log# 2 seq# 8 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:55:32 2007
Thread 1 cannot allocate new log, sequence 9
Checkpoint not complete
Current log# 2 seq# 8 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 9
Current log# 1 seq# 9 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:56:35 2007
Thread 1 cannot allocate new log, sequence 10
Checkpoint not complete
Current log# 1 seq# 9 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 10
Current log# 2 seq# 10 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:56:52 2007
Thread 1 cannot allocate new log, sequence 11
Checkpoint not complete
Current log# 2 seq# 10 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 11
Current log# 1 seq# 11 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:57:04 2007
Thread 1 cannot allocate new log, sequence 12
Checkpoint not complete
Current log# 1 seq# 11 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 12
Current log# 2 seq# 12 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:57:14 2007
Thread 1 cannot allocate new log, sequence 13
Checkpoint not complete
Current log# 2 seq# 12 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 13
Current log# 1 seq# 13 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:57:31 2007
Thread 1 cannot allocate new log, sequence 14
Checkpoint not complete
Current log# 1 seq# 13 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 14
Current log# 2 seq# 14 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 16:58:13 2007
Thread 1 cannot allocate new log, sequence 15
Checkpoint not complete
Current log# 2 seq# 14 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 15
Current log# 1 seq# 15 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 16:59:42 2007
Thread 1 cannot allocate new log, sequence 16
Checkpoint not complete
Current log# 1 seq# 15 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 16
Current log# 2 seq# 16 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:00:33 2007
Thread 1 cannot allocate new log, sequence 17
Checkpoint not complete
Current log# 2 seq# 16 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 17
Current log# 1 seq# 17 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:01:49 2007
Thread 1 cannot allocate new log, sequence 18
Checkpoint not complete
Current log# 1 seq# 17 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 18
Current log# 2 seq# 18 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:02:23 2007
Thread 1 cannot allocate new log, sequence 19
Checkpoint not complete
Current log# 2 seq# 18 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 19
Current log# 1 seq# 19 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:04:06 2007
Thread 1 cannot allocate new log, sequence 20
Checkpoint not complete
Current log# 1 seq# 19 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 20
Current log# 2 seq# 20 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:05:22 2007
Thread 1 cannot allocate new log, sequence 21
Checkpoint not complete
Current log# 2 seq# 20 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 21
Current log# 1 seq# 21 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:06:22 2007
Thread 1 cannot allocate new log, sequence 22
Checkpoint not complete
Current log# 1 seq# 21 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 22
Current log# 2 seq# 22 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:08:42 2007
Thread 1 cannot allocate new log, sequence 23
Checkpoint not complete
Current log# 2 seq# 22 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 23
Current log# 1 seq# 23 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:09:43 2007
Thread 1 cannot allocate new log, sequence 24
Checkpoint not complete
Current log# 1 seq# 23 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 24
Current log# 2 seq# 24 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:10:42 2007
Thread 1 cannot allocate new log, sequence 25
Checkpoint not complete
Current log# 2 seq# 24 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 25
Current log# 1 seq# 25 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:11:31 2007
Thread 1 cannot allocate new log, sequence 26
Checkpoint not complete
Current log# 1 seq# 25 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 26
Current log# 2 seq# 26 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:12:01 2007
Thread 1 cannot allocate new log, sequence 27
Checkpoint not complete
Current log# 2 seq# 26 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 27
Current log# 1 seq# 27 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:13:12 2007
Thread 1 cannot allocate new log, sequence 28
Checkpoint not complete
Current log# 1 seq# 27 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 28
Current log# 2 seq# 28 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:14:05 2007
Thread 1 cannot allocate new log, sequence 29
Checkpoint not complete
Current log# 2 seq# 28 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 29
Current log# 1 seq# 29 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:14:57 2007
Thread 1 cannot allocate new log, sequence 30
Checkpoint not complete
Current log# 1 seq# 29 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Thread 1 advanced to log sequence 30
Current log# 2 seq# 30 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Mon Feb 19 17:16:29 2007
Thread 1 cannot allocate new log, sequence 31
Checkpoint not complete
Current log# 2 seq# 30 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_2.258.614969167
Thread 1 advanced to log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Mon Feb 19 17:17:54 2007
Starting background process EMN0
EMN0 started with pid=25, OS id=15651
Mon Feb 19 17:17:54 2007
Shutting down instance: further logons disabled
Mon Feb 19 17:17:55 2007
Stopping background process QMNC
Mon Feb 19 17:17:55 2007
Stopping background process CJQ0
Mon Feb 19 17:17:57 2007
Stopping background process MMNL
Mon Feb 19 17:17:58 2007
Stopping background process MMON
Mon Feb 19 17:17:59 2007
Shutting down instance (immediate)
License high water mark = 1
Mon Feb 19 17:17:59 2007
Stopping Job queue slave processes
Mon Feb 19 17:17:59 2007
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Feb 19 17:18:01 2007
ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:01 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Feb 19 17:18:03 2007
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thread 1 closed at log sequence 31
Successful close of redo thread 1
Mon Feb 19 17:18:04 2007
Completed: ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:04 2007
ALTER DATABASE DISMOUNT
Mon Feb 19 17:18:04 2007
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 17:18:04 2007
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Feb 19 17:18:09 2007
freeing rdom 0
Mon Feb 19 17:18:12 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =36
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
sga_target = 1073741824
control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
db_block_size = 8192
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database_instances= 1
db_create_file_dest = +ORCL_DATA1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 17:18:13 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=16124
DIAG started with pid=3, OS id=16126
PSP0 started with pid=4, OS id=16128
LMON started with pid=5, OS id=16130
LMD0 started with pid=6, OS id=16132
MMAN started with pid=7, OS id=16134
DBW0 started with pid=8, OS id=16136
LGWR started with pid=9, OS id=16138
CKPT started with pid=10, OS id=16140
SMON started with pid=11, OS id=16142
RECO started with pid=12, OS id=16144
CJQ0 started with pid=13, OS id=16146
MMON started with pid=14, OS id=16148
Mon Feb 19 17:18:13 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=15, OS id=16150
Mon Feb 19 17:18:13 2007
starting up 1 shared server(s) ...
Mon Feb 19 17:18:13 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 17:18:14 2007
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Reconfiguration complete
Mon Feb 19 17:18:14 2007
ALTER DATABASE MOUNT
Mon Feb 19 17:18:14 2007
Starting background process ASMB
ASMB started with pid=19, OS id=16182
Starting background process RBAL
RBAL started with pid=20, OS id=16186
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 17:18:18 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 17:18:23 2007
Setting recovery target incarnation to 1
Mon Feb 19 17:18:23 2007
Successful mount of redo thread 1, with mount id 3309012054
Mon Feb 19 17:18:23 2007
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Mon Feb 19 17:18:23 2007
alter database archivelog
Completed: alter database archivelog
Mon Feb 19 17:18:23 2007
alter database open
Mon Feb 19 17:18:23 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=22, OS id=16393
Mon Feb 19 17:18:23 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=23, OS id=16395
Mon Feb 19 17:18:23 2007
Thread 1 opened at log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 17:18:23 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 17:18:23 2007
SMON: enabling cache recovery
Mon Feb 19 17:18:23 2007
ARC0: STARTING ARCH PROCESSES
Mon Feb 19 17:18:23 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Feb 19 17:18:23 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=24, OS id=16422
Mon Feb 19 17:18:23 2007
Successfully onlined Undo Tablespace 1.
Mon Feb 19 17:18:23 2007
SMON: enabling tx recovery
Mon Feb 19 17:18:23 2007
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=25, OS id=16449
Mon Feb 19 17:18:25 2007
Completed: alter database open
Mon Feb 19 17:18:25 2007
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K,
GROUP 4 SIZE 51200K
Completed: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K,
GROUP 4 SIZE 51200K
Mon Feb 19 17:18:28 2007
ALTER DATABASE ENABLE PUBLIC THREAD 2
Completed: ALTER DATABASE ENABLE PUBLIC THREAD 2
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:18:29 2007
Starting background process EMN0
EMN0 started with pid=26, OS id=16564
Mon Feb 19 17:18:29 2007
Shutting down instance: further logons disabled
Mon Feb 19 17:18:29 2007
Stopping background process QMNC
Mon Feb 19 17:18:29 2007
Stopping background process CJQ0
Mon Feb 19 17:18:30 2007
Stopping background process MMNL
Mon Feb 19 17:18:31 2007
Stopping background process MMON
Mon Feb 19 17:18:32 2007
Shutting down instance (immediate)
License high water mark = 3
Mon Feb 19 17:18:32 2007
Stopping Job queue slave processes
Mon Feb 19 17:18:32 2007
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Feb 19 17:18:34 2007
ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:34 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Feb 19 17:18:34 2007
Shutting down archive processes
Archiving is disabled
Mon Feb 19 17:18:39 2007
ARCH shutting down
ARC2: Archival stopped
Mon Feb 19 17:18:44 2007
ARCH shutting down
ARC1: Archival stopped
Mon Feb 19 17:18:49 2007
ARCH shutting down
ARC0: Archival stopped
Mon Feb 19 17:18:50 2007
Thread 1 closed at log sequence 31
Successful close of redo thread 1
Mon Feb 19 17:18:51 2007
Completed: ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:18:51 2007
ALTER DATABASE DISMOUNT
Mon Feb 19 17:18:51 2007
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 17:18:51 2007
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Feb 19 17:18:56 2007
freeing rdom 0
Mon Feb 19 17:18:59 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
spfile = +BSA_DATA1/bsaprod/spfilebsaprod.ora
sga_target = 1073741824
control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
db_block_size = 8192
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +ORCL_DATA1
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 17:19:00 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=17373
DIAG started with pid=3, OS id=17375
PSP0 started with pid=4, OS id=17377
LMON started with pid=5, OS id=17379
LMD0 started with pid=6, OS id=17381
LMS0 started with pid=7, OS id=17383
LMS1 started with pid=8, OS id=17387
MMAN started with pid=9, OS id=17391
DBW0 started with pid=10, OS id=17393
LGWR started with pid=11, OS id=17395
CKPT started with pid=12, OS id=17397
SMON started with pid=13, OS id=17399
RECO started with pid=14, OS id=17412
CJQ0 started with pid=15, OS id=17428
MMON started with pid=16, OS id=17430
Mon Feb 19 17:19:00 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=17, OS id=17432
Mon Feb 19 17:19:00 2007
starting up 1 shared server(s) ...
Mon Feb 19 17:19:01 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 17:19:01 2007
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon Feb 19 17:19:01 2007
LMS 1: 0 GCS shadows cancelled, 0 closed
Mon Feb 19 17:19:01 2007
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Mon Feb 19 17:19:01 2007
LMS 1: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:19:01 2007
LMS 0: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:19:01 2007
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=20, OS id=17460
Mon Feb 19 17:19:02 2007
ALTER DATABASE MOUNT
Mon Feb 19 17:19:02 2007
This instance was first to mount
Mon Feb 19 17:19:02 2007
Starting background process ASMB
ASMB started with pid=22, OS id=17466
Starting background process RBAL
RBAL started with pid=23, OS id=17470
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 17:19:06 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 17:19:09 2007
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:19:10 2007
Setting recovery target incarnation to 1
Mon Feb 19 17:19:10 2007
Successful mount of redo thread 1, with mount id 3308997510
Mon Feb 19 17:19:10 2007
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Mon Feb 19 17:19:11 2007
ALTER DATABASE OPEN
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Mon Feb 19 17:19:11 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=26, OS id=17679
Mon Feb 19 17:19:11 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=27, OS id=17706
Mon Feb 19 17:19:11 2007
Thread 1 opened at log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 17:19:11 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 17:19:11 2007
ARC0: STARTING ARCH PROCESSES
Mon Feb 19 17:19:11 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Feb 19 17:19:11 2007
SMON: enabling cache recovery
Mon Feb 19 17:19:11 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=28, OS id=17708
Mon Feb 19 17:19:11 2007
Successfully onlined Undo Tablespace 1.
Mon Feb 19 17:19:11 2007
SMON: enabling tx recovery
Mon Feb 19 17:19:11 2007
Database Characterset is WE8ISO8859P1
Mon Feb 19 17:19:11 2007
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Mon Feb 19 17:19:12 2007
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:19:12 2007
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=17735
Mon Feb 19 17:19:13 2007
Completed: ALTER DATABASE OPEN
Mon Feb 19 17:20:11 2007
Shutting down archive processes
Mon Feb 19 17:20:16 2007
ARCH shutting down
ARC2: Archival stopped
Mon Feb 19 17:23:35 2007
Starting background process EMN0
EMN0 started with pid=74, OS id=28350
Mon Feb 19 17:23:35 2007
Shutting down instance: further logons disabled
Mon Feb 19 17:23:36 2007
Stopping background process QMNC
Mon Feb 19 17:23:36 2007
Stopping background process CJQ0
Mon Feb 19 17:23:38 2007
Stopping background process MMNL
Mon Feb 19 17:23:39 2007
Stopping background process MMON
Mon Feb 19 17:23:40 2007
Shutting down instance (immediate)
License high water mark = 34
Mon Feb 19 17:23:40 2007
Stopping Job queue slave processes
Mon Feb 19 17:23:40 2007
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Feb 19 17:24:07 2007
ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:24:07 2007
SMON: disabling tx recovery
SMON: disabling cache recovery
Mon Feb 19 17:24:08 2007
Shutting down archive processes
Archiving is disabled
Mon Feb 19 17:24:18 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC1: Archiving disabled
ARCH shutting down
Mon Feb 19 17:24:18 2007
ARC1: Archival stopped
Mon Feb 19 17:24:18 2007
ARC0: Becoming the heartbeat ARCH
Mon Feb 19 17:24:18 2007
ARC0: Archiving disabled
ARCH shutting down
ARC0: Archival stopped
Mon Feb 19 17:24:19 2007
Thread 1 closed at log sequence 31
Successful close of redo thread 1
Mon Feb 19 17:24:19 2007
Completed: ALTER DATABASE CLOSE NORMAL
Mon Feb 19 17:24:19 2007
ALTER DATABASE DISMOUNT
Mon Feb 19 17:24:19 2007
SUCCESS: diskgroup ORCL_DATA1 was dismounted
Mon Feb 19 17:24:19 2007
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Mon Feb 19 17:24:25 2007
freeing rdom 0
Mon Feb 19 17:24:35 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.250.90.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.250.90.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.2.0.
System parameters with non-default values:
processes = 300
sessions = 335
sga_max_size = 1073741824
__shared_pool_size = 255852544
__large_pool_size = 4194304
__java_pool_size = 4194304
__streams_pool_size = 0
spfile = +BSA_DATA1/bsaprod/spfilebsaprod.ora
sga_target = 1073741824
control_files = +ORCL_DATA1/bsaprod/controlfile/current.256.614969167
db_block_size = 8192
__db_cache_size = 801112064
compatible = 10.2.0.2.0
log_archive_dest_1 = LOCATION=+BSA_DATA1/bsaprod/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +ORCL_DATA1
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=bsaprodXDB)
remote_listener = LISTENERS_BSAPROD
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/bdump
user_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/udump
core_dump_dest = /u01/app/oracle/oracle/admin/bsaprod/cdump
audit_file_dest = /u01/app/oracle/oracle/admin/bsaprod/adump
db_name = bsaprod
open_cursors = 300
pga_aggregate_target = 262144000
Cluster communication is configured to use the following interface(s) for this instance
10.250.90.107
Mon Feb 19 17:24:35 2007
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=29950
DIAG started with pid=3, OS id=29952
PSP0 started with pid=4, OS id=29954
LMON started with pid=5, OS id=29956
LMD0 started with pid=6, OS id=29958
LMS0 started with pid=7, OS id=29960
LMS1 started with pid=8, OS id=29964
MMAN started with pid=9, OS id=29968
DBW0 started with pid=10, OS id=29970
LGWR started with pid=11, OS id=29972
CKPT started with pid=12, OS id=29974
SMON started with pid=13, OS id=29976
RECO started with pid=14, OS id=29978
CJQ0 started with pid=15, OS id=29980
MMON started with pid=16, OS id=29982
Mon Feb 19 17:24:36 2007
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=17, OS id=29984
Mon Feb 19 17:24:36 2007
starting up 1 shared server(s) ...
Mon Feb 19 17:24:36 2007
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Feb 19 17:24:39 2007
Reconfiguration started (old inc 0, new inc 4)
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 not valid according to instance 1
* domain 0 valid = 0 according to instance 1
Mon Feb 19 17:24:39 2007
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon Feb 19 17:24:39 2007
LMS 1: 0 GCS shadows cancelled, 0 closed
Mon Feb 19 17:24:39 2007
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Mon Feb 19 17:24:39 2007
LMS 0: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:24:39 2007
LMS 1: 0 GCS shadows traversed, 0 replayed
Mon Feb 19 17:24:39 2007
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=20, OS id=30069
Mon Feb 19 17:24:40 2007
ALTER DATABASE MOUNT
Mon Feb 19 17:24:41 2007
Starting background process ASMB
ASMB started with pid=22, OS id=30121
Starting background process RBAL
RBAL started with pid=23, OS id=30125
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Mon Feb 19 17:24:45 2007
SUCCESS: diskgroup ORCL_DATA1 was mounted
Mon Feb 19 17:24:49 2007
Setting recovery target incarnation to 1
Mon Feb 19 17:24:49 2007
Successful mount of redo thread 1, with mount id 3309017062
Mon Feb 19 17:24:49 2007
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Mon Feb 19 17:24:51 2007
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
SUCCESS: diskgroup BSA_DATA1 was mounted
SUCCESS: diskgroup BSA_DATA1 was dismounted
Mon Feb 19 17:24:52 2007
Completed: ALTER DATABASE MOUNT
Mon Feb 19 17:24:52 2007
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Mon Feb 19 17:24:52 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=25, OS id=30406
Mon Feb 19 17:24:53 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=26, OS id=30422
Mon Feb 19 17:24:53 2007
Thread 1 opened at log sequence 31
Current log# 1 seq# 31 mem# 0: +ORCL_DATA1/bsaprod/onlinelog/group_1.257.614969167
Successful open of redo thread 1
Mon Feb 19 17:24:53 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Feb 19 17:24:53 2007
ARC0: STARTING ARCH PROCESSES
Mon Feb 19 17:24:53 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Feb 19 17:24:53 2007
SMON: enabling cache recovery
Mon Feb 19 17:24:53 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=27, OS id=30436
Mon Feb 19 17:25:07 2007
Successfully onlined Undo Tablespace 1.
Mon Feb 19 17:25:07 2007
SMON: enabling tx recovery
Mon Feb 19 17:25:07 2007
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=30882
Mon Feb 19 17:26:31 2007
Completed: ALTER DATABASE OPEN
Mon Feb 19 17:26:38 2007
ALTER SYSTEM SET service_names='bsaprod','bsatest' SCOPE=MEMORY SID='bsaprod1';
Mon Feb 19 17:26:53 2007
Shutting down archive processes
Mon Feb 19 17:26:58 2007
ARCH shutting down
ARC2: Archival stopped
Mon Feb 19 17:32:16 2007
Error: KGXGN aborts the instance (6)
Mon Feb 19 17:32:16 2007
Errors in file /u01/app/oracle/oracle/admin/bsaprod/bdump/bsaprod1_lmon_29956.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Mon Feb 19 17:32:17 2007
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/oracle/admin/bsaprod/bdump/bsaprod1_diag_29952.trc
Mon Feb 19 17:32:18 2007
Trace dumping is performing id=[cdmp_20070219173217]
Mon Feb 19 17:32:21 2007
Instance terminated by LMON, pid = 29956
and also the output of crs_stat -t in first node is....
Name Type Target State Host
ora....d1.inst application ONLINE OFFLINE
ora....d2.inst application ONLINE OFFLINE
ora....od1.srv application ONLINE OFFLINE
ora....od2.srv application ONLINE OFFLINE
ora....test.cs application ONLINE OFFLINE
ora.bsaprod.db application ONLINE OFFLINE
ora....SM1.asm application ONLINE UNKNOWN dsrvbd003
ora....ER.lsnr application ONLINE UNKNOWN dsrvbd003
ora....03.lsnr application ONLINE UNKNOWN dsrvbd003
ora....003.gsd application ONLINE UNKNOWN dsrvbd003
ora....003.ons application ONLINE UNKNOWN dsrvbd003
ora....003.vip application ONLINE ONLINE dsrvbd003
ora....SM2.asm application ONLINE OFFLINE
ora....04.lsnr application ONLINE OFFLINE
ora....004.gsd application ONLINE OFFLINE
ora....004.ons application ONLINE OFFLINE
ora....004.vip application ONLINE OFFLINE

Error while upgrade oracle database from 9.2.0.6 to 10.2.0.1

updating the correct ORACLE_HOME and PATH variables
and following the Oracle10 R2 upgrade guide,
oracle.xxxxxx: <10.2.0.1/dbs> sqlplus "/ as sysdba"
SQL*Plus: Release 10.2.0.1.0 - Production on Wed Nov 8 10:53:39 2006
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup upgrade pfile=initCB6ES4.ora
ORA-29702: error occurred in Cluster Group Service operation
SQL>

Verify if the process LMON is active.

In cluster or not ?

Hi,
on 10g R2 on HP Unix :
I'm begining to administer a new database
in alertlog I have :
ORA-29702: error occurred in Cluster Group Service operationBut when :
srvctl status service -d MYDB
PRKR-1001 : cluster database MYDB does not exist
PRKO-2005 : Application error: Failure in getting Cluster Database Configuration for: MYDBAre we in cluster or not ? Any other way to verify ?
Thank you.

ORA-29702: error occurred in Cluster Group Service operation
# problem
SQL> startup nomount;
ORA-29702: error occurred in Cluster Group Service operation
29702, 00000, "error occurred in Cluster Group Service operation"
// *Cause: An unexpected error occurred while performing a CGS operation.
// *Action: Verify that the LMON process is still active. Also,
// check the Oracle LMON trace files for errors.
# cause - cloned a RAC ORACLE_HOME to a server with no CRS
# solution
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk rac_off
make -f ins_rdbms.mk ioracle
source:-http://blog.contractoracle.com/2009/08/ora-29702-error-occurred-in-cluster.html

Can't start ASM,help me

Help I can't start ASM on rhel4(32bit) + oracle 10G 10.2 RAC
Wed Jun 28 17:10:40 2006
Error: KGXGN polling error (15)
Wed Jun 28 17:10:40 2006
Errors in file /oracle/product/10.2.0/db/admin/+ASM/bdump/+asm1_lmon_24375.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Wed Jun 28 17:10:40 2006
Dump system state for local instance only
System State dumped to trace file /oracle/product/10.2.0/db/admin/+ASM/bdump/+asm1_diag_24363.trc
Wed Jun 28 17:10:40 2006
Trace dumping is performing id=[cdmp_20060628171040]
Wed Jun 28 17:10:41 2006
Shutting down instance (abort)
License high water mark = 2
Wed Jun 28 17:10:45 2006
Instance terminated by LMON, pid = 24375
Wed Jun 28 17:10:46 2006
Instance terminated by USER, pid = 29540
Wed Jun 28 17:10:47 2006
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
WARNING: No cluster interconnect has been specified. Depending on
the communication driver configured Oracle cluster traffic
may be directed to the public interface of this machine.
Oracle recommends that RAC clustered databases be configured
with a private interconnect for enhanced security and
performance.
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /oracle/product/10.2.0/db/dbs/arch
Autotune of undo retention is turned off.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
large_pool_size = 12582912
spfile = /dev/raw/raw3
instance_type = asm
cluster_database = TRUE
instance_number = 1
remote_login_passwordfile= EXCLUSIVE
background_dump_dest = /oracle/product/10.2.0/db/admin/+ASM/bdump
user_dump_dest = /oracle/product/10.2.0/db/admin/+ASM/udump
core_dump_dest = /oracle/product/10.2.0/db/admin/+ASM/cdump
asm_diskgroups =
Cluster communication is configured to use the following interface(s) for this instance
203.144.222.81
Wed Jun 28 17:10:48 2006
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=29735
DIAG started with pid=3, OS id=29737
PSP0 started with pid=4, OS id=29739
LMON started with pid=5, OS id=29741
LMD0 started with pid=6, OS id=29743
LMS0 started with pid=7, OS id=29745
MMAN started with pid=8, OS id=29755
DBW0 started with pid=9, OS id=29757
LGWR started with pid=10, OS id=29759
CKPT started with pid=11, OS id=29761
SMON started with pid=12, OS id=29763
RBAL started with pid=13, OS id=29765
GMON started with pid=14, OS id=29767
Wed Jun 28 17:10:48 2006
lmon registered with NM - instance id 1 (internal mem no 0)
Wed Jun 28 17:12:59 2006
Error: KGXGN polling error (15)
Wed Jun 28 17:12:59 2006
Errors in file /oracle/product/10.2.0/db/admin/+ASM/bdump/+asm1_lmon_29741.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Wed Jun 28 17:13:01 2006
Shutting down instance (abort)
License high water mark = 1
Wed Jun 28 17:13:03 2006
Instance terminated by LMON, pid = 29741
and trace file
/oracle/product/10.2.0/db/admin/+ASM/bdump/+asm1_lmon_29741.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /oracle/product/10.2.0/db
System name: Linux
Node name: aadb01
Release: 2.6.9-34.ELsmp
Version: #1 SMP Fri Feb 24 16:54:53 EST 2006
Machine: i686
Instance name: +ASM1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 5
Unix process pid: 29741, image: oracle@aadb01 (LMON)
*** SERVICE NAME:() 2006-06-28 17:10:48.174
*** SESSION ID:(46.1) 2006-06-28 17:10:48.174
GES resources 1890 pool 4
GES enqueues 2510
GES IPC: Receivers 2 Senders 2
GES IPC: Buffers Receive 1000 Send (i:830 b:830) Reserve 81
GES IPC: Msg Size Regular 388 Batch 4096
Batching factor: enqueue replay 99, ack 110
Batching factor: cache replay 71 size per lock 56
kjxggin: receive buffer size = 32768
*** 2006-06-28 17:10:48.509
kjxgmrcfg: Reconfiguration started, reason 1
kjxgmcs: Setting state to 0 0.
*** 2006-06-28 17:10:48.509
Name Service frozen
kjxgmcs: Setting state to 0 1.
kjfcpiora: published my fusion master weight 5622
kjfcpiora: publish my flogb 9
*** 2006-06-28 17:11:53.883
=====================================================
kjxgmpoll: CGS state (0 1) start 0x635d41bb cur 0x672b6877 diff 0x3ce26bc
=====================================================
Group name: +ASM
Member id: 0
Cached KGXGN event: 0
Group State:
State: 0 1
Reconfig started start-tm 0x635d4173 tmout period 0x7270e00
Commited Map: 0 1
New Map: 0 1
KGXGN Map: 0 1
Master node: 0
Memcnt 0 Rcvcnt 0
Substate Proposal: false
Inc Proposal:
incarn 1 memcnt 2 master 0
proposal true matched false
map: 0 1
Master Inc State:
incarn 1 memcnt 2 agrees 1 flag 0xa
wmap: 1
nmap: 0 1
ubmap:
Substate Handler Execution State
substate 0 status done
substate 1 status done
substate 2 status done
substate 3 status done
substate 4 status done
substate 5 status done
substate 6 status done
*** 2006-06-28 17:12:59.243
=====================================================
kjxgmpoll: CGS state (0 1) start 0x635d41bb cur 0x6af9582a diff 0x79c166f
=====================================================
Group name: +ASM
Member id: 0
Cached KGXGN event: 0
Group State:
State: 0 1
Reconfig started start-tm 0x635d4173 tmout period 0x7270e00
Commited Map: 0 1
New Map: 0 1
KGXGN Map: 0 1
Master node: 0
Memcnt 0 Rcvcnt 0
Substate Proposal: false
Inc Proposal:
incarn 1 memcnt 2 master 0
proposal true matched false
map: 0 1
Master Inc State:
incarn 1 memcnt 2 agrees 1 flag 0xa
wmap: 1
nmap: 0 1
ubmap:
Substate Handler Execution State
substate 0 status done
substate 1 status done
substate 2 status done
substate 3 status done
substate 4 status done
substate 5 status done
substate 6 status done
kjxgmpoll: terminate the CGS reconfig.
Error: KGXGN polling error (15)
error 29702 detected in background process
ORA-29702: error occurred in Cluster Group Service operation
ksuitm: waiting for [5] seconds before killing DIAG

Modify the classpath as follows.
set JAVA_HOME=C:\JDK1.3.1
set ANT_HOME=C:\jakarta-tomcat
set TOMCAT_HOME=C:\jakarta-tomcat
set CLASSPATH=%CLASSPATH%;C:\jdk1.3.1\lib\tools.jar;JAVA_HOME\lib;
Modify the Path as follows
set PATH = %PATH%;%JAVA_HOME%\bin;%TOMCAT_HOME%\bin

Install ASM on Solaris RAC 10g

Hello,
I installed CRS and database software on two nodes RAC 10.2.0.4 Solaris x86-64 5.10, latest updates for Solaris. I have no error with crs.
Problems description:
1) When I run DBCA to create ASM it fails to create it on the nodes, with error ORA-03135: connection lost contact.
2) I see ORA-29702 into the logs (error occurred in Cluster Group Service operation
Cause: An unexpected error occurred while performing a CGS operation.
Action: Verify that the LMON process is still active. Also, check the Oracle LMON trace files for errors.)
Question:
Do you think that the problem is that interface 10.0.0.21 node1-priv-fail2 is started? (see bellow bdump/alert_+ASM1.log).
This interface 10.0.0.21 node1-priv-fail2 is the one that frozens the prompt when I try ssh oracle@node1-priv-fail2 from node2?
Possible solution: I saw Metalink 283684.1 but don't know if/what to change in my interfaces.
Details:
I think is something with the interfaces, but I don't know what.
- One thing I noticed is that is not possible to ssh from node1 to node2-priv-fail2 (this I wad told is the private standby loopback insterface). The same is from node2 to node1-priv-fail2, it gives a frozen prompt.
- in /etc/hosts on both nodes I have:
+127.0.0.1 localhost+
+172.17.1.17 node1+
+172.17.1.18 node1-fail1+
+172.17.1.19 node1-fail2+
+172.17.1.20 node1-vip+
+172.17.1.29 node2 loghost+
+172.17.1.30 node2-fail1+
+172.17.1.31 node2-fail2+
+172.17.1.32 node2-vip+
+10.0.0.1 node1-priv+
+10.0.0.11 node1-priv-fail1+
+10.0.0.21 node1-priv-fail2+
+10.0.0.2 node2-priv+
+10.0.0.12 node2-priv-fail1+
+10.0.0.22 node2-priv-fail2+
Do you think that the problem is that interface 10.0.0.21 node1-priv-fail2 is started (see bdump/alert_+ASM1.log).
This interface 10.0.0.21 node1-priv-fail2 is the one that frozens the prompt when I try ssh oracle@node1-priv-fail2 from node2?
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 e1000g2 10.0.0.0 configured from OCR for use as a cluster interconnect
Interface type 1 e1000g3 10.0.0.0 configured from OCR for use as a cluster interconnect
Interface type 1 e1000g0 172.17.0.0 configured from OCR for use as a public interface
Interface type 1 e1000g1 172.17.0.0 configured from OCR for use as a public interface
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
large_pool_size = 12582912
instance_type = asm
cluster_database = TRUE
instance_number = 1
remote_login_passwordfile= EXCLUSIVE
++background_dump_dest = /opt/app/oracle/db/admin/+ASM/bdump++
++user_dump_dest = /opt/app/oracle/db/admin/+ASM/udump++
++core_dump_dest = /opt/app/oracle/db/admin/+ASM/cdump++
Cluster communication is configured to use the following interface(s) for this instance
+10.0.0.1+
+10.0.0.21+
node1:oracle$ oifcfg getif
e1000g0 172.17.0.0 global public
e1000g1 172.17.0.0 global public
e1000g2 10.0.0.0 global cluster_interconnect
e1000g3 10.0.0.0 global cluster_interconnect
node1:oracle$ ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 172.17.1.17 netmask ffff0000 broadcast 172.17.255.255
groupname orapub
e1000g0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 172.17.1.18 netmask ffff0000 broadcast 172.17.255.255
e1000g0:2: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 2
inet 172.17.1.20 netmask ffff0000 broadcast 172.17.255.255
e1000g1: flags=39040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED,STANDBY> mtu 1500 index 3
inet 172.17.1.19 netmask ffff0000 broadcast 172.17.255.255
groupname orapub
e1000g2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 10.0.0.1 netmask ff000000 broadcast 10.255.255.255
groupname oracle_interconnect
e1000g2:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 10.0.0.11 netmask ff000000 broadcast 10.255.255.255
e1000g3: flags=39040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED,STANDBY> mtu 1500 index 5
inet 10.0.0.21 netmask ff000000 broadcast 10.255.255.255
groupname oracle_interconnect

Hi,
for a 10g RAC you need:
- a host-IP for every node
- a private IP for every node
- a virtual IP for every node
Host-IP and Private IP must be assigned to both hosts and conenction between the hosts using either the host-IP or the private IP must be possible.
Is is possible to build the RAC and ASM only with the private IP without the public and the virtual IP, if yes how?The term "private" and "public" does not refer to public IPs. It refers to the fact "private" is only for communication between the nodes and public for communication between the client and database.
For a successful installation you need at least these three IPs on each system.
So for instance your public IPs reside in the network 192.168.1.0/255.255.255.0 and your private interconnect network can be 192.168.2.0/255.255.255.0. Both networks consists of private (i.e. non-routeable) IPs.

When one node reboot other node in RAC

Hi Friends,
I faced one situation where one node of RAC cluster had been rebooted by other node. This happen due to network interconnect link fluctuation.
Sep 13 16:23:48 kkvs1a su: [ID 810491 auth.crit] 'su admin' failed for wipro1 on /dev/pts/3
Sep 14 00:22:17 kkvs1a ixgbe: [ID 611667 kern.info] NOTICE: ixgbe3: link down
Sep 14 00:22:21 kkvs1a ixgbe: [ID 611667 kern.info] NOTICE: ixgbe3: link up, , full duplex
Sep 14 00:22:31 kkvs1a ixgbe: [ID 611667 kern.info] NOTICE: ixgbe1: link down
Sep 14 00:22:31 kkvs1a ixgbe: [ID 611667 kern.info] NOTICE: ixgbe3: link down
/opt/oracle/product/10.2.0/crs/log/node1/alertkk1a.log
==============================================
2013-09-14 00:22:05.180
[cssd(12561)]CRS-1612:node kk1b (2) at 50% heartbeat fatal, eviction in 14.251 seconds
2013-09-14 00:22:12.180
[cssd(12561)]CRS-1611:node kk1b (2) at 75% heartbeat fatal, eviction in 7.251 seconds
2013-09-14 00:22:13.180
[cssd(12561)]CRS-1611:node kk1b (2) at 75% heartbeat fatal, eviction in 6.251 seconds
2013-09-14 00:22:17.179
[cssd(12561)]CRS-1610:node kk1b (2) at 90% heartbeat fatal, eviction in 2.251 seconds
2013-09-14 00:22:18.180
[cssd(12561)]CRS-1610:node kkvs1b (2) at 90% heartbeat fatal, eviction in 1.251 seconds
This clearly shows CSSD of node kkvs1a has given node eviction message to kkvs1b node.
I got following messages on the instance which got rebooted:
ASM alert log:
Sat Sep 14 00:22:25 IST 2013
Error: KGXGN aborts the instance (6)
Sat Sep 14 00:22:25 IST 2013
Errors in file /opt/oracle/admin/+ASM/bdump/+asm2_lmon_8527.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
A network fluctuation shouldn't give reboot like this. Then why oracle design like this way? Is this a bug? My oracle version is: 10.2.0.5.0
Could you tell me the other possible situations when 1 RC instance reboots other RAC instacne.

What you are describing is the expected behaviour: if your interconnect fails, you will have a node eviction. Releases < 11.2.0.2 evict a node by reboot, which can fix the problem: the NIC may come up correctly when the machine re-starts. Releases >= 11.2.0.2 can often evict without a re-boot. But either way, if your interconnect goes down, a node must be evicted to prevent uncoordinated disc writes.
If you are interested, you can find some discussion and demos of this in a series of webcasts I've recorded,
Free Oracle Database Tutorials for Administration and Developers
If you really don't like this behaviour and the problems are transient, you can try 'raising the CSS MISSCOUNT parameter.
John Watson
Oracle Certified Master DBA

10.1.2.3 failed upgrade

db upgrade from 10.1.2.1 to 10.1.2.3 (2 node RAC)failed with following errors in alert log
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
CRS upgrade was sucessfull
any one faced the same issue ?

db upgrade from 10.1.2.1 to 10.1.2.3 (2 node RAC)failed with following errors in alert log
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
CRS upgrade was sucessfull
any one faced the same issue ?

Starting a RAC db

DB version:10gR1
We have two instances (say mysid1 and mysid2) running in our RAC database.Due to a power failure we have to restart this database.But i got the following error when i tried to restart the instance
*$ sqlplus / as sysdba*
SQLPlus: Release 10.1.0.4.0 - Production on Tue Sep 2 05:37:56 2008*
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-29702: error occurred in Cluster Group Service operation
SQL>
After a bit of googling(peace be upon google), i came to know about srvctl. So, i am going to use that. But should i start some Cluster services , before i start using srvctl. I am new to RAC.

You need to use srvctl to start and stop your RAC instances.
for example,
srvctl start instance -d MYSID -i MYSID1
of course check CRS status first.
http://www.orafaq.com/node/70

Oracle Cluster and libnnz10 problem - SOLVED!

Hi to all
I have some machines (XServe) with OSX 10.4.9 and Oracle 10.1.0.3.
In all the machines I follow the instructions find in this forum (the relink libnnz10.dylib trick) and all the databases runs very well.
Now I'm trying to instal two machines in cluster mode.
The node 1 is a clean machine. The node 2 is a machine that has a Oracle single instance running.
All the installation in cluster mode runs smooth and I made the same things that in a single instance (like the relink trick).
With all installed, the CRS daemon is running and the listeners is running (I run NETCA before runnig DBCA).
Using the DBCA works fine until the last step - for the database creation, the last script has a shutdown/startup command to run a full recompile of all packages and procedures in the database.
At this moment the installation fails (with end-of-communication channel). Looking in the logs, this is the _dyld libnnz10.dylib error.
Even after the crash in installation, I quit the DBCA and runs SQLPLUS. I connect in the database, try a startup (that fails with the error mentioned), reconnect again and make two commands : alter database mount; alter database open; (and it works).
I do this in the two nodes and the CRS daemon recognizes the running instances (with crs_stat -t I can see all online) and the database is running and in cluster (what I do in one instance appears in the other and the datafiles are the same), including failover tasks (like simulating a node down - the database still running).
But all tries to run the startup/shutdown commands fails. It fails always when the LMON processes try to start.
I recheck the installation, focusing in the relink trick with the libnnz10. When I use DBCA to install a single instance on node 1 or node 2 in the same ORACLE_HOME of the cluster, it runs fine (including startup/shutdown) - this make me think that all the database relink's are fine.
I try to relink the cluster. In installation it uses 4 makes and I redo this makes without the libnnz10. The makes runs well too, but the error continues (only using startup/shutdown in cluster database).
I'm doing the documentation of the process and will post it, but I want to resolve this issue.
Any ideas ?
Thanks to all
Message was edited by:
alliance01

Hi to all.
I downloaded the 10.1.0.5 as soon I can - thanks to damjanp!
After the download, I follow all the instructions.
I stop the database, all the services and the cluster. Run the installer and it works fine.
When I try to make the post-installation tasks, I cannot startup the database (nor with the error that I was having). I received a Oracle Cluster Configuration Error...
I read all the logs and find 3 errors. I do not know if this occurs because of wrong environment variables.
The commands awk, touch and basename fails - the scripts try to find this commands in /bin (this commands in OSX stay in /usr/bin). I prefer make a link (ln -s) using root :
ln -s /usr/bin/awk /bin/awk
ln -s /usr/bin/touch /bin/touch
ln -s /usr/bin/basename /bin/basename
When I done this, the cluster starts (ocssd.bin, crsd.bin e evmd.bin). But when I try to start the database, it fails (ORA-29702 : error occurred in Cluster Group Service operation) and srvctl gives me PRKH-1010 : Unable to communicate with CRS services. [OCR Error(Native : prsr_initCLSS:[21])].
To resolve this I use localconfig (user root) :
localconfig reset /<ORACLE_HOME>
After this, I shutdown all and restart the CRS services - and for my surprise the CRS automaticaly starts all the services (including the instances).
And the libnnz10 problema was gone!
I read the documentation and found that the "TNS - lost contact" problem in Tiger (yeah - Tiger) was resolved.
I think that 10.1.0.5 can runs on Tiger more easily.
The cluster are running fine now - I shutdown/startup the instance several times to test with no errors.
I will prepare a documentation to post in the Web about this installation. When I finish I post in this forum the link.
Thank's to all.

ORA-29702

Similar Messages

Maybe you are looking for