Getting ORA-32700 while creating database

Question

I am getting following error when creating the database :
ORA-32700 error occurred in DIAG Group Service
The error description is as follows :
Cause: An unexpected error occurred while performing a DIAG Group Service operation.
Action: Verify that the DIAG process is still active. Also, check the Oracle DIAG trace files for errors.
The setup is as follows :
Database : Oracle 9i Rel 2.
OS : IBM AIX 4.3.3
We are trying to create the database from sqlplus/dbca. It seems that in the database creation process it is throwing the error while it is trying to create the control files.
listener.ora :
LSNR_TSEDMDB2 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = prcsmbk1)(PORT = 1522))
TNSNAMES.ORA
LISTENER_TSEDMDB2 =
(ADDRESS = (PROTOCOL = TCP)(HOST = prcsmbk1)(PORT = 1522))
We have IBM HACMP Cluster setup on 3 machines. This machine (prcsmbk1) on which I am getting the error while creating the database is part of that HACMP cluster. While installing the Oracle database software also it gave me a list of the 3 machines which are in HACMP cluster. We are not planning to setup RAC, but still it seems like while creating the database control files Oracle is trying to perform some group service operation.
Alert log :
Sat Jan 3 03:31:18 2004
Starting ORACLE instance (normal)
Sat Jan 3 03:31:18 2004
Global Enqueue Service Resources = 64, pool = 4
Sat Jan 3 03:31:18 2004
Global Enqueue Service Enqueues = 128
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
GES IPC: Receivers 1 Senders 1
GES IPC: Buffers Receive 1000 Send 530 Reserve 300
GES IPC: Msg Size Regular 432 Batch 2048
SCN scheme 3
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.4.0.
System parameters with non-default values:
processes = 150
timed_statistics = TRUE
shared_pool_size = 117440512
large_pool_size = 16777216
java_pool_size = 117440512
control_files = /u01/app/oracle/oradata/TSEDMDB2/control01.ctl
db_block_size = 8192
db_cache_size = 33554432
compatible = 9.2.0.0.0
db_file_multiblock_read_count= 16
fast_start_mttr_target = 300
undo_management = AUTO
undo_tablespace = UNDOTBS1
undo_retention = 10800
remote_login_passwordfile= EXCLUSIVE
db_domain =
instance_name = TSEDMDB2
dispatchers = (PROTOCOL=TCP) (SERVICE=TSEDMDB2XDB)
local_listener = LISTENER_TSEDMDB2
job_queue_processes = 10
hash_join_enabled = TRUE
background_dump_dest = /u01/app/oracle/admin/TSEDMDB2/bdump
user_dump_dest = /u01/app/oracle/admin/TSEDMDB2/udump
core_dump_dest = /u01/app/oracle/admin/TSEDMDB2/cdump
sort_area_size = 524288
db_name = TSEDMDB2
open_cursors = 300
star_transformation_enabled= FALSE
query_rewrite_enabled = FALSE
pga_aggregate_target = 25165824
aq_tm_processes = 1
Sat Jan 3 03:31:19 2004
clUster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2 Version 1.0
PMON started with pid=2
DIAG started with pid=3
LMON started with pid=4
LMD0 started with pid=5
DBW0 started with pid=6
LGWR started with pid=7
CKPT started with pid=8
SMON started with pid=9
RECO started with pid=10
CJQ0 started with pid=11
QMN0 started with pid=12
Sat Jan 3 03:31:24 2004
starting up 1 shared server(s) ...
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Sat Jan 3 03:31:25 2004
Create controlfile reuse set database TSEDMDB2
MAXINSTANCES 1
MAXLOGHISTORY 1
MAXLOGFILES 50
MAXLOGMEMBERS 5
MAXDATAFILES 100
Datafile
'/u01/app/oracle/oradata/TSEDMDB2/cwmlite01.dbf' ,
'/u01/app/oracle/oradata/TSEDMDB2/drsys01.dbf' ,
'/u01/app/oracle/oradata/TSEDMDB2/example01.dbf' ,
'/u01/app/oracle/oradata/TSEDMDB2/indx01.dbf' ,
'/u01/app/oracle/oradata/TSEDMDB2/odm01.dbf' ,
'/u01/app/oracle/oradata/TSEDMDB2/system01.dbf' ,
'/u01/app/oracle/oradata/TSEDMDB2/tools01.dbf' ,
'/u01/app/oracle/oradata/TSEDMDB2/undotbs01.dbf' ,
'/u01/app/oracle/oradata/TSEDMDB2/users01.dbf' ,
'/u01/app/oracle/oradata/TSEDMDB2/xdb01.dbf'
LOGFILE GROUP 1 ('/u01/app/oracle/oradata/TSEDMDB2/redo01.log') SIZE 102400K,
GROUP 2 ('/u01/app/oracle/oradata/TSEDMDB2/redo02.log') SIZE 102400K,
GROUP 3 ('/u01/app/oracle/oradata/TSEDMDB2/redo03.log') SIZE 102400K RESETLOGS
Sat Jan 3 03:31:25 2004
ORA-1503 signalled during: Create controlfile reuse set database TSEDMDB2
MAX...
Sat Jan 3 03:32:28 2004
Restarting dead background process QMN0
QMN0 started with pid=15
Sat Jan 3 03:32:28 2004
Shutting down instance (abort)
License high water mark = 3
Instance terminated by USER, pid = 47128
Diag trace file :
/u01/app/oracle/admin/TSEDMDB2/bdump/tsedmdb2_diag_49754.trc
Oracle9i Enterprise Edition Release 9.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.4.0 - Production
ORACLE_HOME = /u01/app/oracle/product/9.2.0
System name: AIX
Node name: prcsmbk1
Release: 3
Version: 4
Machine: 0002599F4C00
Instance name: TSEDMDB2
Redo thread mounted by this instance: 0 <none>
Oracle process number: 0
49754
Number of Serviceable Networks detected is less than 2
Disabling Transparent Network Failover Feature.
*** SESSION ID:(2.1) 2004-01-03 03:31:19.886
kjzcprt:rcv port created
kjzmreg2: slos err[1 19 [HA_GS_CONNECT_FAILED] sskgxn_gs_in 19]
[kjzmreg2]: Error [category=1] is encountered
Node monitor problem:[32700][kjzmreg2]
Membership is lost in DIAG group
Any help is appreciated. Thanks in advance.

Answer 1

Thanks a lot for all your responses. We found the issue. The issue was that when we installed 9i we selected Enterprise option which installed RAC also. And after that when we were trying to create database Oracle was thinking that we are trying to create database in RAC env. and most likely it was trying to create/sync the control files on all the machines in the cluster.
The sequence of events were as below :
1. We have 8i installed on this machine
2. We installed 9i (9.2.0.1.0) using the same ORACLE_BASE but different ORACLE_HOME. We didn't select "Custom Install". We selected "Enterprise install" option
3. We installed new installer (2.2.0.18.0 ) and then 9i patch (9.2.0.4.0)
4. While creating the database (manually & through DBCA) we were getting ORA-32700
5. We tried to uninstall (using OUI 2.2.0.18.0) 9i (including the patch) completely but it never succeeded. We noticed that when we started uninstall it did show progress for about 20 minutes and after that it didn't show any progress. It was not taking any CPU cycles and It was not writing to the log file either. Basically we were never able to cleanly uninstall 9i. We tried almost 2 to 3 times.
6. Then we installed 9i using a new ORACLE_BASE and this time we selected "Custom Install" and we did not install RAC. This is successful
7. Then we installed new installer (2.2.0.18.0 ) and we tried to install the 9i patch (9.2.0.4.0) using this new installer but it never succeeded. Then we tried to install the 9i patch (9.2.0.4.0) using the old installer version (2.2.0.11.0) and it was also successful.
8. Then we tried to create database using DBCA and it was successful and then we dropped the database which was created using DBCA and then we created it manually through SQLPLUS and this was fine too.
So basically what we learned from this exercise is that if we are installing 9i on a machine which is part of a cluster and if we don't plan to setup RAC then we shouldn't install RAC and we should do a custom install. I wish there was a option in create database stmt to tell Oracle that though we are creating database on a machine which is part of a cluster environment but we don't intend to setup RAC.
And other thing which is still a mystery is that why does the uninstall never completes successfully. One question on this : We were using the installer version 2.2.0.18.0 to uninstall the 9i software. Do we need to use the older version (2.2.0.11.0) of the OUI?