Primary RAC database hung on sqlplus as sysdba, dgmgrl and rman target
I need urgent help here, our qa environment is two nodes RAC primary, and two nodes standby.
Standby environment is ok, so far works, but no entries added to alert log.
however on primary rac, i cannot sqlplus to the db, and rman target hung, and dgmgrl hung.
I tried to srvctl stop database -o abort, it just hung there.
please help me to get that resolved.
Thanks in advance.
982335 wrote:
I need urgent help here, our qa environment is two nodes RAC primary, and two nodes standby.
Standby environment is ok, so far works, but no entries added to alert log.
however on primary rac, i cannot sqlplus to the db, and rman target hung, and dgmgrl hung.
I tried to srvctl stop database -o abort, it just hung there.
please help me to get that resolved.
Thanks in advance.check the primary alert log for error and paste here.
also do network connectivity test....e.g
lsnrctl status
from a client system do:
ping primary_ip
tnsping primary_tns
always make sure Abort and killing OS process is your last option!
Tobi
Similar Messages
-
Sqlplus as sysdba takes 3-4 mins
Hi,
I am trying to login to the database by using sqlplus as sysdba but it takes 3 to 4 mins to get the sql prompt. But for other users it does not take that much time.
Could any one suggest me on this?
Thanks
Venkatuser12052260 wrote:
My problem has solved. Thanks.May be u should tell what exactly solved your problem, so that other users can also benefit from what you did. Just a suggestion. -
Sqlplus / as sysdba hung state
Hi All,
When we try to do a sqlplus / as sysdba it gets into hung state.
sapkxqap03:oraqs9 2> sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Wed Feb 16 19:06:28 2011
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
We dont get anything after this prompt.Is there a way we can get to know as to what might be the issue.Thanks.Hi Michael,
Nice to hear from you Michael.Long time.Thanks for the response
Checked no archiver struck since the file systems have enough space.The problem is that alter log has not been updated for the past 1 week.R3trans also gets struck since it also uses sqlplus to login.Thanks.,
4 ETW000 R3trans version 6.05 (release 46D - 04.12.08 - 14:43:00).
4 ETW000 ===============================================
4 ETW000
4 ETW000 control file: <no ctrlfile>
4 ETW000 R3trans was called as follows: R3trans -d
4 ETW000 date&time : 16.02.2011 - 21:11:13
4 ETW000 trace at level 2 opened for a given file pointer
4 ETW000 [developertra,00000] Wed Feb 16 21:11:13 2011
232 0.000232
4 ETW000 [developertra,00000] db_con_init called
14 0.000246
4 ETW000 [developertra,00000] create_con (con_name=R/3)
34 0.000280
4 ETW000 [developertra,00000] Loading DB library '/usr/sap/QS9/SYS/exe/run/dbo
raslib.so' ...
4 ETW000
30 0.000310
4 ETW000 [developertra,00000] load shared library (/usr/sap/QS9/SYS/exe/run/db
oraslib.so), hdl 0
4 ETW000
15933 0.016243
4 ETW000 [developertra,00000] Library '/usr/sap/QS9/SYS/exe/run/dboraslib.so'
sapkxqap03:qs9adm 3> more trans.log
4 ETW000 R3trans version 6.05 (release 46D - 04.12.08 - 14:43:00).
4 ETW000 ===============================================
4 ETW000
4 ETW000 control file: <no ctrlfile>
4 ETW000 R3trans was called as follows: R3trans -d
4 ETW000 date&time : 16.02.2011 - 21:11:13
4 ETW000 trace at level 2 opened for a given file pointer
4 ETW000 [developertra,00000] Wed Feb 16 21:11:13 2011 232 0.000232
4 ETW000 [developertra,00000] db_con_init called 14 0.000246
4 ETW000 [developertra,00000] create_con (con_name=R/3) 34 0.000280
4 ETW000 [developertra,00000] Loading DB library '/usr/sap/QS9/SYS/exe/run/dboraslib.so' ...
4 ETW000 30 0.000310
4 ETW000 [developertra,00000] load shared library (/usr/sap/QS9/SYS/exe/run/dboraslib.so), hdl 0
4 ETW000 15933 0.016243
4 ETW000 [developertra,00000] Library '/usr/sap/QS9/SYS/exe/run/dboraslib.so' loaded
4 ETW000 18 0.016261
4 ETW000 [developertra,00000] function DbSlExpFuns loaded from library /usr/sap/QS9/SYS/exe/run/dboraslib.so
4 ETW000 22 0.016283
4 ETW000 [developertra,00000] Version of library '/usr/sap/QS9/SYS/exe/run/dboraslib.so' is "46D.00", patchlevel (0.2438)
4 ETW000 99 0.016382
4 ETW000 [developertra,00000] function dsql_db_init loaded from library /usr/sap/QS9/SYS/exe/run/dboraslib.so
4 ETW000 12 0.016394
4 ETW000 [developertra,00000] function dbdd_exp_funs loaded from library /usr/sap/QS9/SYS/exe/run/dboraslib.so
4 ETW000 19 0.016413
4 ETW000 [developertra,00000] New connection 0 created 17 0.016430
4 ETW000 [developertra,00000] db_con_connect (con_name=R/3) 13 0.016443
4 ETW000 [developertra,00000] find_con found the following connection for reuse:
4 ETW000 10 0.016453
4 ETW000 [developertra,00000] Got ORACLE_HOME=/oracle/QS9/102_64 from environment
4 ETW000 434 0.016887
4 ETW000 [developertra,00000] -->oci_initialize 23 0.016910
4 ETW000 [developertra,00000] Got ORACLE_SID=QS9 from environment 24 0.016934
4 ETW000 [developertra,00000] Got NLS_LANG=AMERICAN_AMERICA.WE8DEC from environment
4 ETW000 13 0.016947
4 ETW000 [developertra,00000] Logon as OPS$-user to get SAPR3's password 11 0.016958
4 ETW000 [developertra,00000] Connecting as /@QS9 on connection 0 ... (dbsl 46D 220209)
4 ETW000 15 0.016973
4 ETW000 [developertra,00000] -->oci_logon(con_hdl=0, user='', dbname='QS9') 15 0.016988 -
Upgradation of Primary (RAC) two node database and single node standby DB
We have our primary setup in Oracle RAC in 10g R2 (10.2.0.1.0) in HP UX 11i 64bit (PA RISC). It is a two node RAC.
And we have a single node standby database (DR).
Both the primary and standby database are existing in 10.2.0.1.0
We need to upgrade the primary and standby database from 10.2.0.1.0 to 10.2.0.4.0
Am planning to go through the DBUA method for upgrading the database.
I like to know is there any special cases for upgrading both the primary and standby database.
since as i mentioned i will upgrade the database through dbua method, and if i upgrade my standby database as same as primary,
will it forcefully open the database, because while we upgrade the database through dbua it will open the database at the end of installation.
Is it possible to upgrade the standby database with DBUA method even when my standby database is in Mount state.
I will be helpfull if anyone has done this earlier, are there any know issues and is there any good document which i can refer.
with regards
A.Aravindhi,
thanks for your reply, and i was following the document.
I like to know that is it mandatory to create a new Oracle Home for upgrading from 10.2.0.1.0 to 10.2.0.4.0
or i can proceed the installation of software in the same oracle home.
I will list out my action plan, please correct me if am wrong,
In Primary RAC,
a) Shutdown the database
b) Shutdown the asm instance
c) Stop the nodeapps services
d) Stop the cluster ready services.
e) Stop the enterprise manager
f) Stop the isqlplus
In Standby single node database,
1)Cancel the managed Recovery process
2) shutdown the database
3) Shutdown the asm instance.
PATCHING Process
g) Apply the oracle clusterware patch 10.2.0.4.0 in the same old existing ORA_CRS_HOME
h) Apply the oracle database software patch 10.2.0.4.0 in the same old existing ORACLE_HOME
Note: In my case both the ORACLE_HOME and ASM_HOME are in the same location.
i) Execute dbua to update the database in the primary database.
j) Apply the 10.2.0.4.0 patch in standby database ORACLE_HOME where as ORACLE_HOME and ASM_HOME are in the same location
k) Copy the recently generated archive logs from primary to standby manually and apply in standby database.
l) Mount the database in standby
m) Start the Managed recovery process.
with regards
A.Aravind -
Manual creation of Oracle RAC Database
Hello Guru's,
I successfully installed Oracle Grid(clusterware) and Oracle RDBMS 11.2.0.1.0 for RAC containing 2 nodes on Linux 5.3 64 bit.
All the clusterware services and ASM instance are running fine on both nodes.
Now im planning to create database manually on node1 (RAC-NODE1).
1) configure the environmental variables as follows
export TMP=/tmp
export ORACLE_HOSTNAME=`hostname`
export ORACLE_SID=finance1
export ORACLE_UNQNAME=finance
export ORACLE_BASE=/rdbms1/app/oracle/
export ORACLE_HOME=/rdbms1/app/oracle/product/11.2.0/db_home/
export ORACLE_TERM=xterm
export PATH=$ORACLE_HOME/bin:$PATH:.
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
2) created the pfile and modified the parameters as follows
db_name='finance'
processes = 200
audit_trail ='db'
db_block_size=8192
db_domain='vod.com'
diagnostic_dest='/oradump/oradata/finance/dump'
sessions=200
remote_login_passwordfile='EXCLUSIVE'
undo_tablespace='UNDOTBS1'
control_files = +CTRL
compatible ='11.2.0'
job_queue_processes = 10
undo_management = 'AUTO'
finance1.instance_name = finance1
db_create_file_dest = +DBFILE
db_create_online_log_dest_1 = +REDO1
db_create_online_log_dest_2 = +REDO2
3) Created the password file as follows
$ orapwd file=orapwfinance1 passwors=syspassword entries=10
4) edit the /etc/oratab file as follows
finance1:/rdbms1/app/oracle/product/11.2.0/db_home:N
5) created the rac db script as follows.
CREATE DATABASE FINANCE
DATAFILE '+DBFILE' SIZE 500M
AUTOEXTEND ON NEXT 50M MAXSIZE UNLIMITED
EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE '+DBFILE' SIZE 500M
AUTOEXTEND ON NEXT 50M MAXSIZE UNLIMITED
DEFAULT TEMPORARY TABLESPACE TEMP
TEMPFILE '+DBFILE' SIZE 100M AUTOEXTEND ON NEXT 10M MAXSIZE 500M
UNDO TABLESPACE UNDOTBS1
DATAFILE '+DBFILE' SIZE 100M AUTOEXTEND ON NEXT 10M MAXSIZE 500M
DEFAULT TABLESPACE USERDATA
DATAFILE '+DBFILE' SIZE 100M AUTOEXTEND ON NEXT 10M MAXSIZE 500M
LOGFILE
GROUP 1 ('+REDO1','+REDO2') SIZE 20M,
GROUP 2 ('+REDO1','+REDO2') SIZE 20M,
GROUP 3 ('+REDO1','+REDO2') SIZE 20M
MAXINSTANCES 8
MAXLOGHISTORY 300
MAXLOGFILES 64
MAXLOGMEMBERS 5
MAXDATAFILES 150
USER SYS IDENTIFIED BY "sys_123"
USER SYSTEM IDENTIFIED BY "system_123";
6) started the instance as follows.
$ sqlplus / as sysdba
SQL> startup nomount
ORACLE instance started.
Total System Global Area 217157632 bytes
Fixed Size 2211928 bytes
Variable Size 159387560 bytes
Database Buffers 50331648 bytes
Redo Buffers 5226496 bytes
SQL> @rac_db.sql
CREATE DATABASE FINANCE
ERROR at line 1:
ORA-01501: CREATE DATABASE failed
ORA-00200: control file could not be created
ORA-00202: control file: '+CTRL'
ORA-15045: ASM file name '+CTRL' is not in reference form
ORA-17502: ksfdcre:5 Failed to create file +CTRL
ORA-15081: failed to submit an I/O operation to a disk
It shows that oracle unable to create control file on +CTRL diskgroup, i found that there is a problem with permissions on disks in diskgroups.
brw-rw-r-- 1 grid asmadmin 8, 49 Apr 8 04:25 CTRLDISK1
brw-rw-r-- 1 grid asmadmin 8, 50 Apr 8 04:25 CTRLDISK2
brw-rw-r-- 1 grid asmadmin 8, 51 Apr 8 04:25 CTRLDISK3
brw-rw-r-- 1 grid asmadmin 8, 52 Apr 8 04:25 CTRLDISK4
since all the disks in diskgroup has 664 permision and owned by grid user primary group is asmadmin.
# id grid
uid=1001(grid) gid=501(oinstall) groups=501(oinstall),504(asmadmin),505(asmdba),506(asmoper)
# id oracle
uid=1002(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),505(asmdba)
please help me to overcome the above problem.
regards,Thanks for comments,
I successfully created the database, once i restared the database it is unbale to mount because of unbale to open +CTRL group file.
SQL> startup
ORACLE instance started.
Total System Global Area 217157632 bytes
Fixed Size 2211928 bytes
Variable Size 159387560 bytes
Database Buffers 50331648 bytes
Redo Buffers 5226496 bytes
ORA-00205: error in identifying control file, check alert log for more info
alert log:
ORACLE_BASE from environment = /rdbms1/app/oracle/
Thu Apr 11 09:36:34 2013
ALTER DATABASE MOUNT
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+CTRL'
ORA-17503: ksfdopn:2 Failed to open file +CTRL
ORA-15045: ASM file name '+CTRL' is not in reference form
ORA-205 signalled during: ALTER DATABASE MOUNT...
Thu Apr 11 09:36:42 2013
NOTE: initiating MARK startup
raw devices:
# ls -l /dev/sd*
brw-rw-r-- 1 grid asmadmin 8, 16 Apr 11 09:04 /dev/sdb
brw-rw-r-- 1 grid asmadmin 8, 17 Apr 11 09:51 /dev/sdb1
brw-rw-r-- 1 grid asmadmin 8, 18 Apr 11 09:51 /dev/sdb2
brw-rw-r-- 1 grid asmadmin 8, 19 Apr 11 09:51 /dev/sdb3
brw-rw-r-- 1 grid asmadmin 8, 20 Apr 11 09:45 /dev/sdb4
brw-rw-r-- 1 grid asmadmin 8, 32 Apr 11 09:04 /dev/sdc
brw-rw-r-- 1 grid asmadmin 8, 33 Apr 11 09:30 /dev/sdc1
brw-rw-r-- 1 grid asmadmin 8, 34 Apr 11 09:30 /dev/sdc2
brw-rw-r-- 1 grid asmadmin 8, 35 Apr 11 09:30 /dev/sdc3
brw-rw-r-- 1 grid asmadmin 8, 36 Apr 11 09:05 /dev/sdc4
brw-rw-r-- 1 grid asmadmin 8, 48 Apr 11 09:04 /dev/sdd
brw-rw-r-- 1 grid asmadmin 8, 49 Apr 11 09:51 /dev/sdd1
brw-rw-r-- 1 grid asmadmin 8, 50 Apr 11 09:51 /dev/sdd2
brw-rw-r-- 1 grid asmadmin 8, 51 Apr 11 09:51 /dev/sdd3
brw-rw-r-- 1 grid asmadmin 8, 52 Apr 11 09:30 /dev/sdd4
brw-rw-r-- 1 grid asmadmin 8, 64 Apr 11 09:04 /dev/sde
brw-rw-r-- 1 grid asmadmin 8, 65 Apr 11 09:51 /dev/sde1
brw-rw-r-- 1 grid asmadmin 8, 66 Apr 11 09:51 /dev/sde2
brw-rw-r-- 1 grid asmadmin 8, 67 Apr 11 09:51 /dev/sde3
brw-rw-r-- 1 grid asmadmin 8, 68 Apr 11 09:30 /dev/sde4
brw-rw-r-- 1 grid asmadmin 8, 80 Apr 11 09:04 /dev/sdf
brw-rw-r-- 1 grid asmadmin 8, 81 Apr 11 09:51 /dev/sdf1
brw-rw-r-- 1 grid asmadmin 8, 82 Apr 11 09:51 /dev/sdf2
brw-rw-r-- 1 grid asmadmin 8, 83 Apr 11 09:51 /dev/sdf3
brw-rw-r-- 1 grid asmadmin 8, 84 Apr 11 09:30 /dev/sdf4
brw-rw-r-- 1 grid asmadmin 8, 96 Apr 11 09:04 /dev/sdg
brw-rw-r-- 1 grid asmadmin 8, 97 Apr 11 09:51 /dev/sdg1
brw-rw-r-- 1 grid asmadmin 8, 98 Apr 11 09:51 /dev/sdg2
brw-rw-r-- 1 grid asmadmin 8, 99 Apr 11 09:51 /dev/sdg3
brw-rw-r-- 1 grid asmadmin 8, 100 Apr 11 09:30 /dev/sdg4
brw-rw-r-- 1 grid asmadmin 8, 112 Apr 11 09:04 /dev/sdh
brw-rw-r-- 1 grid asmadmin 8, 113 Apr 11 09:51 /dev/sdh1
brw-rw-r-- 1 grid asmadmin 8, 114 Apr 11 09:51 /dev/sdh2
brw-rw-r-- 1 grid asmadmin 8, 115 Apr 11 09:51 /dev/sdh3
brw-rw-r-- 1 grid asmadmin 8, 116 Apr 11 09:30 /dev/sdh4
asm disks:
[09:52 AM [email protected] disks]# ll
brw-rw-r-- 1 grid asmadmin 8, 65 Apr 11 09:05 ARCLOGDISK1
brw-rw-r-- 1 grid asmadmin 8, 66 Apr 11 09:05 ARCLOGDISK2
brw-rw-r-- 1 grid asmadmin 8, 67 Apr 11 09:05 ARCLOGDISK3
brw-rw-r-- 1 grid asmadmin 8, 68 Apr 11 09:05 ARCLOGDISK4
brw-rw-r-- 1 grid asmadmin 8, 49 Apr 11 09:05 CTRLDISK1
brw-rw-r-- 1 grid asmadmin 8, 50 Apr 11 09:05 CTRLDISK2
brw-rw-r-- 1 grid asmadmin 8, 51 Apr 11 09:05 CTRLDISK3
brw-rw-r-- 1 grid asmadmin 8, 52 Apr 11 09:05 CTRLDISK4
brw-rw-r-- 1 grid asmadmin 8, 33 Apr 11 09:05 DBFILEDISK1
brw-rw-r-- 1 grid asmadmin 8, 34 Apr 11 09:05 DBFILEDISK2
brw-rw-r-- 1 grid asmadmin 8, 35 Apr 11 09:05 DBFILEDISK3
brw-rw-r-- 1 grid asmadmin 8, 36 Apr 11 09:05 DBFILEDISK4
brw-rw-r-- 1 grid asmadmin 8, 113 Apr 11 09:05 FRADISK1
brw-rw-r-- 1 grid asmadmin 8, 114 Apr 11 09:05 FRADISK2
brw-rw-r-- 1 grid asmadmin 8, 115 Apr 11 09:05 FRADISK3
brw-rw-r-- 1 grid asmadmin 8, 116 Apr 11 09:05 FRADISK4
brw-rw-r-- 1 grid asmadmin 8, 81 Apr 11 09:05 REDODISK1
brw-rw-r-- 1 grid asmadmin 8, 82 Apr 11 09:05 REDODISK2
brw-rw-r-- 1 grid asmadmin 8, 83 Apr 11 09:05 REDODISK3
brw-rw-r-- 1 grid asmadmin 8, 84 Apr 11 09:05 REDODISK4
brw-rw-r-- 1 grid asmadmin 8, 97 Apr 11 09:05 REDODISK5
brw-rw-r-- 1 grid asmadmin 8, 98 Apr 11 09:05 REDODISK6
brw-rw-r-- 1 grid asmadmin 8, 99 Apr 11 09:05 REDODISK7
brw-rw-r-- 1 grid asmadmin 8, 100 Apr 11 09:05 REDODISK8
brw-rw-r-- 1 grid asmadmin 8, 17 Apr 11 09:05 VOTEDISK1
brw-rw-r-- 1 grid asmadmin 8, 18 Apr 11 09:05 VOTEDISK2
brw-rw-r-- 1 grid asmadmin 8, 19 Apr 11 09:05 VOTEDISK3
brw-rw-r-- 1 grid asmadmin 8, 20 Apr 11 09:05 VOTEDISK4
[09:53 AM [email protected] ~]# ls -l /rdbms1/app/oracle/product/11.2.0/db_home/bin/oracle
-rwsr-s--x 1 oracle asmadmin 210824720 Apr 8 13:39 /rdbms1/app/oracle/product/11.2.0/db_home/bin/oracle
I know this problem is coming for improper assiging of permissions, but i could not pin point. kindly clarify the same.
one more thing i need to know.., whenever i rebooted nodes, both of raw devices and asm disks are come to default permission brw------, i have to change every time to brw-rw-r--. is there any way to make these disks and devices permissions make permenet.
Thanks in advance. -
Not able to start one rac database
We had a 4 node RAC(11.2) environment in RH Linux.
it was running fine. By mistake we change the owner of the mount point where oracle clusterware and database installed on first node.
chown -R oracle:oinstall /u01
chmod -R 775 /u01
Due to this all services went down.
Later we change the owner, as it is on node2 and restart the services and server.
Now all the services are working fine. But oracle database is not coming up.
It is not able to read the spfile from ASM.
It is giving the error:
[oracle@ch-racsvr1 trace]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Wed Oct 6 21:52:27 2010
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+ORA_DATA/sgspace/spfilesgspace.ora'
ORA-17503: ksfdopn:2 Failed to open file +ORA_DATA/sgspace/spfilesgspace.ora
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 7077890
Additional information: 10
Please suggest, as i Think it is permissions issues. some file have permissions as rwsr-s--x in other nodes.Hi;
You have related permisson on your path? You have permisson error.
By the way i suggest move your thread to Forum Home » Grid Computing » Real Application Clusters which you could get more quick response
Regard
Helios -
RAC database cannot be connected after Grid and RAC DB have been installed
Folks,
Hello. I am installing Oracle 11gR2 RAC using 2 VMs (main VM rac1 and slave VM rac2) with Oracle Linux 5.6
according to the website http://appsdbaworkshop.blogspot.com/2011/10/11gr2-rac-on-linux-56-using-vmware.html
Grid and RAC database have been installed successfully for both VMs rac1 and rac2. I am using the slave VM rac2 to test Grid and RAC database.
I have run the commands as below:
[ora11g@rac2 bin]$ pwd
/u01/app/grid/bin
[ora11g@rac2 bin]$ ./crsctl status resource
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[ora11g@rac2 bin]$ ./srvctl config database -d racdb
PRCD-1027 : Failed to retrieve database racdb
PRCR-1070 : Failed to check if resource ora.racdb.db is registered
Cannot communicate with crsd
[ora11g@rac2 bin]$ ./sqlplus SYS/SYS
SQL*Plus: Release 11.2.0.1.0 Production on Sat Mar 10 14:07:07 2012
Copyright (c) 1982, 2009, Oracle. All rights reserved.
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
Enter user-name: SYS
Enter password:
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
Enter user-name: SYSTEM
Enter password:
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
[ora11g@rac2 /]$ /u02/11g_db/ora11g/racdb/sqldeveloper/sqldeveloper.sh
In its GUI, the inputs are below:
Connection: C1
Username: SYS
PWD: SYS
Role: SYSDBA
Connection Type: Basic
Hostname: rac2.localdomain
Port: 1521
SID: racdb2
Test: failed. Network Adapter could not establish connection.
Browser url: https://rac2.localdomain:1158/em
Its output: cannot connect to the server at https://rac2.localdomain:1158/em
As we see the outputs for commands above, Grid is running at the beginning but now is not running. SQLPLUS and SQLdeveloper cannot connect with database. Browser url cannot connect to the console https://rac2.localdomain:1158/em.
My question is:
Do any folks understand how to solve the above 4 problems ?
Thanks.Folks,
Hello. Thanks a lot for replying. The last 50 lines of the file "ocssd.log" is as below:
[ora11g@rac1 cssd]$ pwd
/u01/app/grid/log/rac1/cssd
[ora11g@rac1 cssd]$ ls
cssdOUT.log ocssd.log ocssd.trc
[ora11g@rac1 cssd]$ cat ocssd.log
2012-03-08 21:13:54.415: [ CLSF][1150617920]Destroying CLSF context
2012-03-08 21:13:55.418: [ CLSF][1150617920]Allocated CLSF context
2012-03-08 21:13:55.418: [ SKGFD][1150617920]Handle 0x7105660 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:ASMDISK1:
2012-03-08 21:13:55.418: [ CLSF][1150617920]Opened hdl:0x710d800 for dev:ORCL:ASMDISK1:
2012-03-08 21:13:55.470: [ CLSF][1150617920]Closing handle:0x710d800
2012-03-08 21:13:55.470: [ SKGFD][1150617920]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x7105660 for disk :ORCL:ASMDISK1:
2012-03-08 21:13:55.470: [ CLSF][1150617920]Destroying CLSF context
2012-03-08 21:13:56.472: [ CSSD][1685425920]clssnmlfmtlease: uniqueness 1331259161, gipc addr nm_rac-scan#192.168.137.35#36456
2012-03-08 21:13:56.608: [ CSSD][1685425920]clssnmvStatusBlkInit: myinfo nodename rac1, uniqueness 0
2012-03-08 21:13:56.697: [ CSSD][1685425920]clssnmvStatusBlkInit: myinfo nodename rac1, uniqueness 0
2012-03-08 21:13:56.754: [ CSSD][1685425920]clssnmvStatusBlkInit: myinfo nodename rac1, uniqueness 0
2012-03-08 21:13:57.425: [ CSSD][1685425920]clssnmlgetslot:lease acquisition for node rac1/slot 1 completed in 19380 msecs
2012-03-08 21:13:57.559: [ CSSD][1685425920]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg 226426240, wrtcnt, 89443, LATS 4294905400, lastSeqNo 89443, uniqueness 0, timestamp 1331258081/38436564
2012-03-08 21:13:57.559: [ CSSD][1685425920]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg 226426240, wrtcnt, 80216, LATS 4294905400, lastSeqNo 80216, uniqueness 1331259094, timestamp 1331259236/4294905820
2012-03-08 21:13:57.860: [ CSSD][1685425920]clssnmvDHBValidateNCopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 226426240, wrtcnt, 89444, LATS 4294905700, lastSeqNo 89444, uniqueness 0, timestamp 1331258081/38436564
2012-03-08 21:13:57.860: [ CSSD][1685425920]clssnmvDHBValidateNCopy: node 2, rac2, has a disk HB, but no network HB, DHB has rcfg 226426240, wrtcnt, 80217, LATS 4294905700, lastSeqNo 80217, uniqueness 1331259094, timestamp 1331259237/4294906920
2012-03-08 21:13:58.049: [ CSSD][1685425920]clssnmvDHBValidate2012-03-10 12:07:08.298: [ CSSD][471957248]clssscmain: Starting CSS daemon, version 11.2.0.1.0, in (clustered) mode with uniqueness value 1331399227
2012-03-10 12:07:08.300: [ CSSD][471957248]clssscmain: Environment is production
2012-03-10 12:07:08.300: [ CSSD][471957248]clssscmain: Core file size limit extended
2012-03-10 12:07:08.518: [ CSSD][471957248]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
2012-03-10 12:07:08.518: [ CSSD][471957248]clssscSetPrivEnv: IPMI device not installed on this node
2012-03-10 12:07:08.519: [ CSSD][471957248]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
2012-03-10 12:07:09.675: [ CSSD][471957248]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
2012-03-10 12:07:09.675: [ CSSD][471957248]clssscExtendLimits: The current soft limit for locked memory is 4294967295, hard limit is 4294967295
2012-03-10 12:07:09.675: [ CSSD][471957248]clssscmain: Running as user ora11g
[ clsdmt][1088276800]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_CSSD))
2012-03-10 12:07:09.821: [ clsdmt][1088276800]PID for the Process [5971], connkey 4
2012-03-10 12:07:09.822: [ CSSD][471957248]clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
2012-03-10 12:07:09.823: [ CSSD][471957248]clssscGetParameterOLR: OLR fetch for parameter diagwait (14) failed with rc 21
2012-03-10 12:07:09.847: [ CSSD][471957248]clssscmain: initgminfo done
2012-03-10 12:07:09.971: [ CSSD][1098766656]clssgmclientlsnr: Spawned
2012-03-10 12:07:09.971: [ CSSD][1098766656]clssgmEvtInformation: reqtype (13) cmProc ((nil)) client ((nil))
2012-03-10 12:07:09.971: [ CSSD][1098766656]clssgmEvtInformation: reqtype (13) req (0xadd7b20)
2012-03-10 12:07:09.971: [ CSSD][1098766656]clssnmQueueNotification: type (13) 0xadd7b20
2012-03-10 12:07:09.973: [ CSSD][1098766656]clssscGetParameterOLR: OLR fetch for parameter GIPC client trclvl (13) failed with rc 21
2012-03-10 12:07:09.975: [ CSSD][1098766656]clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=903cec19-00000000-5971))
2012-03-10 12:07:09.975: [ GPnP][471957248]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
2012-03-10 12:07:09.975: [ GPnP][471957248]clsgpnp_Init: [at clsgpnp0.c:534] '/u01/app/grid' in effect as GPnP home base.
2012-03-10 12:07:10.044: [GIPCCLSA][1098766656]gipcmodClsaCompleteAccept: failed on clsaauthstart ret clsaretOSD (8), endp 0xade1aa0 [000000000000002f] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=e1fdf67e-e0b9ca68-5971))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=e0b9ca68-e1fdf67e-5976))', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x1758, pidPeer 0, flags 0x603710, usrFlags 0x14000 }
2012-03-10 12:07:10.044: [GIPCCLSA][1098766656]gipcmodClsaCompleteAccept: slos op : mkdir
2012-03-10 12:07:10.044: [GIPCCLSA][1098766656]gipcmodClsaCompleteAccept: slos dep : No space left on device (28)
2012-03-10 12:07:10.044: [GIPCCLSA][1098766656]gipcmodClsaCompleteAccept: slos loc : authprep6
2012-03-10 12:07:10.044: [GIPCCLSA][1098766656]gipcmodClsaCompleteAccept: slos info: failed to make dir /u01/app/grid/auth/css/rac1/A2370613
2012-03-10 12:07:10.045: [GIPCXCPT][1098766656]gipcmodMuxTransferAccept: internal accept request failed endp 0xade0740 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=903cec19-00000000-5971))', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x30613, usrFlags 0x10010 }, ret gipcretAuthFail (22)
2012-03-10 12:07:10.045: [ GIPCMUX][1098766656]gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail (22) ] error during accept on endp 0xade0740 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=903cec19-00000000-5971))', remoteAddr '', numPend 02012-03-10 12:21:34.610: [ CSSD][4216872704]clssscmain: Starting CSS daemon, version 11.2.0.1.0, in (clustered) mode with uniqueness value 1331400094
2012-03-10 12:21:34.612: [ CSSD][4216872704]clssscmain: Environment is production
2012-03-10 12:21:34.612: [ CSSD][4216872704]clssscmain: Core file size limit extended
2012-03-10 12:21:34.675: [ CSSD][4216872704]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
2012-03-10 12:21:34.675: [ CSSD][4216872704]clssscSetPrivEnv: IPMI device not installed on this node
2012-03-10 12:21:34.676: [ CSSD][4216872704]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
2012-03-10 12:21:34.770: [ CSSD][4216872704]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
2012-03-10 12:21:34.770: [ CSSD][4216872704]clssscExtendLimits: The current soft limit for locked memory is 4294967295, hard limit is 4294967295
2012-03-10 12:21:34.771: [ CSSD][4216872704]clssscmain: Running as user ora11g
[ clsdmt][1123064128]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_CSSD))
2012-03-10 12:21:37.926: [ clsdmt][1123064128]PID for the Process [6687], connkey 4
2012-03-10 12:21:38.276: [ CSSD][4216872704]clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
2012-03-10 12:21:38.276: [ CSSD][4216872704]clssscGetParameterOLR: OLR fetch for parameter diagwait (14) failed with rc 21
2012-03-10 12:21:39.171: [ CSSD][4216872704]clssscmain: initgminfo done
2012-03-10 12:21:42.815: [ CSSD][1144043840]clssgmclientlsnr: Spawned
2012-03-10 12:21:42.815: [ CSSD][1144043840]clssgmEvtInformation: reqtype (13) cmProc ((nil)) client ((nil))
2012-03-10 12:21:42.815: [ CSSD][1144043840]clssgmEvtInformation: reqtype (13) req (0x1a5a7160)
2012-03-10 12:21:42.815: [ CSSD][1144043840]clssnmQueueNotification: type (13) 0x1a5a7160
2012-03-10 12:21:42.822: [ CSSD][1144043840]clssscGetParameterOLR: OLR fetch for parameter GIPC client trclvl (13) failed with rc 21
2012-03-10 12:21:42.872: [ CSSD][1144043840]clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=45c3536d-00000000-6687))
2012-03-10 12:21:42.873: [ GPnP][4216872704]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
2012-03-10 12:21:42.873: [ GPnP][4216872704]clsgpnp_Init: [at clsgpnp0.c:534] '/u01/app/grid' in effect as GPnP home base.
2012-03-10 12:21:43.190: [GIPCCLSA][1144043840]gipcmodClsaCompleteAccept: failed on clsaauthstart ret clsaretOSD (8), endp 0x1a5b10e0 [000000000000002f] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=c53aa194-7b930898-6687))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=7b930898-c53aa194-5928))', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x1728, pidPeer 0, flags 0x603710, usrFlags 0x14000 }
2012-03-10 12:21:43.190: [GIPCCLSA][1144043840]gipcmodClsaCompleteAccept: slos op : mkdir
2012-03-10 12:21:43.190: [GIPCCLSA][1144043840]gipcmodClsaCompleteAccept: slos dep : No space left on device (28)
2012-03-10 12:21:43.190: [GIPCCLSA][1144043840]gipcmodClsaCompleteAccept: slos loc : authprep6
2012-03-10 12:21:43.190: [GIPCCLSA][1144043840]gipcmodClsaCompleteAccept: slos info: failed to make dir /u01/app/grid/auth/css/rac1/A3702366
2012-03-10 12:21:43.201: [GIPCXCPT][1144043840]gipcmodMuxTransferAccept: internal accept request failed endp 0x1a5afd80 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=45c3536d-00000000-6687))', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x30613, usrFlags 0x10010 }, ret gipcretAuthFail (22)
2012-03-10 12:21:43.201: [ GIPCMUX][1144043840]gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail (22) ] error during accept on endp 0x1a5afd80 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=45c3536d-00000000-6687))', remot2012-03-11 11:17:15.417: [ CSSD][1249378048]clssscmain: Starting CSS daemon, version 11.2.0.1.0, in (clustered) mode with uniqueness value 1331479035
2012-03-11 11:17:15.418: [ CSSD][1249378048]clssscmain: Environment is production
2012-03-11 11:17:15.418: [ CSSD][1249378048]clssscmain: Core file size limit extended
2012-03-11 11:17:15.797: [ CSSD][1249378048]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
2012-03-11 11:17:15.797: [ CSSD][1249378048]clssscSetPrivEnv: IPMI device not installed on this node
2012-03-11 11:17:15.798: [ CSSD][1249378048]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
2012-03-11 11:17:17.472: [ CSSD][1249378048]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
2012-03-11 11:17:17.472: [ CSSD][1249378048]clssscExtendLimits: The current soft limit for locked memory is 4294967295, hard limit is 4294967295
2012-03-11 11:17:17.473: [ CSSD][1249378048]clssscmain: Running as user ora11g
[ clsdmt][1098090816]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_CSSD))
2012-03-11 11:17:17.517: [ clsdmt][1098090816]PID for the Process [5960], connkey 4
2012-03-11 11:17:17.573: [ CSSD][1249378048]clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
2012-03-11 11:17:17.574: [ CSSD][1249378048]clssscGetParameterOLR: OLR fetch for parameter diagwait (14) failed with rc 21
2012-03-11 11:17:17.586: [ CSSD][1249378048]clssscmain: initgminfo done
2012-03-11 11:17:17.617: [ CSSD][1141242176]clssgmclientlsnr: Spawned
2012-03-11 11:17:17.617: [ CSSD][1141242176]clssgmEvtInformation: reqtype (13) cmProc ((nil)) client ((nil))
2012-03-11 11:17:17.617: [ CSSD][1141242176]clssgmEvtInformation: reqtype (13) req (0x129d5160)
2012-03-11 11:17:17.617: [ CSSD][1141242176]clssnmQueueNotification: type (13) 0x129d5160
2012-03-11 11:17:17.619: [ CSSD][1141242176]clssscGetParameterOLR: OLR fetch for parameter GIPC client trclvl (13) failed with rc 21
2012-03-11 11:17:17.621: [ CSSD][1141242176]clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=38fc0b43-00000000-5960))
2012-03-11 11:17:17.621: [ GPnP][1249378048]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
2012-03-11 11:17:17.621: [ GPnP][1249378048]clsgpnp_Init: [at clsgpnp0.c:534] '/u01/app/grid' in effect as GPnP home base.
2012-03-11 11:17:18.167: [GIPCCLSA][1141242176]gipcmodClsaCompleteAccept: failed on clsaauthstart ret clsaretOSD (8), endp 0x129df0e0 [000000000000002f] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=22574215-7bcb9bab-5960))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=7bcb9bab-22574215-5919))', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x171f, pidPeer 0, flags 0x603710, usrFlags 0x14000 }
2012-03-11 11:17:18.167: [GIPCCLSA][1141242176]gipcmodClsaCompleteAccept: slos op : mkdir
2012-03-11 11:17:18.167: [GIPCCLSA][1141242176]gipcmodClsaCompleteAccept: slos dep : No space left on device (28)
2012-03-11 11:17:18.167: [GIPCCLSA][1141242176]gipcmodClsaCompleteAccept: slos loc : authprep6
2012-03-11 11:17:18.167: [GIPCCLSA][1141242176]gipcmodClsaCompleteAccept: slos info: failed to make dir /u01/app/grid/auth/css/rac1/A2554722
2012-03-11 11:17:18.168: [GIPCXCPT][1141242176]gipcmodMuxTransferAccept: internal accept request failed endp 0x129ddd80 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=38fc0b43-00000000-5960))', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x30613, usrFlags 0x10010 }, ret gipcretAuthFail (22)
2012-03-11 11:17:18.168: [ GIPCMUX][1141242176]gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail (22) ] error during accept on endp 0x129ddd80 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=38fc0b43-00000000-5960))', remot2012-03-11 13:50:45.588: [ CSSD][1847344896]clssscmain: Starting CSS daemon, version 11.2.0.1.0, in (clustered) mode with uniqueness value 1331488245
2012-03-11 13:50:45.589: [ CSSD][1847344896]clssscmain: Environment is production
2012-03-11 13:50:45.589: [ CSSD][1847344896]clssscmain: Core file size limit extended
2012-03-11 13:50:45.812: [ CSSD][1847344896]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
2012-03-11 13:50:45.812: [ CSSD][1847344896]clssscSetPrivEnv: IPMI device not installed on this node
2012-03-11 13:50:45.813: [ CSSD][1847344896]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
2012-03-11 13:50:46.364: [ CSSD][1847344896]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
2012-03-11 13:50:46.364: [ CSSD][1847344896]clssscExtendLimits: The current soft limit for locked memory is 4294967295, hard limit is 4294967295
2012-03-11 13:50:46.365: [ CSSD][1847344896]clssscmain: Running as user ora11g
[ clsdmt][1106819392]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_CSSD))
2012-03-11 13:50:46.512: [ clsdmt][1106819392]PID for the Process [5598], connkey 4
2012-03-11 13:50:46.518: [ CSSD][1847344896]clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
2012-03-11 13:50:46.521: [ CSSD][1847344896]clssscGetParameterOLR: OLR fetch for parameter diagwait (14) failed with rc 21
2012-03-11 13:50:46.548: [ CSSD][1847344896]clssscmain: initgminfo done
2012-03-11 13:50:46.611: [ CSSD][1090939200]clssgmclientlsnr: Spawned
2012-03-11 13:50:46.611: [ CSSD][1090939200]clssgmEvtInformation: reqtype (13) cmProc ((nil)) client ((nil))
2012-03-11 13:50:46.611: [ CSSD][1090939200]clssgmEvtInformation: reqtype (13) req (0x196b6160)
2012-03-11 13:50:46.611: [ CSSD][1090939200]clssnmQueueNotification: type (13) 0x196b6160
2012-03-11 13:50:46.618: [ CSSD][1090939200]clssscGetParameterOLR: OLR fetch for parameter GIPC client trclvl (13) failed with rc 21
2012-03-11 13:50:46.621: [ CSSD][1090939200]clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=e5a0c08e-00000000-5598))
2012-03-11 13:50:46.621: [ GPnP][1847344896]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
2012-03-11 13:50:46.622: [ GPnP][1847344896]clsgpnp_Init: [at clsgpnp0.c:534] '/u01/app/grid' in effect as GPnP home base.
2012-03-11 13:50:47.158: [GIPCCLSA][1090939200]gipcmodClsaCompleteAccept: failed on clsaauthstart ret clsaretOSD (8), endp 0x196c00e0 [000000000000002f] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=629302b2-9f0ad2aa-5598))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=9f0ad2aa-629302b2-5616))', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x15f0, pidPeer 0, flags 0x603710, usrFlags 0x14000 }
2012-03-11 13:50:47.158: [GIPCCLSA][1090939200]gipcmodClsaCompleteAccept: slos op : mkdir
2012-03-11 13:50:47.158: [GIPCCLSA][1090939200]gipcmodClsaCompleteAccept: slos dep : No space left on device (28)
2012-03-11 13:50:47.158: [GIPCCLSA][1090939200]gipcmodClsaCompleteAccept: slos loc : authprep6
2012-03-11 13:50:47.158: [GIPCCLSA][1090939200]gipcmodClsaCompleteAccept: slos info: failed to make dir /u01/app/grid/auth/css/rac1/A8205079
2012-03-11 13:50:47.159: [GIPCXCPT][1090939200]gipcmodMuxTransferAccept: internal accept request failed endp 0x196bed80 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=e5a0c08e-00000000-5598))', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x30613, usrFlags 0x10010 }, ret gipcretAuthFail (22)
2012-03-11 13:50:47.159: [ GIPCMUX][1090939200]gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail (22) ] error during accept on endp 0x196bed80 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=e5a0c08e-00000000-5598))', remot2012-03-11 14:15:32.370: [ CSSD][2372304640]clssscmain: Starting CSS daemon, version 11.2.0.1.0, in (clustered) mode with uniqueness value 1331489730
2012-03-11 14:15:32.383: [ CSSD][2372304640]clssscmain: Environment is production
2012-03-11 14:15:32.384: [ CSSD][2372304640]clssscmain: Core file size limit extended
2012-03-11 14:15:32.933: [ CSSD][2372304640]clssscGetParameterOLR: OLR fetch for parameter logsize (8) failed with rc 21
2012-03-11 14:15:32.933: [ CSSD][2372304640]clssscSetPrivEnv: IPMI device not installed on this node
2012-03-11 14:15:32.935: [ CSSD][2372304640]clssscGetParameterOLR: OLR fetch for parameter priority (15) failed with rc 21
2012-03-11 14:15:48.413: [ CSSD][2372304640]clssscExtendLimits: The current soft limit for file descriptors is 65536, hard limit is 65536
2012-03-11 14:15:48.413: [ CSSD][2372304640]clssscExtendLimits: The current soft limit for locked memory is 4294967295, hard limit is 4294967295
2012-03-11 14:15:48.421: [ CSSD][2372304640]clssscmain: Running as user ora11g
[ clsdmt][1119824192]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_CSSD))
2012-03-11 14:16:21.102: [ clsdmt][1119824192]PID for the Process [7155], connkey 4
2012-03-11 14:16:22.219: [ CSSD][2372304640]clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
2012-03-11 14:16:22.221: [ CSSD][2372304640]clssscGetParameterOLR: OLR fetch for parameter diagwait (14) failed with rc 21
2012-03-11 14:16:27.367: [ CSSD][2372304640]clssscmain: initgminfo done
2012-03-11 14:16:43.377: [ CSSD][1140803904]clssgmclientlsnr: Spawned
2012-03-11 14:16:43.378: [ CSSD][1140803904]clssgmEvtInformation: reqtype (13) cmProc ((nil)) client ((nil))
2012-03-11 14:16:43.378: [ CSSD][1140803904]clssgmEvtInformation: reqtype (13) req (0x6964160)
2012-03-11 14:16:43.378: [ CSSD][1140803904]clssnmQueueNotification: type (13) 0x6964160
2012-03-11 14:16:43.389: [ CSSD][1140803904]clssscGetParameterOLR: OLR fetch for parameter GIPC client trclvl (13) failed with rc 21
2012-03-11 14:16:43.554: [ CSSD][1140803904]clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=f8ea6223-00000000-7155))
2012-03-11 14:16:43.555: [ GPnP][2372304640]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
2012-03-11 14:16:43.555: [ GPnP][2372304640]clsgpnp_Init: [at clsgpnp0.c:534] '/u01/app/grid' in effect as GPnP home base.
2012-03-11 14:16:44.146: [GIPCCLSA][1140803904]gipcmodClsaCompleteAccept: failed on clsaauthstart ret clsaretOSD (8), endp 0x696e0e0 [000000000000002f] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=0448e4a3-ad06465f-7155))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=ad06465f-0448e4a3-7132))', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x1bdc, pidPeer 0, flags 0x603710, usrFlags 0x14000 }
2012-03-11 14:16:44.147: [GIPCCLSA][1140803904]gipcmodClsaCompleteAccept: slos op : mkdir
2012-03-11 14:16:44.147: [GIPCCLSA][1140803904]gipcmodClsaCompleteAccept: slos dep : No space left on device (28)
2012-03-11 14:16:44.147: [GIPCCLSA][1140803904]gipcmodClsaCompleteAccept: slos loc : authprep6
2012-03-11 14:16:44.147: [GIPCCLSA][1140803904]gipcmodClsaCompleteAccept: slos info: failed to make dir /u01/app/grid/auth/css/rac1/A8989982
2012-03-11 14:16:44.147: [GIPCXCPT][1140803904]gipcmodMuxTransferAccept: internal accept request failed endp 0x696cd80 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=f8ea6223-00000000-7155))', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x30613, usrFlags 0x10010 }, ret gipcretAuthFail (22)
2012-03-11 14:16:44.147: [ GIPCMUX][1140803904]gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail (22) ] error during accept on endp 0x696cd80 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=f8ea6223-00000000-7155))', remoteAddr[ora11g@rac1 cssd]$
My question is:
Could any folk find out the problem that Grid becomes not running after install RAC database ?
Thanks. -
Problem starting RAC database instance on Node 2
Hi all,
I have the following problem after installing RAC 10.2.0 with 2 nodes (clusternode1 & clusternode2).
I'll greatly appreciate if someone could help fix this:
oracle@clusternode2:~> srvctl status nodeapps -n clusternode1
VIP is running on node: clusternode1
GSD is running on node: clusternode1
Listener is running on node: clusternode1
ONS daemon is running on node: clusternode1
oracle@clusternode2:~> srvctl status nodeapps -n clusternode2
VIP is running on node: clusternode2
GSD is running on node: clusternode2
Listener is running on node: clusternode2
ONS daemon is running on node: clusternode2
oracle@clusternode2:~>
oracle@clusternode2:~> srvctl status asm -n clusternode1
ASM instance +ASM1 is running on node clusternode1.
oracle@clusternode2:~> srvctl status asm -n clusternode2
ASM instance +ASM2 is running on node clusternode2.
oracle@clusternode2:~>
oracle@clusternode2:~> srvctl status database -d orcl
Instance orcl1 is running on node clusternode1
Instance orcl2 is not running on node clusternode2 <-------Here is the problem!!!!!!!!!!!!!!!!
oracle@clusternode2:~>
So when I try to start the orcl2 instance I receive:
oracle@clusternode2:~> srvctl start instance -d orcl -i orcl2
PRKP-1001 : Error starting instance orcl2 on node clusternode2
CRS-0215: Could not start resource 'ora.orcl.orcl2.inst'.
oracle@clusternode2:~>
I tried to do it per SQL*PLus:
oracle@clusternode2:~> echo $ORACLE_SID
orcl2
oracle@clusternode2:~> sqlplus "/as sysdba"
SQL*Plus: Release 10.2.0.1.0 - Production on Mon Mar 27 21:08:59 2006
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+ORCLDATAGRP1/orcl/spfileorcl.ora'
ORA-17503: ksfdopn:2 Failed to open file +ORCLDATAGRP1/orcl/spfileorcl.ora
ORA-15077: could not locate ASM instance serving a required diskgroup
SQL>
But ASM is there and running !!!!!!!!!!!
Thanks very much for any tip how to get ride of this problem.
Kind regards.
Askia.muba_zak wrote:
Thanks John,
How the cron jobs takes care, is there any different is there between Admin Managed & Policy Managed
Thanks.
Muba
I don't know what your cron jobs do. If (for example) they logon to the database through a service published by the scan listeners, they will be happy. If they logon using a deterministic connect string, you have a problem - a problem caused by mis-using the system. -
Oracle instance is running but sqlplus "/as sysdba" says connected to idle.
Hi,
Oracle RAC is running(started the RAC using "srvctl start database -d hacmp") but when i am trying to connect to database using sqlplus "/as sysdba" it says "connected to idle instance".
Any idea what could be the issue here?
I have ORACLE_SID exported.
bash-3.00$ ps -ef | grep pmon
oracle 507946 798870 0 17:23:42 pts/0 0:00 grep pmon
oracle 872584 1 0 17:03:16 - 0:00 ora_pmon_hacmp1
bash-3.00$ echo $ORACLE_HOME
/u01/app/oracle/
bash-3.00$
bash-3.00$ echo $ORACLE_SID
hacmp1
bash-3.00$
bash-3.00$
bash-3.00$ sqlplus "/as sysdba"
SQL*Plus: Release 10.2.0.4.0 - Production on Mon Sep 14 17:24:43 2009
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to an idle instance.
SQLIt's not a SQL*Plus bug but a Oracle feature: when you try to connect to a local Oracle instance Oracle computes the shared memory key (used by the instance SGA) using ORACLE_HOME and ORACLE_SID: if you add a character to ORACLE_HOME, the shared memory key will be different and Oracle does not find the instance and says "connected to an idle instance".
-
Why sqlplus / as sysdba create 4 connections?
Hi all,
I have a 4 node rac set-up running 11.2.0.2.
If I log on to one of the 4 servers and connect to the database using sqlplus "/ as sysdba" it connects to the local database but also
creates another 4 connections (one on each instance) so I end up with 5 connections. I wanted to know
a) why it does this,
b) how it does this
c) can I stop it doing this (if I just so wanted to?)
Many thanksHi Billy,
your response made me dig a bit deeper, and I found out what it was!
My query to return sessions was from gv$session, and I was using the gv$session.module column to show the connection.
However, if I changed this from module to gv$session, it returned the oracle PZ99 process.
Looking on Metalink, I see this process is used to return data from remote nodes and was introduced in 10g RAC.
Looking at the columns together, you will see way I thought the way I did originally!
Inst SID SERIAL# USERNAME MODULE PROGRAM
1 35 17861 SYS sqlplus@db01 (TNS V1-V3) sqlplus@db01 (TNS V1-V3)
3 324 34635 SYS sqlplus@db01 (TNS V1-V3) oracle@db03 (PZ99)
2 320 46165 SYS sqlplus@db01 (TNS V1-V3) oracle@db02 (PZ99)
1 641 36247 SYS sqlplus@db01 (TNS V1-V3) oracle@db01 (PZ99)
4 639 39893 SYS sqlplus@db01 (TNS V1-V3) oracle@db04 (PZ99)Thanks -
Enable Archivelog Mode On 2 Node RAC DATABASE
hi,
We are having 2 Node Rac 10g Database Installed on Windows.
Currently the Database is in Non Archivelog Mode.
I want to convert it to Archivelog Mode.
I come to know the steps to convert cluster database to Archivelog mode.
i.e.
set archive parameters like Disk Location and Archivelog File format.
then Convert the database in Exclusive Mode
ALTER SYSTEM SET cluster_database=FALSE SCOPE=spfile;
then
C:\> srvctl stop database -d <Global_DB_SID>
then enable Archivelog Mode.
SQL> STARTUP MOUNT;
SQL> ARCHIVE LOG START;
SQL> ALTER DATABASE ARCHIVELOG;
SQL> ALTER DATABASE SET cluster_database=TRUE SCOPE=spfile;
SQL> SHUTDOWN IMMEDIATE
C:\> srvctl start database -d <Global_DB_SID>
But I am confuse with Do i need to RUN this commands in both the nodes seperatly or only on any one node.
Please Reply,comments embeddedf
Node 1: OSERP1
Log on to OSERP1 Server
C:\> sqlplus sys/oracle@racdb2 as sysdba
SQL> ALTER SYSTEM SET log_archive_format='arch_%t_%s.arc' SCOPE=spfile ; -- this needs to be database wide
SQL > ALTER SYSTEM SET log_archive_dest_1='location=\\OSERP1\archive$\ERP1 ' SCOPE=spfile sid ='ERP1'; -- required quotes added
-- not required
SQL> ALTER SYSTEM SET cluster_database=FALSE SCOPE=spfile;
not required to connect to other node
-- not required
SQL> ALTER SYSTEM SET log_archive_format='arch_%t_%s.arc' SCOPE=spfile sid =ERP2;
SQL > ALTER SYSTEM SET log_archive_dest_1='location=\\OSERP2\archive$\ERP2 ' SCOPE=spfile sid ='ERP2'; -- quotes added
-- not required, and database wide setting
SQL> ALTER SYSTEM SET cluster_database=FALSE SCOPE=spfile;
Then following commands will be run connecting any of the instance
-- database has to be stopped first, if you stick to switching it to a non-cluster database
C:\> set ORACLE_SID=<local_instance_id>
C:\> sqlplus / as sysdba
SQL> STARTUP MOUNT;
-- wrong sequence of commands
SQL> ARCHIVE LOG START;
SQL> ALTER DATABASE ARCHIVELOG;
-- not required (see previous steps)
SQL> ALTER DATABASE SET cluster_database=TRUE SCOPE=spfile;
SQL> SHUTDOWN IMMEDIATE;
6. Start the cluster database.
C:\> srvctl start database -d <Global_DB_SID>
-- There is no such thing as a <Global_DB_SID>. There is a database name.
Please note your tnsnames.ora, in a standard install, needs to have the following service names.
ERP
ERP1
ERP2
I'm not sure why you have the non-standard service_name racdb2.
Somehow it looks like you both don't understand the RAC concept, and didn't understand my answer. I had to correct several things I already corrected in my previous response. I'm not sure why I'm not getting through.
Sybrand Bakker
Senior Oracle DBA -
Unable to startup 12c RAC database, can't open spfile in ASM
hello,
I'm testing 12cRAC database on RHEL5 and need your help to troubleshoot and fix the following issue -
DBCA fails at the end of the configuration to create and startup new RAC database with the following errors
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/TDB1/spfileTDB1.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/TDB1/spfileTDB1.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:2 Failed to open file +DATA/TDB1/spfiletdb1.ora
ORA-01017: invalid username/password; logon denied
I also tried executing script (which was created by DBCA) to create the new DB manually and got the same results - it happens during first attempt to start it up using SPFILE after new DB was successfully created (it was started up using pfile)
Clusterware with ASM install was successful, binaries were also installed without issues,
cluster seems healthy, I see correct files within ASM using asmcmd etc ..
got stuck here and need some directions ... very confused by ORA-01017 password error
wonder if anyone had same or similar issues ?
Thank you !Fixed.
Thank you everyone for suggestions and recommendations -
ASM is accessible using asmcmd and I can see all the DB files incuding spfile, they were created by DBCA, alert log showed same information I posted
After extensive troubleshooting and testing the issue was idedntified and fixed - 'dba' group needs to be the primary group for 'oracle' user, not the secondary one. I suspected that something was not right with 'oracle' user access to asm based on that strange password error. There is more I need to check as the issue and fix with swapping groups seems strange. Secondary group is not enough ti have proper rights ??? my first reaction was - seriously ? wtf ?
Anyway - I need to move one now ...
Thank you again ! -
Data Guard configuration for RAC database disappeared from Grid control
Primary Database Environment - Three node cluster
RAC Database 10.2.0.1.0
Linux Red Hat 4.0 2.6.9-22 64bit
ASM 10.2.0.1.0
Management Agent 10.2.0.2.0
Standby Database Environment - one Node database
Oracle Enterprise Edition 10.2.0.1.0 Single standby
Linux Red Hat 4.0 2.6.9-22 64bit
ASM 10.2.0.1.0
Management Agent 10.2.0.2.0
Grid Control 10.2.0.1.0 - Node separate from standby and cluster environments
Oracle 10.1.0.1.0
Grid Control 10.2.0.1.0
Red Hat 4.0 2.6.9-22 32bit
After adding a logical standby database through Grid Control for a RAC database, I noticed sometime later the Data Guard configuration disappeared from Grid Control. Not sure why but it is gone. I did notice that something went wrong with the standby creation but i did not get much feedback from Grid Control. The last thing I did was to view the configuration, see output below.
Initializing
Connected to instance qdcls0427:ELCDV3
Starting alert log monitor...
Updating Data Guard link on database homepage...
Data Protection Settings:
Protection mode : Maximum Performance
Log Transport Mode settings:
ELCDV.qdx.com: ARCH
ELXDV: ARCH
Checking standby redo log files.....OK
Checking Data Guard status
ELCDV.qdx.com : ORA-16809: multiple warnings detected for the database
ELXDV : Creation status unknown
Checking Inconsistent Properties
Checking agent status
ELCDV.qdx.com
qdcls0387.qdx.com ... OK
qdcls0388.qdx.com ... OK
qdcls0427.qdx.com ... OK
ELXDV ... WARNING: No credentials available for target ELXDV
Attempting agent ping ... OK
Switching log file 672.Done
WARNING: Skipping check for applied log on ELXDV : disabled
Processing completed.
Here are the steps followed to add the standby database in Grid Control
Maintenance tab
Setup and Manage Data Guard
Logged in as sys
Add standby database
Create a new logical standby database
Perform a live backup of the primary database
Specify backup directory for staging area
Specify standby database name and Oracle home location
Specify file location staging area on standby node
At the end am presented with a review of the selected options and then the standby database is created
Has any body come across a similar issue?
Thanks,Any resolution on this?
I just created a Logical Standby database and I'm getting the same warning (WARNING: No credentials available for target ...) when I do a 'Verify Configuration' from the Data Guard page.
Everything else seems to be working fine. Logs are being applied, etc.
I can't figure out what credentials its looking for. -
"Alter system set command" in a RAC database!!
Hi, all.
The database is (10.2.0.2.0) 2- node RAC database on 32-bit windows 2003
EE SP1.
I issued the following command on Node 1 database.
--> Alter system set db_block_buffers= xxx sid='rac1';
I was able to see "PE enqueue" in top 5 wait event section from an AWR report.
In addition, I was able to find "PZ99","PZ98" process dump file in BDUMP.
Soon later, I could find CKPT and DBWR hung.
Is there anyone who experienced this issue?
Thanks and Regards.
Message was edited by:
user50729010.2.0.2 has some bug which is fixed 10.2.0.3
You check sequence cache. If it has less value, increase it to 10000.
select CACHE_SIZE from dba_sequences where SEQUENCE_OWNER='SYS' and SEQUENCE_NAME='AUDSES$';
SQL> alter sequence sys.audses$ cache 10000;
Ashok -
Not able to connect RAC database from client
Hi there
Recently I have configured RAC in test environment. version 11.2.0.1. OS Redhat 5.9. Everything seems to be fine except not able to connect rac database from client. Error is as under :
C:\Documents and Settings\pbl>sqlplus test1/test1@myrac
SQL*Plus: Release 10.2.0.1.0 - Production on Mon Nov 17 14:29:06 2014
Copyright (c) 1982, 2005, Oracle. All rights reserved.
ERROR:
ORA-12545: Connect failed because target host or object does not exist
Enter user-name:
myrac =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = racdb.testdb.com.bd)
Please give me your valuable suggestion to overcome the issue.
Regards
Jeweluser13134974 wrote:
ORA-12545: Connect failed because target host or object does not exist
This error means that the hostname or IP address used in the TNS connection string, failed to resolve or connect.
Your client is making two connections. The first connection is to the SCAN listener. It matches a db instance for your connection request based on service requested, available registered service handlers, load balancing, and so on. It then send a redirect to your client informing it of the handler for that service.
Your client then does a second connection to this service (a local RAC listener that will provide you with a connection to the local RAC instance). This is what seems to be failing in your case.
The SCAN listener's redirect uses the hostname of the server that the local listener is running on. Your client needs to resolve that hostname (of a RAC node) to an IP address. This likely fails.
You can add the RAC node hostnames to your client platforms hosts file. The appropriate action however would be to ensure that DNS is used for name resolution instead.
Maybe you are looking for
-
BADI/User Exit for WBS element creation (CRM to R/3)
In CRM 4.0 once a Campaign or Trade Promotion is created, a WBS is also automatically in R/3 (tcode CJ20N) I have a requirement to fill out the 'user fields' at the WBS element level when the campaign/TPM status is set to Release. I also need to fill
-
What is the effect of change material price in previous period?
i have changed the material price in 31/12/2010 and we are in period 01/2011 but the moving price doesn't changed in period 01/2011 its still the old price. i want to know why the system didn't change the price for period 01/2011 and i want to chang
-
Laser Jet Printer/Color Text Prints Grey
Hi everyone, I have a HP LaserJet 2430, black & white printer hooked up to my iMac. I'm running Snow Leopard 10.6.6. When I want to print, for example, an email where a color was used for the font, the colored font prints in grey. When I was able to
-
how do i duplicate certain parts of a path? cuz i'm tryin to join curves together and they are not smooth is if i try to trace it again [IMG]http://i40.photobucket.com/albums/e227/fusioniz/Adjoining.jpg[/IMG]
-
Inbound IDOC User Exit ZXVEDU04 change IDoc Data
Hi all, We have to develop an user exit for changing the Plant, Shipping Point and Storage location We have identified the exit ZXVEDU04 . Is there some other exit for this ?? But which internal table we are supposed to change this to change the data