Orarootagent.bin process
My topas command shows the 100% disk busy :
Topas Monitor for host: EVENTS/QUEUES FILE/TTY
Mon Jul 9 11:49:29 2012 Interval: 2 Cswitch 14657 Readch 27.4M
Syscall 33187 Writech 5215.3K
CPU User% Kern% Wait% Idle% Physc Entc Reads 1389 Rawin 0
ALL 65.3 11.1 3.7 19.9 3.11 77.6 Writes 630 Ttyout 739
Forks 2 Igets 0
Network KBPS I-Pack O-Pack KB-In KB-Out Execs 1 Namei 1160
Total 2555.8 3039.0 3171.5 1138.8 1417.0 Runqueue 3.5 Dirblk 0
Waitqueue 1.5
Disk Busy% KBPS TPS KB-Read KB-Writ MEMORY
Total 100.0 9554.0 423.0 4280.5 5273.5 PAGING Real,MB 32768
Faults 4956 % Comp 94
FileSystem KBPS TPS KB-Read KB-Writ Steals 0 % Noncomp 4
Total 32.2K 1.1K 27.4K 4.9K PgspIn 0 % Client 4
PgspOut 0
Name PID CPU% PgSp Owner PageIn 2356 PAGING SPACE
oracle 3489998 19.9 121.2 oraprod PageOut 1298 Size,MB 24576
oracle 2211986 17.6 187.0 oraprod Sios 3663 % Used 83
oracle 1605722 10.1 166.6 oraprod % Free 17
oracle 1937608 1.7 22.0 oraprod NFS (calls/sec)
oracle 3371058 1.4 27.8 oraprod SerV2 0 WPAR Activ 0
oracle 1286236 1.2 12.5 oraprod CliV2 0 WPAR Total 0
aioserve 3801164 1.1 0.4 oraprod SerV3 0 Press: "h"-help
oracle 2142450 1.1 10.7 oraprod CliV3 0 "q"-quit
oracle 3162156 1.0 32.7 oraprod
oracle 2461744 0.8 23.6 oraprod
aioserve 3191026 0.8 0.4 oraprod
oracle 3244104 0.7 14.1 oraprod
oracle 1896628 0.5 14.1 oraprod
oraroota 991252 0.5 19212.0 root
oracle 3518704 0.5 28.0 oraprod
oracle 1028232 0.4 35.3 oraprod
oracle 4083846 0.4 13.8 oraprod
oracle 1245258 0.4 20.6 oraprod
oracle 2043980 0.4 27.4 oraprod
aioserve 1876136 0.3 0.4 oraprod The following process is having large pgsp
ps -ef |grep 991252
root 991252 1 0 Dec 25 - 4792:11 /oracrs/apps/11.2.0/bin/orarootagent.bin
I don't know which process is responsible for this high disk utilization.
Kindly help :(
Hi,
have a look at --> ORAAGENT or ORAROOTAGENT High Resource (CPU, Memory etc) Usage (Doc ID 1062676.1)
Anand
Similar Messages
-
Stop oraagent.bin & orarootagent.bin in a Server
Hi
I am trying to install an Oracle Server in a RAC node. There are 2 services blocking the installation. The Wizard tells me that I have to stop those services.
grid 4456516 1 0 14:17:09 - 0:00 /u01/app/11.2.0/grid/bin/oraagent.bin
root 3604948 1 0 Jan 09 - 504:03 /u01/app/11.2.0/grid/bin/orarootagent.bin
I am trying to to do it so:
[root@node1 root]# crsctl stop crs
but those processes are still alive. How can I do to stop those services ?
ThanksI resolved the problem. Applying this command and rebooting the OS.
crsctl disable crs
Thanks -
First of all, sorry for my pour English.
O.S.:
pceudb9:~ # cat /etc/*release
SUSE Linux Enterprise Server 10 (x86_64)
VERSION = 10
PATCHLEVEL = 2
Database:
SQL> select * from v$version;
BANNER
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE 11.2.0.1.0 Production
TNS for Linux: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production
SQL>
I'm having this process on TOP whith high CPU usage:
oracle@pceudb9:~> top
top - 15:22:25 up 576 days, 21:09, 3 users, load average: 1.32, 1.50, 1.53
Tasks: 326 total, 1 running, 325 sleeping, 0 stopped, 0 zombie
Cpu(s): 12.7%us, 0.1%sy, 0.1%ni, 86.7%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 66012172k total, 61524088k used, 4488084k free, 1595320k buffers
Swap: 67108856k total, 64748k used, 67044108k free, 18705464k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16026 oracle 16 0 1440m 807m 20m S 100 1.3 28575:34 /u01/app/oragrid/11.2.0/jdk/jre//bin/java -Doracle.supercluster.cluster.server=eonsd -Djava.net.p
6925 oracle 16 0 9880 1528 976 R 0 0.0 0:00.01 top
oracle@pceudb9:~> ps auxww|grep -i 16026|grep -iv grep|grep -iv /u01/app/oragrid/11.2.0/bin/orarootagent.bin
oracle 16026 16.4 1.2 1475532 826580 ? Sl Jul20 28576:32 /u01/app/oragrid/11.2.0/jdk/jre//bin/java -Doracle.supercluster.cluster.server=eonsd -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/u01/app/oragrid/11.2.0/srvm/admin/logging.properties -classpath /u01/app/oragrid/11.2.0/jdk/jre//lib/rt.jar:/u01/app/oragrid/11.2.0/jlib/srvm.jar:/u01/app/oragrid/11.2.0/jlib/srvmhas.jar:/u01/app/oragrid/11.2.0/jlib/supercluster.jar:/u01/app/oragrid/11.2.0/jlib/supercluster-common.jar:/u01/app/oragrid/11.2.0/ons/lib/ons.jar oracle.supercluster.impl.cluster.EONSServerImpl
CPU is always 100% (sometimes more...)
Can somebody help me please? What is this process? This is a bug? I can't find anything on metalink.
Thanks in advice.
Regards
VitorThanks for reply. Answer for questions:
There was a bug where if you had a DB that was registered in CRS using an older version of srvctl it would cause eons to spin.
Is there only one instance on this node? Has it always been 11gR2? Are there repeating messages in $GRID_HOME/log/pceudb9/agent/crsd/oraagent_oracle/oraagent_oracle.log ?
Edit: I found the MOS note: 1062675.1
I was remembering the bug incorrectly -- it happens if you register a DB with a newer version of srvctl than the DB is. Same questions would apply, though.
Is there only one instance on this node?
oracle@pceudb9:~> ps auxww|grep -i pmon
oracle 15555 0.0 0.1 190492 88356 ? Ss Jul20 0:05 asm_pmon_+ASM1
oracle 16181 0.0 50.8 214080 33576804 ? Ss Jul20 1:52 ora_pmon_jpharm1
oracle 21318 0.0 0.0 3932 836 pts/2 S+ 16:07 0:00 grep -i pmon
oracle@pceudb9:~>
Has it always been 11gR2? No. We upgraded this db.
Are there repeating messages in $GRID_HOME/log/pceudb9/agent/crsd/oraagent_oracle/oraagent_oracle.log ?
oracle@pceudb9:/u01/app/oragrid/11.2.0/log> ls -ltrh pceudb9/agent/crsd/oraagent_oracle/oraagent_oracle.log
-rw-r--r-- 1 oracle oinstall 1.9M 2011-11-18 16:10 pceudb9/agent/crsd/oraagent_oracle/oraagent_oracle.log
tail on file:
2011-11-18 16:12:02.617: [ora.asm][1510000960] [check] CrsCmd::ClscrsCmdData filter on LAST_SERVER eq pceudb9
2011-11-18 16:12:02.617: [ora.asm][1510000960] [check] CrsCmd::ClscrsCmdData filter on NAME eq ora.asm
2011-11-18 16:12:02.617: [ora.asm][1510000960] [check] CrsCmd::stat resName ora.asm statflag 1 useFilter 0
2011-11-18 16:12:02.617: [ora.asm][1510000960] [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 1 useFilter 0
2011-11-18 16:12:02.640: [ora.asm][1510000960] [check] CrsCmd::ClscrsCmdData::stat resname ora.asm 1 1 attr STATE value ONLINE
2011-11-18 16:12:02.640: [ora.asm][1510000960] [check] CrsCmd::stat resName ora.asm state ONLINE
2011-11-18 16:12:02.641: [ora.asm][1510000960] [check] CrsCmd::ClscrsCmdData::stat resname ora.asm 1 1 attr STATE_DETAILS value Started
2011-11-18 16:12:02.641: [ora.asm][1510000960] [check] CrsCmd::stat resName ora.asm state details Started
2011-11-18 16:12:02.641: [ora.asm][1510000960] [check] CrsCmd::ClscrsCmdData::stat resname ora.asm 1 1 attr INCARNATION value 1
2011-11-18 16:12:02.641: [ AGFW][1510000960] check for resource: ora.asm pceudb9 1 completed with status: ONLINE
2011-11-18 16:12:03.117: [ AGFW][1342208320] CHECK initiated by timer for: ora.jpharm.db 1 1
2011-11-18 16:12:03.118: [ AGFW][1543559488] Executing command: check for resource: ora.jpharm.db 1 1
2011-11-18 16:12:03.118: [ora.jpharm.db][1543559488] [check] Gimh::check condition (GIMH_NEXT_NUM) 9 exists
2011-11-18 16:12:03.118: [ AGFW][1543559488] check for resource: ora.jpharm.db 1 1 completed with status: ONLINE
2011-11-18 16:12:04.117: [ AGFW][1342208320] CHECK initiated by timer for: ora.jpharm.db 1 1
2011-11-18 16:12:04.118: [ AGFW][1711352128] Executing command: check for resource: ora.jpharm.db 1 1
2011-11-18 16:12:04.118: [ora.jpharm.db][1711352128] [check] Gimh::check condition (GIMH_NEXT_NUM) 9 exists
2011-11-18 16:12:04.118: [ AGFW][1711352128] check for resource: ora.jpharm.db 1 1 completed with status: ONLINE
2011-11-18 16:12:05.117: [ AGFW][1342208320] CHECK initiated by timer for: ora.jpharm.db 1 1
2011-11-18 16:12:05.118: [ AGFW][1778469184] Executing command: check for resource: ora.jpharm.db 1 1
2011-11-18 16:12:05.118: [ora.jpharm.db][1778469184] [check] Gimh::check condition (GIMH_NEXT_NUM) 9 exists
2011-11-18 16:12:05.118: [ AGFW][1778469184] check for resource: ora.jpharm.db 1 1 completed with status: ONLINE
2011-11-18 16:12:06.117: [ AGFW][1342208320] CHECK initiated by timer for: ora.jpharm.db 1 1
2011-11-18 16:12:06.118: [ AGFW][1744910656] Executing command: check for resource: ora.jpharm.db 1 1
2011-11-18 16:12:06.118: [ora.jpharm.db][1744910656] [check] Gimh::check condition (GIMH_NEXT_NUM) 9 exists
2011-11-18 16:12:06.118: [ AGFW][1744910656] check for resource: ora.jpharm.db 1 1 completed with status: ONLINE
2011-11-18 16:12:07.117: [ AGFW][1342208320] CHECK initiated by timer for: ora.jpharm.db 1 1
2011-11-18 16:12:07.118: [ AGFW][1308649792] Executing command: check for resource: ora.jpharm.db 1 1
2011-11-18 16:12:07.118: [ora.jpharm.db][1308649792] [check] Gimh::check condition (GIMH_NEXT_NUM) 9 exists
2011-11-18 16:12:07.118: [ AGFW][1308649792] check for resource: ora.jpharm.db 1 1 completed with status: ONLINE
2011-11-18 16:12:08.117: [ AGFW][1342208320] CHECK initiated by timer for: ora.jpharm.db 1 1
2011-11-18 16:12:08.118: [ AGFW][1644235072] Executing command: check for resource: ora.jpharm.db 1 1
2011-11-18 16:12:08.118: [ora.jpharm.db][1644235072] [check] Gimh::check condition (GIMH_NEXT_NUM) 9 exists
2011-11-18 16:12:08.118: [ AGFW][1644235072] check for resource: ora.jpharm.db 1 1 completed with status: ONLINE
oracle@pceudb9:/u01/app/oragrid/11.2.0/log>
Thanks again! :) -
HI all,
I have non rac asm instance installed on 11gr2 database
$ ps -ef |grep -i asm
oracle 9219 1126 0 Dec 18 ? 15:04 /u01/app/asmgrid/product/11.2.0/grid/bin/ohasd.bin reboot
oracle 9623 1126 0 Dec 18 ? 20:55 /u01/app/asmgrid/product/11.2.0/grid/bin/ocssd.bin
oracle 9611 1126 0 Dec 18 ? 4:15 /u01/app/asmgrid/product/11.2.0/grid/bin/cssdagent
oracle 9613 1126 0 Dec 18 ? 3:05 /u01/app/asmgrid/product/11.2.0/grid/bin/orarootagent.bin
oracle 9633 1126 0 Dec 18 ? 6:35 /u01/app/asmgrid/product/11.2.0/grid/bin/diskmon.bin -d -f
I had shutdown the asm instance and the database but still i can see the above processes running, If i want to reboot the server i needed to stop these processes without using kill option. please help me how to stop the above processes for rebooting the server?
thanks allI know this thread is old, but I'll post on it anyway, because I think this should be useful:
If you tun the following command (crs_stat -t) you'll find something like the following:
-bash-3.00$ crs_stat -t
Name Type Target State Host
ora....ER.lsnr ora....er.type ONLINE ONLINE test02
ora.SYSTEM.dg ora....up.type ONLINE ONLINE test02
ora.asm ora.asm.type ONLINE ONLINE test02
ora.cssd ora.cssd.type ONLINE ONLINE test02
ora.diskmon ora....on.type ONLINE ONLINE test02
(Assuming you're not running clustered instance.)
What you have to do is:
First of all, shutdown instances, ASM first and RDBMS in second place. After that you need to re-run the above command (crs_stat -t) to identify processes that still are alive.
Once you find them, run the following command:
crs_stop <resource>
e.g.:
crs_stop ora.cssd
Attempting to stop `ora.cssd` on member `test02`
Stop of `ora.cssd` on member `test02` succeeded.
The process 'ohas' (Oracle High Availability Service) runs at startup with respawn option, so, you cannot kill it. Normally it's safe to reboot your server with this process up.
Best regards! -
Hi All,
Running 11.2.0.1.0 on RHEL5.3.
I've got a box that seems to be leaking memory over time. Slowly, more and more memory is allocated, and more and more swap gets allocated. At first, I thought I was a bit too aggressive with my SGA memory/HugePages reservations...but we recently reduced those, and we still see swap usage slowly but steadily increasing.
I was poking around a bit, and I ran:
ps aux --sort=vszand I see:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
...many lines omitted....
oracle 15493 0.7 0.0 492100 38596 ? Ss Jul08 79:28 asm_lmd0_+ASM4
oracle 15777 0.0 0.0 494204 26440 ? Ss Jul08 0:05 asm_smon_+ASM4
oracle 15783 0.1 0.0 547480 20384 ? Ss Jul08 17:34 asm_gmon_+ASM4
oracle 13087 1.5 0.4 556524 368056 ? SLl Jul08 166:32 /grid/product/11.2.0/grid/bin/ocssd.bin
oracle 12952 0.1 0.0 587004 15188 ? Ssl Jul08 20:04 /grid/product/11.2.0/grid/bin/oraagent.bin
root 17895 0.2 0.0 835060 12768 ? Ssl Jul08 31:35 /grid/product/11.2.0/grid/bin/orarootagent.bin
root 7924 0.3 0.3 978556 271116 ? Ssl Jul08 37:50 /grid/product/11.2.0/grid/bin/ohasd.bin reboot
oracle 18447 0.4 0.2 1634656 180016 ? Sl Jul08 53:19 /grid/product/11.2.0/grid/jdk/jre//bin/java -Doracle.
root 17510 0.3 2.6 4220208 1964940 ? Ssl Jul08 32:26 /grid/product/11.2.0/grid/bin/crsd.bin reboot
oracle 17887 5.6 13.4 19577880 9975984 ? Ssl Jul08 601:36 /grid/product/11.2.0/grid/bin/oraagent.binIt's that last one, oraagent.bin that concerns me.
It seems that oraagent.bin VSZ and RSS are slowly, but steadily, climbing.
Looking at a pmap -x, it seems to be allocating a whole bunch of '[anon]' memory chunks, mostly 32k or 64k in size:
-bash-3.2$ pmap -x 17887
17887: /grid/product/11.2.0/grid/bin/oraagent.bin
Address Kbytes RSS Anon Locked Mode Mapping
.........many, many lines omitted for brevity......
00002aac3c000000 65536 - - - rw--- [ anon ]
00002aac40000000 65516 - - - rw--- [ anon ]
00002aac43ffb000 20 - - - ----- [ anon ]
00002aac44000000 65536 - - - rw--- [ anon ]
00002aac48000000 65536 - - - rw--- [ anon ]
00002aac4c000000 65536 - - - rw--- [ anon ]
00002aac50000000 65536 - - - rw--- [ anon ]
00002aac54000000 65536 - - - rw--- [ anon ]
00002aac58000000 65536 - - - rw--- [ anon ]
00002aac5c000000 65532 - - - rw--- [ anon ]
00002aac5ffff000 4 - - - ----- [ anon ]
00002aac60000000 65536 - - - rw--- [ anon ]
00002aac64000000 65536 - - - rw--- [ anon ]
00002aac68000000 65536 - - - rw--- [ anon ]
00002aac6c000000 65516 - - - rw--- [ anon ]
00002aac6fffb000 20 - - - ----- [ anon ]
00002aac70000000 65536 - - - rw--- [ anon ]
00002aac74000000 65444 - - - rw--- [ anon ]
00002aac77fe9000 92 - - - ----- [ anon ]
00002aac78000000 65536 - - - rw--- [ anon ]
00002aac7c000000 65536 - - - rw--- [ anon ]
00002aac80000000 65532 - - - rw--- [ anon ]
00002aac83fff000 4 - - - ----- [ anon ]
00002aac84000000 65532 - - - rw--- [ anon ]
00002aac87fff000 4 - - - ----- [ anon ]
00002aac88000000 65524 - - - rw--- [ anon ]
00002aac8bffd000 12 - - - ----- [ anon ]
00002aac8c000000 65516 - - - rw--- [ anon ]
00002aac8fffb000 20 - - - ----- [ anon ]
00002aac90000000 65536 - - - rw--- [ anon ]
00002aac94000000 65536 - - - rw--- [ anon ]
00002aac98000000 65536 - - - rw--- [ anon ]
00002aac9c000000 65524 - - - rw--- [ anon ]
00002aac9fffd000 12 - - - ----- [ anon ]
00002aaca0000000 65496 - - - rw--- [ anon ]
00002aaca3ff6000 40 - - - ----- [ anon ]
00002aaca4000000 65528 - - - rw--- [ anon ]
00002aaca7ffe000 8 - - - ----- [ anon ]
00002aaca8000000 65536 - - - rw--- [ anon ]
00002aacac000000 65536 - - - rw--- [ anon ]
00002aacb0000000 65532 - - - rw--- [ anon ]
00002aacb3fff000 4 - - - ----- [ anon ]
00002aacb4000000 65524 - - - rw--- [ anon ]
00002aacb7ffd000 12 - - - ----- [ anon ]
00002aacb8000000 65532 - - - rw--- [ anon ]
00002aacbbfff000 4 - - - ----- [ anon ]
00002aacbc000000 65512 - - - rw--- [ anon ]
00002aacbfffa000 24 - - - ----- [ anon ]
00002aacc0000000 65536 - - - rw--- [ anon ]
00002aacc4000000 65536 - - - rw--- [ anon ]
00002aacc8000000 65536 - - - rw--- [ anon ]
00002aaccc000000 65532 - - - rw--- [ anon ]
00002aaccffff000 4 - - - ----- [ anon ]
00002aacd0000000 65524 - - - rw--- [ anon ]
00002aacd3ffd000 12 - - - ----- [ anon ]
00002aacd4000000 65536 - - - rw--- [ anon ]
00002aacd8000000 65536 - - - rw--- [ anon ]
00002aacdc000000 65536 - - - rw--- [ anon ]
00002aace0000000 65536 - - - rw--- [ anon ]
00002aace4000000 65524 - - - rw--- [ anon ]
00002aace7ffd000 12 - - - ----- [ anon ]
00002aace8000000 65536 - - - rw--- [ anon ]
00002aacec000000 65444 - - - rw--- [ anon ]
00002aaceffe9000 92 - - - ----- [ anon ]
00002aacf0000000 65536 - - - rw--- [ anon ]
00002aacf4000000 65536 - - - rw--- [ anon ]
00002aacf8000000 65536 - - - rw--- [ anon ]
00002aacfc000000 65536 - - - rw--- [ anon ]
00002aad00000000 65528 - - - rw--- [ anon ]
00002aad03ffe000 8 - - - ----- [ anon ]
00002aad04000000 65536 - - - rw--- [ anon ]
00002aad08000000 65536 - - - rw--- [ anon ]
00002aad0c000000 65524 - - - rw--- [ anon ]
00002aad0fffd000 12 - - - ----- [ anon ]
00002aad10000000 65536 - - - rw--- [ anon ]
00002aad14000000 65536 - - - rw--- [ anon ]
00002aad18000000 65532 - - - rw--- [ anon ]
00002aad1bfff000 4 - - - ----- [ anon ]
00002aad1c000000 65536 - - - rw--- [ anon ]
00002aad20000000 32768 - - - rw--- [ anon ]
00002aad22001000 31744 - - - rw--- [ anon ]
00002aad24000000 65536 - - - rw--- [ anon ]
00002aad28000000 65520 - - - rw--- [ anon ]
00002aad2bffc000 16 - - - ----- [ anon ]
00002aad2c000000 65536 - - - rw--- [ anon ]
00002aad30000000 65536 - - - rw--- [ anon ]
00002aad34000000 65520 - - - rw--- [ anon ]
00002aad37ffc000 16 - - - ----- [ anon ]
00002aad38000000 65536 - - - rw--- [ anon ]
00002aad3c000000 65536 - - - rw--- [ anon ]
00002aad40000000 32768 - - - rw--- [ anon ]
00002aad42001000 31744 - - - rw--- [ anon ]
00002aad44000000 65532 - - - rw--- [ anon ]
00002aad47fff000 4 - - - ----- [ anon ]
00002aad48000000 65536 - - - rw--- [ anon ]
00002aad4c000000 65536 - - - rw--- [ anon ]
00002aad50000000 65532 - - - rw--- [ anon ]
00002aad53fff000 4 - - - ----- [ anon ]
00002aad54000000 65536 - - - rw--- [ anon ]
00002aad58000000 65536 - - - rw--- [ anon ]
00002aad5c000000 65532 - - - rw--- [ anon ]
00002aad5ffff000 4 - - - ----- [ anon ]
00002aad60000000 65536 - - - rw--- [ anon ]
00002aad64000000 65504 - - - rw--- [ anon ]
00002aad67ff8000 32 - - - ----- [ anon ]
00002aad68000000 65536 - - - rw--- [ anon ]
00002b4865c05000 4 - - - rw--- [ anon ]
00002b4865c06000 4256 - - - r-x-- libhasgen11.so
00002b486602e000 1024 - - - ----- libhasgen11.so
00002b486612e000 72 - - - rw--- libhasgen11.so
00002b4866140000 16 - - - rw--- [ anon ]
00002b4866144000 35368 - - - r-x-- libclntsh.so.11.1
00002b48683ce000 2044 - - - ----- libclntsh.so.11.1
00002b48685cd000 1524 - - - rw--- libclntsh.so.11.1
00002b486874a000 144 - - - rw--- [ anon ]
00002b486876e000 8 - - - r-x-- libskgxn2.so
00002b4868770000 1020 - - - ----- libskgxn2.so
00002b486886f000 4 - - - rw--- libskgxn2.so
00002b4868885000 8 - - - rw--- [ anon ]
00002b4868887000 444 - - - r-x-- libasmclnt11.so
00002b48688f6000 1024 - - - ----- libasmclnt11.so
00002b48689f6000 72 - - - rw--- libasmclnt11.so
00002b4868a08000 4 - - - rw--- [ anon ]
00002b4868a09000 316 - - - r-x-- libcell11.so
00002b4868a58000 1020 - - - ----- libcell11.so
00002b4868b57000 32 - - - rw--- libcell11.so
00002b4868b5f000 4 - - - rw--- [ anon ]
00002b4868b60000 732 - - - r-x-- libskgxp11.so
00002b4868c17000 1024 - - - ----- libskgxp11.so
00002b4868d17000 8 - - - rw--- libskgxp11.so
00002b4868d19000 2580 - - - r-x-- libnnz11.so
00002b4868f9e000 1020 - - - ----- libnnz11.so
00002b486909d000 268 - - - rw--- libnnz11.so
00002b48690e0000 4 - - - rw--- [ anon ]
00002b48690e1000 1160 - - - r-x-- libgns11.so
00002b4869203000 1020 - - - ----- libgns11.so
00002b4869302000 20 - - - rw--- libgns11.so
00002b4869307000 4 - - - rw--- [ anon ]
00002b4869308000 136 - - - r-x-- libeons.so
00002b486932a000 1020 - - - ----- libeons.so
00002b4869429000 8 - - - rw--- libeons.so
00002b486942b000 100 - - - r-x-- libonsx.so
00002b4869444000 1024 - - - ----- libonsx.so
00002b4869544000 4 - - - rw--- libonsx.so
00002b4869545000 12 - - - rw--- [ anon ]
00002b4869548000 4 - - - r-x-- libaio.so.1.0.1
00002b4869549000 2044 - - - ----- libaio.so.1.0.1
00002b4869748000 4 - - - rw--- libaio.so.1.0.1
00002b4869749000 292 - - - rw--- [ anon ]
00002b48697a7000 40 - - - r-x-- libnss_files-2.5.so
00002b48697b1000 2044 - - - ----- libnss_files-2.5.so
00002b48699b0000 4 - - - r---- libnss_files-2.5.so
00002b48699b1000 4 - - - rw--- libnss_files-2.5.so
00002b48699b2000 160 - - - rw--- [ anon ]
00007fff452c2000 140 - - - rwx-- [ stack ]
00007fff452e5000 4 - - - rw--- [ anon ]
ffffffffff600000 8192 - - - ----- [ anon ]
total kB 19586072 - - -Is anyone aware of a memory leak in oraagent.bin in 11.2.0.1.0? I did a Google search and a MOS search, no luck at all.
Any thoughts or suggestions are welcome.
Thanks,
-MarkAlso noticed, this is only happening on node 4 of a 4 node cluster. The other three nodes, the memory usage of oraagent.bin is an order of magnitude lower, and does not appear to be growing. On node 4, it has huge memory allocation and continues to grow, and also, seems to have accrued a lot more CPU time (which, I suppose is consistent).
And yes, all 4 nodes have been up almost exactly the same amount of time (1 week, 1 day), within a few minutes of each other.
-Mark -
Stock transfer from bin to bin
hi,
In stock transfer from bin to bin process i need solution for the below requirement...
a)The system must raise a warning when a material is transferred to another bin, which already contains the same material with a different batch number.
b)Transferring stock to a single batch location which already contains another material/batch is not allowed.
In bin to bin stock transfer process, it is possible by system to propose destination bin automatically ?
Thanks
Muthuraman.DThe requirements that you have may be fulfilled using 'mixed storage' options available in the storage type configuration. However, there are rigid prerequisities to use certain options such as usage of storage unit management and bulk storage strategy. You may look into a custom putaway strategy (MWMTO003) to fulfill your requirements.
As for automatic proposal of bin during bin-bin transfer, it is pretty much the same set up that you'd use for any other putaway. This includes, enabling storage type search (sections & bin type, if applicable) along with indicators, defining the relevant putaway strategy, and assignment of indicators in the material masters. -
Any tips on Controlling interactive bash shell on Cygwin via Java process?
Ideally, I'd like to be able to control an interactive bash shell on Cygwin from a Java process. But for starters, I want to do 'ls > /cygdrive/c/fred.txt'. The little program to do this, below, doesn't work. It simply runs without error.
Any ideas? And also, any tips on doing a fully interactive shell?
--> Java program to test simple command:
package com.treelogic_swe.real_estate_explorer.utils;
import java.io.File;
import java.io.InputStream;
* Quick and dirty test of Java to Cygwin bash shell process control.
* Thanks to [email protected]
* re: http://groups.google.com/group/comp.lang.java.help/browse_thread/thread/914403fbdd1de127/40363d2e7cbae823?lnk=st&q=cygwin+shell+java+process+runtime&rnum=1&hl=en#40363d2e7cbae823
public class RunSystemCommand {
public static void main(String args[]) throws Throwable {
// System command to run
String cmd = "ls > /cygdrive/c/fred.txt";
String [] sysArgs = { };
// Set the working directory for the OS command processor
File workDir = new File( "c:/cygwin/bin" );
Process bash = Runtime.getRuntime().exec( cmd, sysArgs, workDir );
InputStream bashIn = bash.getInputStream();
while( bashIn.read() != -1 ); // Added, as otherwise the ls hangs.
bash.waitFor();
}Ideally, I'd like to be able to control an
interactive bash shell on Cygwin
What do you mean by this? Do you just want to start a bash shell, and have it's input and output come from and go to your Java program?
Please be specific about what you want, as there are a couple of interpretations for what you said. You need to understand the differences and relationships among terminals, shells, shell builtins, and executables.
You'll want to read this:
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html
from a Java
process. But for starters, I want to do 'ls >
/cygdrive/c/fred.txt'. The little program to do
this, below, doesn't work. It simply runs without
error.
, <, |, and others are specific to the shell. When you're in a terminal, and you type ls > /tmp/zzz, it's the bash shell that interprets the >. Cygwin doesn't know anything about it. ls doesn't know anything about it.So if you want to to that, the command you'd execute from Java is not ls, but bash (or some other shell). You'd then tell bash to execute the rest. I'm not sure what the command line arg is for bash to tell it the rest of the command line args are a command that bash should interpret and execute. I think it might be -c like it is for zsh. So you'd need to do something like Runtie.getRuntime().exec("bash -c 'ls > /tmp/zzz'"); -
Issues with processing on 64bit (java)
Anyone running processing (http://processing.org) on a 64bit machine?
I'm having some issues getting it running, works fine on my 32bit system though.
My error is:
./processing: line 100: /home/action-owl/dev/processing/processing-1.0.9/java/bin/java: No such file or directory
The file does exist, and it's executable, the processing forums have been a bit quiet lately.
Thankscarlosgottberg wrote:Looking at /usr/bin/processing found out that you have to symlink /usr/bin/java to /usr/share/processing/java/bin/java .
This doesn't seem to work for me, but maybe I'm doing it wrongly... Is this close to what you meant?:
sudo ln -s /usr/bin/java /usr/share/processing/java/bin/java -
Failed root.sh on 1st Node of 11.2.0.2.0 RAC on HP-UX 11.31 Itanium 64
Started with 11.2.0.2.0 Grid Installation for 2 Node RAC on HP-UX 11.31 Itanium 64.
Copying Software to remote node & linking libraries were successfully without any issue (upto 76%). But got issue while executing root.sh on Node1
sph1erp:/oracle/11.2.0/grid #sh root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'sph1erp'
CRS-2676: Start of 'ora.mdnsd' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'sph1erp'
CRS-2676: Start of 'ora.gpnpd' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'sph1erp'
CRS-2672: Attempting to start 'ora.gipcd' on 'sph1erp'
CRS-2676: Start of 'ora.gipcd' on 'sph1erp' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'sph1erp'
CRS-2672: Attempting to start 'ora.diskmon' on 'sph1erp'
CRS-2676: Start of 'ora.diskmon' on 'sph1erp' succeeded
CRS-2676: Start of 'ora.cssd' on 'sph1erp' succeeded
ASM created and started successfully.
Disk Group OCRVOTE created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk ab847ed2b4f04f2dbfb875226d2bb194.
Successful addition of voting disk 85c05a5b30384f8dbff48cc069de7a7c.
Successful addition of voting disk 649196fbdd614f9cbf26a9a0e6670a6e.
Successful addition of voting disk 8815dfcee2e64f64bf00b9c76626ab41.
Successful addition of voting disk 8ce55fe5534f4f77bfa9f54187592707.
Successfully replaced voting disk group with +OCRVOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE ab847ed2b4f04f2dbfb875226d2bb194 (/dev/oracle/ocrvote1) [OCRVOTE]
2. ONLINE 85c05a5b30384f8dbff48cc069de7a7c (/dev/oracle/ocrvote2) [OCRVOTE]
3. ONLINE 649196fbdd614f9cbf26a9a0e6670a6e (/dev/oracle/ocrvote3) [OCRVOTE]
4. ONLINE 8815dfcee2e64f64bf00b9c76626ab41 (/dev/oracle/ocrvote4) [OCRVOTE]
5. ONLINE 8ce55fe5534f4f77bfa9f54187592707 (/dev/oracle/ocrvote5) [OCRVOTE]
Located 5 voting disk(s).
Start of resource "ora.cluster_interconnect.haip" failed
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'sph1erp'
CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
Start action for HAIP aborted
CRS-2674: Start of 'ora.cluster_interconnect.haip' on 'sph1erp' failed
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'sph1erp'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'sph1erp' succeeded
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Clusterware stack
Failed to start High Availability IP at /oracle/11.2.0/grid/crs/install/crsconfig_lib.pm line 1046.
*/oracle/11.2.0/grid/perl/bin/perl -I/oracle/11.2.0/grid/perl/lib -I/oracle/11.2.0/grid/crs/install /oracle/11.2.0/grid/crs/install/rootcrs.pl execution failed*
sph1erp:/oracle/11.2.0/grid #
Last few lines from CRS Log for node 1, where error came
[ctssd(6467)]CRS-2401:The Cluster Time Synchronization Service started on host sph1erp.
2011-02-25 23:04:16.491
[oracle/11.2.0/grid/bin/orarootagent.bin(6423)]CRS-5818:Aborted command 'start for resource: ora.cluster_interconnect.haip 1 1' for resource 'ora.cluster_int
erconnect.haip'. Details at (:CRSAGF00113:) {0:0:178} in */oracle/11.2.0/grid/log/sph1erp/agent/ohasd/orarootagent_root/orarootagent_root.log.*
2011-02-25 23:04:20.521
[ohasd(5513)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:178} in
*/oracle/11.2.0/grid/log/sph1erp/ohasd/ohasd.log.*
Few lines from */oracle/11.2.0/grid/log/sph1erp/agent/ohasd/orarootagent_root/orarootagent_root.log.*
=====================================================================================================
2011-02-25 23:04:16.823: [ USRTHRD][16] {0:0:178} Starting Probe for ip 169.254.74.54
2011-02-25 23:04:16.823: [ USRTHRD][16] {0:0:178} Transitioning to Probe State
2011-02-25 23:04:17.177: [ USRTHRD][15] {0:0:178} [NetHAMain] thread stopping
2011-02-25 23:04:17.177: [ USRTHRD][15] {0:0:178} Thread:[NetHAMain]isRunning is reset to false here
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} Thread:[NetHAMain]stop }
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} thread cleaning up
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} pausing thread
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} posting thread
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop {
2011-02-25 23:04:17.645: [ USRTHRD][16] {0:0:178} [NetHAWork] thread stopping
2011-02-25 23:04:17.645: [ USRTHRD][16] {0:0:178} Thread:[NetHAWork]isRunning is reset to false here
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop }
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop {
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop }
2011-02-25 23:04:17.891: [ora.cluster_interconnect.haip][12] {0:0:178} [start] Start of HAIP aborted
2011-02-25 23:04:17.892: [ AGENT][12] {0:0:178} UserErrorException: Locale is
2011-02-25 23:04:17.893: [ora.cluster_interconnect.haip][12] {0:0:178} [start] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
Start action for HAIP aborted
2011-02-25 23:04:17.893: [ AGFW][12] {0:0:178} sending status msg [CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the foll
owing error:
Start action for HAIP aborted
] for start for resource: ora.cluster_interconnect.haip 1 1
2011-02-25 23:04:17.893: [ora.cluster_interconnect.haip][12] {0:0:178} [start] clsn_agent::start }
2011-02-25 23:04:17.894: [ AGFW][10] {0:0:178} Agent sending reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:661
2011-02-25 23:04:18.552: [ora.diskmon][12] {0:0:154} [check] DiskmonAgent::check {
2011-02-25 23:04:18.552: [ora.diskmon][12] {0:0:154} [check] DiskmonAgent::check } - 0
2011-02-25 23:04:19.573: [ AGFW][10] {0:0:154} Agent received the message: AGENT_HB[Engine] ID 12293:669
2011-02-25 23:04:20.510: [ora.cluster_interconnect.haip][18] {0:0:178} [start] got lock
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] tryActionLock }
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] abort }
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] clsn_agent::abort }
2011-02-25 23:04:20.511: [ AGFW][18] {0:0:178} Command: start for resource: ora.cluster_interconnect.haip 1 1 completed with status: TIMEDOUT
2011-02-25 23:04:20.512: [ora.cluster_interconnect.haip][8] {0:0:178} [check] NetworkAgent::init enter {
2011-02-25 23:04:20.513: [ora.cluster_interconnect.haip][8] {0:0:178} [check] NetworkAgent::init exit }
2011-02-25 23:04:20.517: [ AGFW][10] {0:0:178} Agent sending reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:661
2011-02-25 23:04:20.519: [ USRTHRD][8] {0:0:178} Ocr Context init default level 23886304
2011-02-25 23:04:20.519: [ default][8]clsvactversion:4: Retrieving Active Version from local storage.
[ CLWAL][8]clsw_Initialize: OLR initlevel [70000]
Few lines from */oracle/11.2.0/grid/log/sph1erp/ohasd/ohasd.log.*
=====================================================================================================
2011-02-25 23:04:21.627: [UiServer][30] {0:0:180} Done for ctx=6000000002604ce0
2011-02-25 23:04:21.642: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:26.139: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:26.139: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:26.973: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:26.973: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:26.992: [UiServer][30] {0:0:181} processMessage called
2011-02-25 23:04:26.993: [UiServer][30] {0:0:181} Sending message to PE. ctx= 6000000001b440f0
2011-02-25 23:04:26.993: [UiServer][30] {0:0:181} Sending command to PE: 67
2011-02-25 23:04:26.994: [ CRSPE][29] {0:0:181} Processing PE command id=173. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:26.997: [UiServer][30] {0:0:181} Done for ctx=6000000001b440f0
2011-02-25 23:04:27.012: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:31.135: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:31.135: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:32.318: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:32.318: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:32.332: [UiServer][30] {0:0:182} processMessage called
2011-02-25 23:04:32.333: [UiServer][30] {0:0:182} Sending message to PE. ctx= 6000000001b45ef0
2011-02-25 23:04:32.333: [UiServer][30] {0:0:182} Sending command to PE: 68
2011-02-25 23:04:32.334: [ CRSPE][29] {0:0:182} Processing PE command id=174. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:32.338: [UiServer][30] {0:0:182} Done for ctx=6000000001b45ef0
2011-02-25 23:04:32.352: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:36.155: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:36.155: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:37.683: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:37.683: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:37.702: [UiServer][30] {0:0:183} processMessage called
2011-02-25 23:04:37.703: [UiServer][30] {0:0:183} Sending message to PE. ctx= 6000000002604ce0
2011-02-25 23:04:37.703: [UiServer][30] {0:0:183} Sending command to PE: 69
2011-02-25 23:04:37.704: [ CRSPE][29] {0:0:183} Processing PE command id=175. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:37.708: [UiServer][30] {0:0:183} Done for ctx=6000000002604ce0
2011-02-25 23:04:37.722: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:41.156: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:41.156: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
What could be the issue ????
Experts Please help me. Doing setup for the PRoduction Env...
Do response ASAP...... Thanks
Regards,
ManishThanks Sebastian for your input.
yes. my lan2 is used for Cluster_interconnect which is having subnet 255.255.255.240.
Below are IPs used for RAC
Public
Node1: 10.10.1.173/255.255.240.0
Node2: 10.10.1.174/255.255.240.0
Private
Node1: 10.10.16.50/255.255.255.240
Node2: 10.10.16.51/255.255.255.240
Virtual
Node1: 10.10.1.191/255.255.240.0
Node2: 10.10.1.192/255.255.240.0
SCAN (Defined in DNS)
10.10.1.193/255.255.240.0
10.10.1.194/255.255.240.0
10.10.1.195/255.255.240.0
As you said, I will scrap GI Software again & will try with 255.255.255.0.
I Believe this Redundant Interconnect and ora.cluster_interconnect.haip present in 11.2.0.2.0 Version.
Oracle says:
Redundant Interconnect without any 3rd-party IP failover technology (bond, IPMP or similar) is supported natively by Grid Infrastructure starting from 11.2.0.2. Multiple private network adapters can be defined either during the installation phase or afterward using the oifcfg. Oracle Database, CSS, OCR, CRS, CTSS, and EVM components in 11.2.0.2 employ it automatically.
Grid Infrastructure can activate a maximum of four private network adapters at a time even if more are defined. The ora.cluster_interconnect.haip resource will start one to four link local HAIP on private network adapters for interconnect communication for Oracle RAC, Oracle ASM, and Oracle ACFS etc.
Grid automatically picks link local addresses from reserved 169.254.*.* subnet for HAIP, and it will not attempt to use any 169.254.*.* address if it's already in use for another purpose. With HAIP, by default, interconnect traffic will be load balanced across all active interconnect interfaces, and corresponding HAIP address will be failed over transparently to other adapters if one fails or becomes non-communicative. .
The number of HAIP addresses is decided by how many private network adapters are active when Grid comes up on the first node in the cluster . If there's only one active private network, Grid will create one; if two, Grid will create two; and if more than two, Grid will create four HAIPs. The number of HAIPs won't change even if more private network adapters are activated later, a restart of clusterware on all nodes is required for new adapters to become effective.
In my Setup, I am having Teaming for NIC's for Public & Private Interface. So I am thinking to break teaming of NICs because HAIP internally searching for next available NIC & not getting as all 4 are already in used with OS level NIC teaming.
My only Concern is, as I am going to change subnet for the Private IPs, should I change Private IP address ????
Thanks for the Support...
Regards,
Manish -
Failing all pre-requisite checks for 11.2.0.2.0 RAC on HP-UX Itanium 64
Hi Guys,
For Typical OR Advanced Grid Installation (GUI), all pre-requisites are failing on both nodes, like memory, swap, node reachability, groups, user & lot more.....
But when I try to check Pre-requisite from command line, everything seems to be ok.
This is what the output...
$ ./runcluvfy.sh stage -pre crsinst -n sph1erp,sph2erp -fixup -verbose
Performing pre-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "sph1erp"
Destination Node Reachable?
sph1erp yes
sph2erp yes
Result: Node reachability check passed from node "sph1erp"
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
sph2erp passed
sph1erp passed
Result: User equivalence check passed for user "oracle"
Checking node connectivity...
Checking hosts config file...
Node Name Status Comment
sph2erp passed
sph1erp passed
Verification of the hosts config file successful
Interface information for node "sph2erp"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
lan2 10.10.16.51 10.10.16.48 10.10.16.51 10.10.1.12 78:AC:C0:8A:07:76 1500
lan900 10.10.1.174 10.10.0.0 10.10.1.174 10.10.1.12 78:AC:C0:8A:07:6E 1500
Interface information for node "sph1erp"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
lan2 10.10.16.50 10.10.16.48 10.10.16.50 10.10.1.12 3C:4A:92:48:71:BE 1500
lan900 10.10.1.173 10.10.0.0 10.10.1.173 10.10.1.12 3C:4A:92:48:71:B6 1500
Check: Node connectivity of subnet "10.10.16.48"
Source Destination Connected?
sph2erp[10.10.16.51] sph1erp[10.10.16.50] yes
Result: Node connectivity passed for subnet "10.10.16.48" with node(s) sph2erp,sph1erp
Check: TCP connectivity of subnet "10.10.16.48"
Source Destination Connected?
sph1erp:10.10.16.50 sph2erp:10.10.16.51 passed
Result: TCP connectivity check passed for subnet "10.10.16.48"
Check: Node connectivity of subnet "10.10.0.0"
Source Destination Connected?
sph2erp[10.10.1.174] sph1erp[10.10.1.173] yes
Result: Node connectivity passed for subnet "10.10.0.0" with node(s) sph2erp,sph1erp
Check: TCP connectivity of subnet "10.10.0.0"
Source Destination Connected?
sph1erp:10.10.1.173 sph2erp:10.10.1.174 passed
Result: TCP connectivity check passed for subnet "10.10.0.0"
Interfaces found on subnet "10.10.0.0" that are likely candidates for VIP are:
sph2erp lan900:10.10.1.174
sph1erp lan900:10.10.1.173
Interfaces found on subnet "10.10.16.48" that are likely candidates for a private interconnect are:
sph2erp lan2:10.10.16.51
sph1erp lan2:10.10.16.50
Result: Node connectivity check passed
Check: Total memory
Node Name Available Required Comment
sph2erp 63.9GB (6.7004024E7KB) 4GB (4194304.0KB) passed
sph1erp 63.9GB (6.7004024E7KB) 4GB (4194304.0KB) passed
Result: Total memory check passed
Check: Available memory
Node Name Available Required Comment
sph2erp 53.3556GB (5.5947372E7KB) 50MB (51200.0KB) passed
sph1erp 52.2323GB (5.47695E7KB) 50MB (51200.0KB) passed
Result: Available memory check passed
Check: Swap space
Node Name Available Required Comment
sph2erp 188.7773GB (1.97947352E8KB) 4GB (4194304.0KB) passed
sph1erp 188.7773GB (1.97947352E8KB) 4GB (4194304.0KB) passed
Result: Swap space check passed
Check: Free disk space for "sph2erp:/var/tmp/"
Path Node Name Mount point Available Required Comment
/var/tmp/ sph2erp /var 13.8074GB 1GB passed
Result: Free disk space check passed for "sph2erp:/var/tmp/"
Check: Free disk space for "sph1erp:/var/tmp/"
Path Node Name Mount point Available Required Comment
/var/tmp/ sph1erp /var 13.8158GB 1GB passed
Result: Free disk space check passed for "sph1erp:/var/tmp/"
Check: User existence for "oracle"
Node Name Status Comment
sph2erp exists(199) passed
sph1erp exists(199) passed
Checking for multiple users with UID value 199
Result: Check for multiple users with UID value 199 passed
Result: User existence check passed for "oracle"
Check: Group existence for "oinstall"
Node Name Status Comment
sph2erp exists passed
sph1erp exists passed
Result: Group existence check passed for "oinstall"
Check: Group existence for "dba"
Node Name Status Comment
sph2erp exists passed
sph1erp exists passed
Result: Group existence check passed for "dba"
Check: Membership of user "oracle" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Comment
sph2erp yes yes yes yes passed
sph1erp yes yes yes yes passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed
Check: Membership of user "oracle" in group "dba"
Node Name User Exists Group Exists User in Group Comment
sph2erp yes yes yes passed
sph1erp yes yes yes passed
Result: Membership check for user "oracle" in group "dba" passed
Check: Run level
Node Name run level Required Comment
sph2erp 3 3 passed
sph1erp 3 3 passed
Result: Run level check passed
Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Comment
sph2erp hard 63488 63488 passed
sph1erp hard 63488 63488 passed
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Comment
sph2erp soft 1024 1024 passed
sph1erp soft 63488 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name Type Available Required Comment
sph2erp hard 3687 3686 passed
sph1erp hard 3687 3686 passed
Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes"
Node Name Type Available Required Comment
sph2erp soft 3687 2047 passed
sph1erp soft 3687 2047 passed
Result: Soft limits check passed for "maximum user processes"
Check: System architecture
Node Name Available Required Comment
sph2erp ia64 ia64 passed
sph1erp ia64 ia64 passed
Result: System architecture check passed
Check: Kernel version
Node Name Available Required Comment
sph2erp HP-UX B.11.31 B.11.31 passed
sph1erp HP-UX B.11.31 B.11.31 passed
Result: Kernel version check passed
Check: Kernel parameter for "ksi_alloc_max"
Node Name Configured Required Comment
sph2erp 32768 32768 passed
sph1erp 32768 32768 passed
Result: Kernel parameter check passed for "ksi_alloc_max"
Check: Kernel parameter for "executable_stack"
Node Name Configured Required Comment
sph2erp 0 0 passed
sph1erp 0 0 passed
Result: Kernel parameter check passed for "executable_stack"
Check: Kernel parameter for "max_thread_proc"
Node Name Configured Required Comment
sph2erp 1024 1024 passed
sph1erp 1024 1024 passed
Result: Kernel parameter check passed for "max_thread_proc"
Check: Kernel parameter for "maxdsiz"
Node Name Configured Required Comment
sph2erp 1073741824 1073741824 passed
sph1erp 1073741824 1073741824 passed
Result: Kernel parameter check passed for "maxdsiz"
Check: Kernel parameter for "maxdsiz_64bit"
Node Name Configured Required Comment
sph2erp 2147483648 2147483648 passed
sph1erp 2147483648 2147483648 passed
Result: Kernel parameter check passed for "maxdsiz_64bit"
Check: Kernel parameter for "maxssiz"
Node Name Configured Required Comment
sph2erp 134217728 134217728 passed
sph1erp 134217728 134217728 passed
Result: Kernel parameter check passed for "maxssiz"
Check: Kernel parameter for "maxssiz_64bit"
Node Name Configured Required Comment
sph2erp 1073741824 1073741824 passed
sph1erp 1073741824 1073741824 passed
Result: Kernel parameter check passed for "maxssiz_64bit"
Check: Kernel parameter for "maxuprc"
Node Name Configured Required Comment
sph2erp 3686 3686 passed
sph1erp 3686 3686 passed
Result: Kernel parameter check passed for "maxuprc"
Check: Kernel parameter for "msgmni"
Node Name Configured Required Comment
sph2erp 4096 4096 passed
sph1erp 4096 4096 passed
Result: Kernel parameter check passed for "msgmni"
Check: Kernel parameter for "msgtql"
Node Name Configured Required Comment
sph2erp 4096 4096 passed
sph1erp 4096 4096 passed
Result: Kernel parameter check passed for "msgtql"
Check: Kernel parameter for "ncsize"
Node Name Configured Required Comment
sph2erp 35840 35840 passed
sph1erp 35840 35840 passed
Result: Kernel parameter check passed for "ncsize"
Check: Kernel parameter for "shmmax"
Node Name Configured Required Comment
sph2erp 1073741824 1073741824 passed
sph1erp 1073741824 1073741824 passed
Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni"
Node Name Configured Required Comment
sph2erp 4096 4096 passed
sph1erp 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmseg"
Node Name Configured Required Comment
sph2erp 512 512 passed
sph1erp 512 512 passed
Result: Kernel parameter check passed for "shmseg"
Check: Kernel parameter for "tcp_smallest_anon_port"
Node Name Configured Required Comment
sph2erp 9000 9000 passed
sph1erp 9000 9000 passed
Result: Kernel parameter check passed for "tcp_smallest_anon_port"
Check: Kernel parameter for "tcp_largest_anon_port"
Node Name Configured Required Comment
sph2erp 65500 65500 passed
sph1erp 65500 65500 passed
Result: Kernel parameter check passed for "tcp_largest_anon_port"
Check: Kernel parameter for "udp_smallest_anon_port"
Node Name Configured Required Comment
sph2erp 9000 9000 passed
sph1erp 9000 9000 passed
Result: Kernel parameter check passed for "udp_smallest_anon_port"
Check: Kernel parameter for "udp_largest_anon_port"
Node Name Configured Required Comment
sph2erp 65500 65500 passed
sph1erp 65500 65500 passed
Result: Kernel parameter check passed for "udp_largest_anon_port"
Check: Package existence for "OS-Core-B.11.31( ia64)"
Node Name Available Required Comment
sph2erp OS-Core-B.11.31-0 OS-Core-B.11.31( ia64) passed
sph1erp OS-Core-B.11.31-0 OS-Core-B.11.31( ia64) passed
Result: Package existence check passed for "OS-Core-B.11.31( ia64)"
Check: Operating system patch for "Patch PHKL_38938"
Node Name Applied Required Comment
sph2erp Patch PHKL_39646 Patch PHKL_38938 passed
sph1erp Patch PHKL_39646 Patch PHKL_38938 passed
Result: Operating system patch check passed for "Patch PHKL_38938"
Check: Operating system patch for "Patch PHKL_39351"
Node Name Applied Required Comment
sph2erp Patch PHKL_40207 Patch PHKL_39351 passed
sph1erp Patch PHKL_40207 Patch PHKL_39351 passed
Result: Operating system patch check passed for "Patch PHKL_39351"
Check: Operating system patch for "Patch PHSS_36354"
Node Name Applied Required Comment
sph2erp Patch PHSS_40546 Patch PHSS_36354 passed
sph1erp Patch PHSS_40546 Patch PHSS_36354 passed
Result: Operating system patch check passed for "Patch PHSS_36354"
Check: Operating system patch for "Patch PHSS_37042"
Node Name Applied Required Comment
sph2erp Patch PHSS_37042 Patch PHSS_37042 passed
sph1erp Patch PHSS_37042 Patch PHSS_37042 passed
Result: Operating system patch check passed for "Patch PHSS_37042"
Check: Operating system patch for "Patch PHSS_37959"
Node Name Applied Required Comment
sph2erp Patch PHSS_40804 Patch PHSS_37959 passed
sph1erp Patch PHSS_40804 Patch PHSS_37959 passed
Result: Operating system patch check passed for "Patch PHSS_37959"
Check: Operating system patch for "Patch PHSS_39094"
Node Name Applied Required Comment
sph2erp Patch PHSS_40538 Patch PHSS_39094 passed
sph1erp Patch PHSS_40538 Patch PHSS_39094 passed
Result: Operating system patch check passed for "Patch PHSS_39094"
Check: Operating system patch for "Patch PHSS_39100"
Node Name Applied Required Comment
sph2erp Patch PHSS_40540 Patch PHSS_39100 passed
sph1erp Patch PHSS_40540 Patch PHSS_39100 passed
Result: Operating system patch check passed for "Patch PHSS_39100"
Check: Operating system patch for "Patch PHSS_39102"
Node Name Applied Required Comment
sph2erp Patch PHSS_41496 Patch PHSS_39102 passed
sph1erp Patch PHSS_41496 Patch PHSS_39102 passed
Result: Operating system patch check passed for "Patch PHSS_39102"
Check: Operating system patch for "Patch PHSS_38141"
Node Name Applied Required Comment
sph2erp Patch PHSS_38141 Patch PHSS_38141 passed
sph1erp Patch PHSS_38141 Patch PHSS_38141 passed
Result: Operating system patch check passed for "Patch PHSS_38141"
Check: Operating system patch for "Patch PHCO_40381"
Node Name Applied Required Comment
sph2erp Patch PHCO_40381 Patch PHCO_40381 passed
sph1erp Patch PHCO_40381 Patch PHCO_40381 passed
Result: Operating system patch check passed for "Patch PHCO_40381"
Check: Operating system patch for "Patch PHKL_38038"
Node Name Applied Required Comment
sph2erp Patch PHKL_41005 Patch PHKL_38038 passed
sph1erp Patch PHKL_41005 Patch PHKL_38038 passed
Result: Operating system patch check passed for "Patch PHKL_38038"
Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID
Result: Current group ID check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Checking Core file name pattern consistency...
Core file name pattern consistency check passed.
Checking to make sure user "oracle" is not in "root" group
Node Name Status Comment
sph2erp does not exist passed
sph1erp does not exist passed
Result: User "oracle" is not part of "root" group. Check passed
Check default user file creation mask
Node Name Available Required Comment
sph2erp 022 0022 passed
sph1erp 022 0022 passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes
Node Name Status
sph2erp passed
sph1erp passed
The DNS response time for an unreachable node is within acceptable limit on all nodes
File "/etc/resolv.conf" is consistent across nodes
Check: Time zone consistency
Result: Time zone consistency check passed
Checking settings of device file "/dev/async"
Node Name Available Comment
sph2erp yes failed (incorrect setting for minor number.)
sph1erp yes failed (incorrect setting for minor number.)
Result: Check for settings of device file "/dev/async" failed.
Starting check for The SSH LoginGraceTime setting ...
WARNING:
PRVE-0038 : The SSH LoginGraceTime setting on node "sph2erp" may result in users being disconnected before login is completed
PRVE-0038 : The SSH LoginGraceTime setting on node "sph1erp" may result in users being disconnected before login is completed
Check for The SSH LoginGraceTime setting passed
Fixup information has been generated for following node(s):
sph2erp,sph1erp
Please run the following script on each node as "root" user to execute the fixups:
'/tmp/CVU_11.2.0.2.0_oracle/runfixup.sh'
Pre-check for cluster services setup was unsuccessful on all the nodes.
What could be the issue ????
Any help would be appreciated...
Regards,
ManishStarted with Grid Installation. Copying Software to remote node & linking libraries were successfully without any issue (upto 76%). But got issue while executing root.sh on Node1
sph1erp:/oracle/11.2.0/grid #sh root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'sph1erp'
CRS-2676: Start of 'ora.mdnsd' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'sph1erp'
CRS-2676: Start of 'ora.gpnpd' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'sph1erp'
CRS-2672: Attempting to start 'ora.gipcd' on 'sph1erp'
CRS-2676: Start of 'ora.gipcd' on 'sph1erp' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'sph1erp' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'sph1erp'
CRS-2672: Attempting to start 'ora.diskmon' on 'sph1erp'
CRS-2676: Start of 'ora.diskmon' on 'sph1erp' succeeded
CRS-2676: Start of 'ora.cssd' on 'sph1erp' succeeded
ASM created and started successfully.
Disk Group OCRVOTE created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'sys'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk ab847ed2b4f04f2dbfb875226d2bb194.
Successful addition of voting disk 85c05a5b30384f8dbff48cc069de7a7c.
Successful addition of voting disk 649196fbdd614f9cbf26a9a0e6670a6e.
Successful addition of voting disk 8815dfcee2e64f64bf00b9c76626ab41.
Successful addition of voting disk 8ce55fe5534f4f77bfa9f54187592707.
Successfully replaced voting disk group with +OCRVOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE ab847ed2b4f04f2dbfb875226d2bb194 (/dev/oracle/ocrvote1) [OCRVOTE]
2. ONLINE 85c05a5b30384f8dbff48cc069de7a7c (/dev/oracle/ocrvote2) [OCRVOTE]
3. ONLINE 649196fbdd614f9cbf26a9a0e6670a6e (/dev/oracle/ocrvote3) [OCRVOTE]
4. ONLINE 8815dfcee2e64f64bf00b9c76626ab41 (/dev/oracle/ocrvote4) [OCRVOTE]
5. ONLINE 8ce55fe5534f4f77bfa9f54187592707 (/dev/oracle/ocrvote5) [OCRVOTE]
Located 5 voting disk(s).
Start of resource "ora.cluster_interconnect.haip" failed
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'sph1erp'
CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
Start action for HAIP aborted
CRS-2674: Start of 'ora.cluster_interconnect.haip' on 'sph1erp' failed
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'sph1erp'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'sph1erp' succeeded
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Clusterware stack
Failed to start High Availability IP at /oracle/11.2.0/grid/crs/install/crsconfig_lib.pm line 1046.
/oracle/11.2.0/grid/perl/bin/perl -I/oracle/11.2.0/grid/perl/lib -I/oracle/11.2.0/grid/crs/install /oracle/11.2.0/grid/crs/install/rootcrs.pl execution failed
sph1erp:/oracle/11.2.0/grid #
Last few lines from CRS Log for node 1, where error came
[ctssd(6467)]CRS-2401:The Cluster Time Synchronization Service started on host sph1erp.
2011-02-25 23:04:16.491
[oracle/11.2.0/grid/bin/orarootagent.bin(6423)]CRS-5818:Aborted command 'start for resource: ora.cluster_interconnect.haip 1 1' for resource 'ora.cluster_int
erconnect.haip'. Details at (:CRSAGF00113:) {0:0:178} in */oracle/11.2.0/grid/log/sph1erp/agent/ohasd/orarootagent_root/orarootagent_root.log.*
2011-02-25 23:04:20.521
[ohasd(5513)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:178} in
*/oracle/11.2.0/grid/log/sph1erp/ohasd/ohasd.log.*
Few lines from */oracle/11.2.0/grid/log/sph1erp/agent/ohasd/orarootagent_root/orarootagent_root.log.*
=====================================================================================================
2011-02-25 23:04:16.823: [ USRTHRD][16] {0:0:178} Starting Probe for ip 169.254.74.54
2011-02-25 23:04:16.823: [ USRTHRD][16] {0:0:178} Transitioning to Probe State
2011-02-25 23:04:17.177: [ USRTHRD][15] {0:0:178} [NetHAMain] thread stopping
2011-02-25 23:04:17.177: [ USRTHRD][15] {0:0:178} Thread:[NetHAMain]isRunning is reset to false here
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} Thread:[NetHAMain]stop }
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} thread cleaning up
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} pausing thread
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} posting thread
2011-02-25 23:04:17.178: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop {
2011-02-25 23:04:17.645: [ USRTHRD][16] {0:0:178} [NetHAWork] thread stopping
2011-02-25 23:04:17.645: [ USRTHRD][16] {0:0:178} Thread:[NetHAWork]isRunning is reset to false here
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop }
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop {
2011-02-25 23:04:17.645: [ USRTHRD][12] {0:0:178} Thread:[NetHAWork]stop }
2011-02-25 23:04:17.891: [ora.cluster_interconnect.haip][12] {0:0:178} [start] Start of HAIP aborted
2011-02-25 23:04:17.892: [ AGENT][12] {0:0:178} UserErrorException: Locale is
2011-02-25 23:04:17.893: [ora.cluster_interconnect.haip][12] {0:0:178} [start] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
Start action for HAIP aborted
2011-02-25 23:04:17.893: [ AGFW][12] {0:0:178} sending status msg [CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the foll
owing error:
Start action for HAIP aborted
] for start for resource: ora.cluster_interconnect.haip 1 1
2011-02-25 23:04:17.893: [ora.cluster_interconnect.haip][12] {0:0:178} [start] clsn_agent::start }
2011-02-25 23:04:17.894: [ AGFW][10] {0:0:178} Agent sending reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:661
2011-02-25 23:04:18.552: [ora.diskmon][12] {0:0:154} [check] DiskmonAgent::check {
2011-02-25 23:04:18.552: [ora.diskmon][12] {0:0:154} [check] DiskmonAgent::check } - 0
2011-02-25 23:04:19.573: [ AGFW][10] {0:0:154} Agent received the message: AGENT_HB[Engine] ID 12293:669
2011-02-25 23:04:20.510: [ora.cluster_interconnect.haip][18] {0:0:178} [start] got lock
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] tryActionLock }
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] abort }
2011-02-25 23:04:20.511: [ora.cluster_interconnect.haip][18] {0:0:178} [start] clsn_agent::abort }
2011-02-25 23:04:20.511: [ AGFW][18] {0:0:178} Command: start for resource: ora.cluster_interconnect.haip 1 1 completed with status: TIMEDOUT
2011-02-25 23:04:20.512: [ora.cluster_interconnect.haip][8] {0:0:178} [check] NetworkAgent::init enter {
2011-02-25 23:04:20.513: [ora.cluster_interconnect.haip][8] {0:0:178} [check] NetworkAgent::init exit }
2011-02-25 23:04:20.517: [ AGFW][10] {0:0:178} Agent sending reply for: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:661
2011-02-25 23:04:20.519: [ USRTHRD][8] {0:0:178} Ocr Context init default level 23886304
2011-02-25 23:04:20.519: [ default][8]clsvactversion:4: Retrieving Active Version from local storage.
[ CLWAL][8]clsw_Initialize: OLR initlevel [70000]
Few lines from */oracle/11.2.0/grid/log/sph1erp/ohasd/ohasd.log.*
=====================================================================================================
2011-02-25 23:04:21.627: [UiServer][30] {0:0:180} Done for ctx=6000000002604ce0
2011-02-25 23:04:21.642: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:26.139: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:26.139: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:26.973: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:26.973: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:26.992: [UiServer][30] {0:0:181} processMessage called
2011-02-25 23:04:26.993: [UiServer][30] {0:0:181} Sending message to PE. ctx= 6000000001b440f0
2011-02-25 23:04:26.993: [UiServer][30] {0:0:181} Sending command to PE: 67
2011-02-25 23:04:26.994: [ CRSPE][29] {0:0:181} Processing PE command id=173. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:26.997: [UiServer][30] {0:0:181} Done for ctx=6000000001b440f0
2011-02-25 23:04:27.012: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:31.135: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:31.135: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:32.318: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:32.318: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:32.332: [UiServer][30] {0:0:182} processMessage called
2011-02-25 23:04:32.333: [UiServer][30] {0:0:182} Sending message to PE. ctx= 6000000001b45ef0
2011-02-25 23:04:32.333: [UiServer][30] {0:0:182} Sending command to PE: 68
2011-02-25 23:04:32.334: [ CRSPE][29] {0:0:182} Processing PE command id=174. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:32.338: [UiServer][30] {0:0:182} Done for ctx=6000000001b45ef0
2011-02-25 23:04:32.352: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:36.155: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:36.155: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
2011-02-25 23:04:37.683: [UiServer][31] CS(60000000014b0790)set Properties ( root,60000000012e0260)
2011-02-25 23:04:37.683: [UiServer][31] SS(6000000001372270)Accepted client connection: saddr =(ADDRESS=(PROTOCOL=ipc)(DEV=92)(KEY=OHASD_UI_SOCKET))daddr = (A
DDRESS=(PROTOCOL=ipc)(KEY=OHASD_UI_SOCKET))
2011-02-25 23:04:37.702: [UiServer][30] {0:0:183} processMessage called
2011-02-25 23:04:37.703: [UiServer][30] {0:0:183} Sending message to PE. ctx= 6000000002604ce0
2011-02-25 23:04:37.703: [UiServer][30] {0:0:183} Sending command to PE: 69
2011-02-25 23:04:37.704: [ CRSPE][29] {0:0:183} Processing PE command id=175. Description: [Stat Resource : 600000000135f760]
2011-02-25 23:04:37.708: [UiServer][30] {0:0:183} Done for ctx=6000000002604ce0
2011-02-25 23:04:37.722: [UiServer][31] Closed: remote end failed/disc.
2011-02-25 23:04:41.156: [ CLSINET][33]Returning NETDATA: 1 interfaces
2011-02-25 23:04:41.156: [ CLSINET][33]# 0 Interface 'lan2',ip='10.10.16.50',mac='3c-4a-92-48-71-be',mask='255.255.255.240',net='10.10.16.48',use='cluster_int
erconnect'
What could be the issue ????
Experts Please help me. Doing setup for the PRoduction Env...
Do response ASAP...... Thanks
Regards,
Manish -
Error in executing root.sh - due to network failure
Hi,
My Grid Installation were almost complete. I am already executing root.sh for the first node but unfortunately the network seems to get hang based on the error I encountered while tailing the +/app/oraInventory/logs/installActions2010-07-31_09-38-00AM.log+ from node1
Below is the part that encountered error.
INFO: InstallProgressMonitor: Starting phase 512
INFO: Saving Cluster Inventory
INFO: Saving Cluster Inventory
INFO: Running command '/tmp/OraInstall2010-07-31_09-38-00AM/oui/bin/runInstaller -jreLoc /tmp/OraInstall2010-07-31_09-38-00AM/jdk/jre -paramFile /tmp/OraInstall2010-07-31_09-38-00AM/oui/clusterparam.ini -silent -ignoreSysPrereqs -attachHome -noClusterEnabled ORACLE_HOME=/app/11.2.0/grid ORACLE_HOME_NAME=Ora11g_gridinfrahome1 CLUSTER_NODES=racnode01,racnode02 "INVENTORY_LOCATION=/app/oraInventory" LOCAL_NODE=racnode02 -remoteInvocation -invokingNodeName racnode01 -logFilePath "/app/oraInventory/logs" -timestamp 2010-07-31_09-38-00AM' on the nodes 'racnode02'.
INFO: Invoking OUI on cluster nodes racnode02
INFO: /tmp/OraInstall2010-07-31_09-38-00AM/oui/bin/runInstaller -jreLoc /tmp/OraInstall2010-07-31_09-38-00AM/jdk/jre -paramFile /tmp/OraInstall2010-07-31_09-38-00AM/oui/clusterparam.ini -silent -ignoreSysPrereqs -attachHome -noClusterEnabled ORACLE_HOME=/app/11.2.0/grid ORACLE_HOME_NAME=Ora11g_gridinfrahome1 CLUSTER_NODES=racnode01,racnode02 "INVENTORY_LOCATION=/app/oraInventory" LOCAL_NODE=racnode02 -remoteInvocation -invokingNodeName racnode01 -logFilePath "/app/oraInventory/logs" -timestamp 2010-07-31_09-38-00AM
INFO: InstallProgressMonitor: Completed phase 512
INFO: Checkpoint:Failed Checkpoint found returning it for getAllFailedCheckPoints.
INFO: Checkpoint:Failed Checkpoint found returning null for getLastFailedCheckPoint.
INFO: Checkpoint:Setting status in index file as passed,as no last failed checkpoint is available.
INFO: Checkpoint:Index file written and updated
*INFO: Checkpoint:OUI-35006: Error in copying file '/app/grid/checkpoints/oui/checkpoint_Ora11g_gridinfrahome1.xml' present inside directory '/app/grid/checkpoints/oui' on nodes ''. [PRKC-1062 : Node list is null]*
*INFO: Checkpoint:OUI-35006: Error in copying file '/app/grid/checkpoints/oui/index.xml' present inside directory '/app/grid/checkpoints/oui' on nodes ''. [PRKC-1062 : Node list is null]*
*INFO: Checkpoint:Error in copying file '/app/grid/checkpoints/oui/index.xml' present inside directory '/app/grid/checkpoints/oui' on nodes ''. [PRKC-1062 : Node list is null]*
INFO: Checkpoint:Transfer of file done from remote node
INFO: Adding iterator oracle.sysman.oii.oiif.oiifw.OiifwRootShWCDE
INFO: Updating the global context
INFO: Path To 'globalcontext.xml' = /app/11.2.0/grid/install/chainedInstall/globalcontext
INFO: Since operation was successful, move the current OiicAPISessionDetails to installed list
INFO: Number of root scripts to be executed = 2
INFO: isSuccessfullInstallation: true
INFO: isSuccessfullRemoteInstallation: true
INFO: Adding ExitStatus SUCCESS to the exit status set
INFO: Completed setting up crssetup
INFO: Number of root scripts to be executed = 2
SEVERE: occured inside verify methodPRCI-1108 : Failed to check CRS running state for CRS home /app/11.2.0/grid on node racnode02
PRCT-1003 : Failed to run "crsctl" on node "racnode02"
INFO: Shutting down OUISetupDriver.JobExecutorThread
INFO: Cleaning up, please wait...
INFO: Dispose the install area control object
INFO: Update the state machine to STATE_CLEAN
INFO: All forked task are completed at state setup
INFO: Completed background operations
i tried to restart or even reload the service network but connections failed and the only way to fix this is to restart both nodes. no problem with private connection because they can ping each other. I know the only last part in installation is running the root.sh script. And I know I am almost done.
Note: I am trying to create virtual RAC servers using Virtualbox.
Can someone help to fix the problem with running root.sh scripts?
I tried to research related to my issue but it seems I did not find similar situation, I saw something about deconfigure root.sh but I am still reviewing it. Any help really appreciated.Thanks Belman for reply.
I already tried the deconfigure of root.sh script, then after successful deconfiguration for both nodes, I tried to run the root.sh again, but the last part which updating the oraInventory failed, I am almost there.... (quite frustrated) :) but still smiling, this is the last part ... still have issues :)
[root@racnode01 grid]# /app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /app/11.2.0/grid
Enter the full pathname of the local bin directory: [usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-08-01 06:24:56: Parsing the host name
2010-08-01 06:24:56: Checking for super user privileges
2010-08-01 06:24:56: User has super user privileges
Using configuration parameter file: /app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode01'
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode01'
CRS-2676: Start of 'ora.gipcd' on 'racnode01' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode01'
CRS-2676: Start of 'ora.gpnpd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode01'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode01'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode01'
CRS-2676: Start of 'ora.diskmon' on 'racnode01' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode01'
CRS-2676: Start of 'ora.ctssd' on 'racnode01' succeeded
ASM created and started successfully.
DiskGroup CRS created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'racnode01'
CRS-2676: Start of 'ora.crsd' on 'racnode01' succeeded
Successful addition of voting disk 54643cab96eb4ff2bf66885ad582b6fb.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
1. ONLINE 54643cab96eb4ff2bf66885ad582b6fb (ORCL:CRS) [CRS]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode01'
CRS-2677: Stop of 'ora.crsd' on 'racnode01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode01'
CRS-2677: Stop of 'ora.asm' on 'racnode01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode01'
CRS-2677: Stop of 'ora.ctssd' on 'racnode01' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode01'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode01'
CRS-2677: Stop of 'ora.cssd' on 'racnode01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode01'
CRS-2677: Stop of 'ora.gpnpd' on 'racnode01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode01'
CRS-2677: Stop of 'ora.gipcd' on 'racnode01' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode01'
CRS-2677: Stop of 'ora.mdnsd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode01'
CRS-2676: Start of 'ora.mdnsd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode01'
CRS-2676: Start of 'ora.gipcd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode01'
CRS-2676: Start of 'ora.gpnpd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode01'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode01'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode01'
CRS-2676: Start of 'ora.diskmon' on 'racnode01' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode01'
CRS-2676: Start of 'ora.ctssd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode01'
CRS-2676: Start of 'ora.asm' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racnode01'
CRS-2676: Start of 'ora.crsd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'racnode01'
CRS-2676: Start of 'ora.evmd' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode01'
CRS-2676: Start of 'ora.asm' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.CRS.dg' on 'racnode01'
CRS-2676: Start of 'ora.CRS.dg' on 'racnode01' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'racnode01'
CRS-5014: Agent "/app/11.2.0/grid/bin/orarootagent.bin" timed out starting process "/app/11.2.0/grid/bin/acfsregistrymount" for action "start": details at "(:CLSN00009:)" in "/app/11.2.0/grid/log/racnode01/agent/crsd/orarootagent_root/orarootagent_root.log"
(:CLSN00009:)Utils:execCmd aborted
CRS-2676: Start of 'ora.registry.acfs' on 'racnode01' succeeded
racnode01 2010/08/01 06:49:31 /app/11.2.0/grid/cdata/racnode01/backup_20100801_064931.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
You do not have sufficient permissions to access the inventory '/app/oraInventory/y'. Installation cannot continue. Make sure that you have read/write permissions to the inventory directory and restart the installer.: Permission denied+
I did check the directory
+[root@racnode01 ~]# ls -lrta /app/oraInventory/y+
total 28
-rwxr-xr-x 1 grid root 5036 Jul 31 11:56 oraenv
-rwxr-xr-x 1 grid root 2415 Jul 31 11:56 dbhome
-rwxr-xr-x 1 grid root 4143 Jul 31 11:56 coraenv
drwxrwx--- 7 grid oinstall 4096 Jul 31 11:56 ..
drwxr-xr-x 2 root root 4096 Jul 31 11:56 .
I am not sure how will I deal with this one, if I missed something. If someone out there familiar with this, any help really appreciated. I am thinking if this is due to my previous run of root.sh (but not really sure)? -
Root.sh failed at second node OUL 6.3 Oracle GRID 11.2.0.3
Hi, im installing a two node cluster mounted on Oracle Linux 6.3 with Oracle DB 11.2.0.3, the installation went smooth up until the execution of the root.sh script on the second node.
THe script return this final lines:
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Start of resource "ora.crsd" failed
CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'nodo2'
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Grid Infrastructure stack
Failed to start Cluster Ready Services at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1286.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
In $GRID_HOME/log/node2/alertnode.log It appears to be a Cluster Time Synchronization Service issue, (i didn't synchronyze the nodes..) however the CTSS is running in observer mode, wich i believe it shouldn't affect the installation process. After that i lost it...there's an entry CRS-5018 indicating that an unused HAIP route was removed... and then, out of the blue: CRS-5818:Aborted command 'start' for resource 'ora.asm'. Some clarification will be deeply apreciated.
Here's the complete log:
2013-04-01 13:39:35.358
[client(12163)]CRS-2101:The OLR was formatted using version 3.
2013-04-01 19:40:19.597
[ohasd(12338)]CRS-2112:The OLR service started on node nodo2.
2013-04-01 19:40:19.657
[ohasd(12338)]CRS-1301:Oracle High Availability Service started on node nodo2.
[client(12526)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
[client(12528)]CRS-10001:01-Apr-13 13:41 ACFS-9201: Not Supported
[client(12603)]CRS-10001:01-Apr-13 13:41 ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.39-400.17.2.el6uek.i686'
2013-04-01 19:41:17.509
[ohasd(12338)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2013-04-01 19:41:17.618
[gpnpd(12695)]CRS-2328:GPNPD started on node nodo2.
2013-04-01 19:41:21.363
[cssd(12755)]CRS-1713:CSSD daemon is started in exclusive mode
2013-04-01 19:41:23.194
[ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2013-04-01 19:41:56.144
[cssd(12755)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
2013-04-01 19:41:57.545
[cssd(12755)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
[cssd(12755)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node nodo1 and is terminating; details at (:CSSNM00006:) in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log
2013-04-01 19:41:58.549
[ohasd(12338)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'nodo2'.
2013-04-01 19:42:10.025
[gpnpd(12695)]CRS-2329:GPNPD on node nodo2 shutdown.
2013-04-01 19:42:11.407
[mdnsd(12685)]CRS-5602:mDNS service stopping by request.
2013-04-01 19:42:29.642
[gpnpd(12947)]CRS-2328:GPNPD started on node nodo2.
2013-04-01 19:42:33.241
[cssd(13012)]CRS-1713:CSSD daemon is started in clustered mode
2013-04-01 19:42:35.104
[ohasd(12338)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2013-04-01 19:42:44.065
[cssd(13012)]CRS-1707:Lease acquisition for node nodo2 number 2 completed
2013-04-01 19:42:45.484
[cssd(13012)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/ASM_DISK_1; details in /u01/app/11.2.0/grid/log/nodo2/cssd/ocssd.log.
2013-04-01 19:42:52.138
[cssd(13012)]CRS-1601:CSSD Reconfiguration complete. Active nodes are nodo1 nodo2 .
2013-04-01 19:42:55.081
[ctssd(13076)]CRS-2403:The Cluster Time Synchronization Service on host nodo2 is in observer mode.
2013-04-01 19:42:55.581
[ctssd(13076)]CRS-2401:The Cluster Time Synchronization Service started on host nodo2.
2013-04-01 19:42:55.581
[ctssd(13076)]CRS-2407:The new Cluster Time Synchronization Service reference node is host nodo1.
2013-04-01 19:43:08.875
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 19:43:08.876
[ctssd(13076)]CRS-2409:The clock on host nodo2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2013-04-01 19:43:13.565
[u01/app/11.2.0/grid/bin/orarootagent.bin(13064)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.0.0 / 255.255.0.0 / 0.0.0.0 / eth0
2013-04-01 19:53:09.800
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5818:Aborted command 'start' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log.
2013-04-01 19:53:11.827
[ohasd(12338)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) {0:0:223} in /u01/app/11.2.0/grid/log/nodo2/ohasd/ohasd.log.
2013-04-01 19:53:12.779
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:53:13.892
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:53:43.877
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:54:13.891
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:54:43.906
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:55:13.914
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:55:43.918
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:56:13.922
[u01/app/11.2.0/grid/bin/oraagent.bin(12922)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/nodo2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2013-04-01 19:56:53.209
[crsd(13741)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:07:01.128
[crsd(13741)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:07:01.278
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:07:08.689
[crsd(15248)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:13:10.138
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 20:17:13.024
[crsd(15248)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:17:13.171
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:17:20.826
[crsd(16746)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:27:25.020
[crsd(16746)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:27:25.176
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:27:31.591
[crsd(18266)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:37:35.668
[crsd(18266)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:37:35.808
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:37:43.209
[crsd(19762)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:43:11.160
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 20:47:47.487
[crsd(19762)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:47:47.637
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:47:55.086
[crsd(21242)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 20:57:59.343
[crsd(21242)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 20:57:59.492
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 20:58:06.996
[crsd(22744)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:08:11.046
[crsd(22744)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:08:11.192
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:08:18.726
[crsd(24260)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:13:12.000
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 21:18:22.262
[crsd(24260)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:18:22.411
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:18:29.927
[crsd(25759)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:28:34.467
[crsd(25759)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:28:34.616
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:28:41.990
[crsd(27291)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:38:45.012
[crsd(27291)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:38:45.160
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:38:52.790
[crsd(28784)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:43:12.378
[ctssd(13076)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/nodo2/ctssd/octssd.log.
2013-04-01 21:48:56.285
[crsd(28784)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:48:56.435
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:49:04.421
[crsd(30272)]CRS-1012:The OCR service started on node nodo2.
2013-04-01 21:59:08.183
[crsd(30272)]CRS-0810:Cluster Ready Service aborted due to failure to communicate with Event Management Service with error [1]. Details at (:CRSD00120:) in /u01/app/11.2.0/grid/log/nodo2/crsd/crsd.log.
2013-04-01 21:59:08.318
[ohasd(12338)]CRS-2765:Resource 'ora.crsd' has failed on server 'nodo2'.
2013-04-01 21:59:15.860
[crsd(31772)]CRS-1012:The OCR service started on node nodo2.Hi santysharma, thanks for the reply, i have two ethernet interfaces: eth0 (public network 192.168.1.0) and eth1 (private network 10.5.3.0), there is no device using that ip range, here's the output of route command:
(Sorry for the alignment, i tried to tab it but the editor trims it again)
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
private * 255.255.255.0 U 0 0 0 eth1
link-local * 255.255.0.0 U 1002 0 0 eth0
link-local * 255.255.0.0 U 1003 0 0 eth1
public * 255.255.255.0 U 0 0 0 eth0
And the /etc/hosts file
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.5.3.1 nodo1.cluster nodo1
10.5.3.2 nodo2.cluster nodo2
192.168.1.13 cluster-scan
192.168.1.14 nodo1-vip
192.168.1.15 nodo2-vip
And the ifconfig -a
eth0 Link encap:Ethernet HWaddr C8:3A:35:D9:C6:2B
inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::ca3a:35ff:fed9:c62b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:34708 errors:0 dropped:18 overruns:0 frame:0
TX packets:24693 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:48545969 (46.2 MiB) TX bytes:1994381 (1.9 MiB)
eth1 Link encap:Ethernet HWaddr 00:0D:87:D0:A3:8E
inet addr:10.5.3.2 Bcast:10.5.3.255 Mask:255.255.255.0
inet6 addr: fe80::20d:87ff:fed0:a38e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:5344 (5.2 KiB)
Interrupt:23 Base address:0x6000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:20 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1320 (1.2 KiB) TX bytes:1320 (1.2 KiB)
Now that i'm thinking i've read somewhere that ipv6 was no supported...yet there's no relation with the 169.254.x.x ip range. -
Potential memory leak in 11.2.0.1.0 cluster stack components
Hi All,
We are running 11.2.0.1.0 RAC on OEL 5.5 and OEL 5.5. On all of our boxes, we have observed a gradual increase and filling of swap space. Swap usage slowly but steadily increase to a point where either the Node boots automatically or we have to manually restart it.
To get more insight into the issues, we recently upgraded one of the nodes to Oracle unbreakable kernel and installed smem to profile the memory consumption pattern. smem will give the process set size (RSS), the unique set size (USS) and the proportional set size (PSS) which is the unique set size plus a portion of the shared memory that is being used by this process.
It appears that there is a gradual increase in memory/swap consumption of some clusterware components. Also the weird part is that the process is filling up the swap space and not utilizing the unallocated RAM.
Memory footprint at the Node start...
PID User Command Swap USS PSS RSS
4001 root /u01/app/11.2.0/grid/bin/ohasd.bin 16236 18128 18829 30284
5200 oracle /u01/app/11.2.0/grid/bin/oraagent.bin 13536 74512 75587 87156
4255 grid /u01/app/11.2.0/grid/bin/oraagent.bin 13484 2644 3628 15056
4688 root /u01/app/11.2.0/grid/bin/crsd.bin 8240 71584 72628 85692
Memory footprint after 24 hrs...
PID User Command Swap USS PSS RSS
5200 oracle /u01/app/11.2.0/grid/bin/oraagent.bin 94952 121168 122161 133488
4688 root /u01/app/11.2.0/grid/bin/crsd.bin 66220 76684 77723 90776
4001 root /u01/app/11.2.0/grid/bin/ohasd.bin 21448 24708 25410 36892
4255 grid /u01/app/11.2.0/grid/bin/oraagent.bin 13840 2372 3202 14316
#free -m
total used free shared buffers cached
Mem: 3964 3856 108 0 5 1846
-/+ buffers/cache: 2004 1959
Swap: 4094 617 3477
Has anyone experience similar situation? I did google search as well as Metalink, but did not find anything useful.
Any thoughts/suggestions are welcome.
Thanks,
-Sanjeev
Edited by: user12219014 on Jan 9, 2011 5:58 AMThanks for pointing to MOS notes, they were quite helpful. Though sometime on our system, ohasd.bin consumes more resources. Is it safe to kill it?
Also, we have observed that there are multiple oraagents belonging to different users such as root,grid and oracle.
grid 14620 1 0 20:32 ? 00:00:14 /u01/app/11.2.0/grid/bin/oraagent.bin
root 14625 1 0 20:32 ? 00:00:02 /u01/app/11.2.0/grid/bin/orarootagent.bin
root 14627 1 0 20:32 ? 00:00:00 /u01/app/11.2.0/grid/bin/cssdagent
grid 14803 1 0 20:32 ? 00:00:06 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle 14807 1 0 20:32 ? 00:01:53 /u01/app/11.2.0/grid/bin/oraagent.bin
root 14811 1 0 20:32 ? 00:00:38 /u01/app/11.2.0/grid/bin/orarootagent.bin
When these are killed, not all are re-spawned automatically - typically oraagent belonging to "oracle" user is left out. Is this an expected behaviour or it will cause some instability in the clusterware?
Thanks -
11gR2 Cluster node does not start, start of ora.gipcd fails
Hi
I have a clusert node which just does not start, and i do know understand what the problem is.
Does anyone have an idea what the problem cound be?
I am using Oracle 11.2.0.1.0 on OEL.
I appreciate any help, thank you.
Regards Bernhard
NAME TARGET STATE SERVER STATE_DETAILS
Cluster Resources
ora.asm
1 ONLINE OFFLINE
ora.crsd
1 ONLINE INTERMEDIATE node001
ora.cssd
1 ONLINE OFFLINE
ora.cssdmonitor
1 ONLINE OFFLINE
ora.ctssd
1 ONLINE OFFLINE
ora.diskmon
1 ONLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE node001
ora.evmd
1 ONLINE OFFLINE
ora.gipcd
1 ONLINE OFFLINE
ora.gpnpd
1 ONLINE OFFLINE
ora.mdnsd
1 ONLINE ONLINE node001 alertnode001.log
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2012-06-29 08:43:52.511
[/opt/grid/product/11gR2/grid/bin/cssdmonitor(5144)]CRS-5822:Agent '/opt/grid/product/11gR2/grid/bin/cssdmonitor_root' disconnected from server. Details at (:CRSAGF00117:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/oracssdmonitor_root.log.
2012-06-29 08:48:49.713
[ohasd(4346)]CRS-2112:The OLR service started on node node001.
2012-06-29 08:48:50.010
[ohasd(4346)]CRS-8017:location: /etc/oracle/lastgasp has 50 reboot advisory log files, 0 were announced and 0 errors occurred
2012-06-29 08:48:55.134
[ohasd(4346)]CRS-2772:Server 'node001' has been assigned to pool 'Free'.
[client(4634)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(4652)]CRS-10001:ACFS-9322: done.
2012-06-29 08:50:59.362
[ohasd(4346)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.oracssdmonitor_root.log
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2012-06-29 08:48:51.003: [ AGFW][1810312288] Starting the agent: /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdmonitor_root/
2012-06-29 08:48:51.003: [ AGENT][1810312288] Agent framework initialized, Process Id = 4414
2012-06-29 08:48:51.003: [ USRTHRD][1810312288] to enter agent main
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: setting priority to 4
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] *** Agent Framework Started ***
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_vmondisv: Compatible vendor clusterware not in use
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_thrdspawn: spawn OMON succ
2012-06-29 08:48:54.939: [ USRTHRD][1810312288] clsncssd_main: spawn omon succ
2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: default values for omon, initrate 1000, pollrate 500.
2012-06-29 08:48:54.939: [ USRTHRD][1098914112] clsnomon_main: starting omon
2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:54.942: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
2012-06-29 08:48:55.041: [ AGFW][1810312288] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Inited lsf context 0x1ae1cdd0
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] Initing CLS Framework messaging
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] New Framework state: 2
2012-06-29 08:48:55.041: [CLSFRAME][1810312288] M2M is starting...
2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] m_pClscCtx=0x1ae21d70m_pUgblm=0x1ae754b0
2012-06-29 08:48:55.047: [ CRSCOMM][1810312288] Starting send thread
2012-06-29 08:48:55.052: [ CRSCOMM][1119893824] clsIpc: sendWork thread started.
2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] IPC Client thread started listening
2012-06-29 08:48:55.057: [ CRSCOMM][1130383680] init data sent from server
2012-06-29 08:48:55.057: [CLSFRAME][1810312288] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:node001
2012-06-29 08:48:55.058: [CLSFRAME][1810312288] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:node001
2012-06-29 08:48:55.066: [CLSFRAME][1810312288] Starting thread model named: MultiThread
2012-06-29 08:48:55.073: [CLSFRAME][1810312288] Starting thread model named: SingleThread
2012-06-29 08:48:55.076: [CLSFRAME][1810312288] Starting thread model named: SingleThreadT
2012-06-29 08:48:55.080: [CLSFRAME][1810312288] New Framework state: 3
2012-06-29 08:48:55.080: [ AGFW][1810312288] Agent Framework started successfully
2012-06-29 08:48:55.080: [ AGFW][1182832960] Agfw engine module has enabled...
2012-06-29 08:48:55.080: [CLSFRAME][1182832960] Module Enabling is complete
2012-06-29 08:48:55.080: [CLSFRAME][1182832960] New Framework state: 6
2012-06-29 08:48:55.080: [ AGFW][1182832960] Agent is started with userid: root , expected user: root
2012-06-29 08:48:55.081: [ AGFW][1182832960] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-06-29 08:48:55.088: [ AGFW][1182832960] Agent received the message: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
2012-06-29 08:48:55.089: [ AGFW][1182832960] Added new restype: ora.cssdmonitor.type
2012-06-29 08:48:55.089: [ AGFW][1182832960] Agent sending last reply for: RESTYPE_ADD[ora.cssdmonitor.type] ID 8196:299
2012-06-29 08:48:55.090: [ AGFW][1182832960] Agent received the message: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
2012-06-29 08:48:55.090: [ AGFW][1182832960] Added new resource: ora.cssdmonitor 1 1 to the agfw
2012-06-29 08:48:55.091: [ AGFW][1182832960] Agent sending last reply for: RESOURCE_ADD[ora.cssdmonitor 1 1] ID 4356:301
2012-06-29 08:48:55.091: [ AGFW][1182832960] Agent received the message: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
2012-06-29 08:48:55.091: [ AGFW][1182832960] Preparing CHECK command for: ora.cssdmonitor 1 1
2012-06-29 08:48:55.096: [ AGFW][1172343104] Executing command: check for resource: ora.cssdmonitor 1 1
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_INITRATE, value 1000
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr OMON_POLLRATE, value 500
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr ORA_OPROCD_MODE, value
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr PROCD_TIMEOUT, value 1000
2012-06-29 08:48:55.096: [ora.cssdmonitor][1172343104] [check] clsncssd_getattr: attr LOGGING_LEVEL, value 1
2012-06-29 08:48:55.096: [ AGFW][1172343104] check for resource: ora.cssdmonitor 1 1 completed with status: OFFLINE
2012-06-29 08:48:55.096: [CRSTIMER][1203812672] Timer Thread Starting.
2012-06-29 08:48:55.096: [ AGFW][1182832960] ora.cssdmonitor 1 1 state changed from: UNKNOWN to: OFFLINE
2012-06-29 08:48:55.096: [ AGFW][1182832960] Agent sending last reply for: RESOURCE_PROBE[ora.cssdmonitor 1 1] ID 4097:302
2012-06-29 08:48:55.096: [ AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
2012-06-29 08:48:55.096: [ AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:28
2012-06-29 08:48:55.100: [ AGFW][1182832960] Agent received the message: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
2012-06-29 08:48:55.100: [ AGFW][1182832960] Agent sending last reply for: RESOURCE_DELETE[ora.cssdmonitor 1 1] ID 4358:317
2012-06-29 08:48:55.100: [ AGFW][1182832960] Agent has no resources to be monitored.Sending suicide request.
2012-06-29 08:48:55.100: [ AGFW][1182832960] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:34
2012-06-29 08:48:55.101: [ AGFW][1182832960] ora.cssdmonitor 1 1 marked as deleted.
2012-06-29 08:48:55.101: [ AGFW][1182832960] Deleting the resource: ora.cssdmonitor 1 1
2012-06-29 08:48:55.101: [ AGFW][1182832960] Agent is commiting suicide.
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: CSSD Agent was asked to exit with exit code 1
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_becomeactive: Signaling active state
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_term: still threads up (1)
2012-06-29 08:48:55.101: [ USRTHRD][1182832960] clsncssd_exit: Agent is done.
2012-06-29 08:48:55.101: [ AGFW][1182832960] Agent is exiting with exit code: 1
2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:55.947: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29
2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clssscConnect: gipc request failed with 29 (0x16)
2012-06-29 08:48:56.952: [ CSSCLNT][1098914112]clsssInitNative: connect failed, rc 29Edited by: Bernhard W on 03-Jul-2012 02:14I am really getting desperate about this.
The ASM persmissions are on both nodes the same, so this should not be the problem and i can not find any problem on the network.
The log files of the healthy node do not say anything about the node which does not startup now, the last log entry was from 6 months ago, when i shut that node down. There seems to be no communication between the two nodes now.
I tried to manually start the ora.cssd but somehow ora.gipcd just does not start and i can not find the problem.
[oracle@dmrs11srvcla001 ~]$ crsctl stat res -t -init
NAME TARGET STATE SERVER STATE_DETAILS
Cluster Resources
ora.asm
1 ONLINE OFFLINE
ora.crsd
1 ONLINE INTERMEDIATE dmrs11srvcla001
ora.cssd
1 ONLINE OFFLINE
ora.cssdmonitor
1 ONLINE ONLINE dmrs11srvcla001
ora.ctssd
1 ONLINE OFFLINE
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE dmrs11srvcla001
ora.evmd
1 ONLINE OFFLINE
ora.gipcd
1 ONLINE OFFLINE
ora.gpnpd
1 ONLINE OFFLINE
ora.mdnsd
1 ONLINE ONLINE dmrs11srvcla001
[oracle@dmrs11srvcla001 ~]$ crsctl start res ora.cssd -init
CRS-2672: Attempting to start 'ora.gipcd' on 'dmrs11srvcla001'
Start action for daemon aborted
CRS-2674: Start of 'ora.gipcd' on 'dmrs11srvcla001' failed
CRS-2679: Attempting to clean 'ora.gipcd' on 'dmrs11srvcla001'
CRS-2681: Clean of 'ora.gipcd' on 'dmrs11srvcla001' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dmrs11srvcla001'
Start action for daemon aborted
CRS-2674: Start of 'ora.gpnpd' on 'dmrs11srvcla001' failed
CRS-2679: Attempting to clean 'ora.gpnpd' on 'dmrs11srvcla001'
CRS-2681: Clean of 'ora.gpnpd' on 'dmrs11srvcla001' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dmrs11srvcla001'
CRS-2672: Attempting to start 'ora.diskmon' on 'dmrs11srvcla001'
CRS-2676: Start of 'ora.diskmon' on 'dmrs11srvcla001' succeededalertnode001.log
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:11:40.789
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:11:41.654
[ohasd(4351)]CRS-2765:Resource 'ora.mdnsd' has failed on server 'node001'.
2012-07-02 17:11:43.404
[cssd(4997)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-02 17:11:46.634
[cssd(4997)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-02 17:11:46.745
[ohasd(4351)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'node001'.
2012-07-02 17:12:42.384
[/opt/grid/product/11gR2/grid/bin/orarootagent.bin(4488)]CRS-5818:Aborted command 'start for resource: ora.diskmon 1 1' for resource 'ora.diskmon'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/orarootagent_root/orarootagent_root.log.
2012-07-02 17:12:46.390
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.diskmon'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:14:50.768
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:16:55.819
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:16:58.354
[cssd(5137)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-02 17:16:58.425
[cssd(5137)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-02 17:25:23.194
[ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
2012-07-02 17:25:23.200
[ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
2012-07-02 17:26:57.384
[/opt/grid/product/11gR2/grid/bin/cssdagent(5116)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-07-02 17:27:02.514
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-02 17:37:11.971
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.evmd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 02:13:57.821
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 05:14:37.915
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 05:44:44.630
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 06:14:51.303
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 06:44:57.981
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:15:04.655
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:42:45.015
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-03 07:44:50.068
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.
2012-07-03 07:44:52.369
[cssd(10209)]CRS-1713:CSSD daemon is started in clustered mode
2012-07-03 07:44:52.434
[cssd(10209)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2012-07-03 07:45:11.331
[ohasd(4351)]CRS-2765:Resource 'ora.crsd' has failed on server 'node001'.
2012-07-03 07:53:17.222
[ohasd(4351)]CRS-2765:Resource 'ora.diskmon' has failed on server 'node001'.
2012-07-03 07:53:17.222
[ohasd(4351)]CRS-2767:Target resource 'ora.diskmon' is offline, will not recover.
2012-07-03 07:54:51.401
[/opt/grid/product/11gR2/grid/bin/cssdagent(10188)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) in /opt/grid/product/11gR2/grid/log/node001/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2012-07-03 07:54:56.524
[ohasd(4351)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00111:) in /opt/grid/product/11gR2/grid/log/node001/ohasd/ohasd.log.ohasd.log
2012-07-03 07:45:11.330: [ AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:24919
2012-07-03 07:45:11.330: [ AGFW][1347250496] Received state change for ora.crsd 1 1 [old state = PARTIAL, new state = OFFLINE]
2012-07-03 07:45:11.330: [ AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23258]
2012-07-03 07:45:11.330: [ AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:24919
2012-07-03 07:45:11.330: [ CRSPE][1357756736] State change received from node001 for ora.crsd 1 1
2012-07-03 07:45:11.331: [ CRSPE][1357756736] Processing PE command id=139. Description: [Resource State Change (ora.crsd 1 1) : 0x1c284d40]
2012-07-03 07:45:11.331: [ CRSPE][1357756736] RI [ora.crsd 1 1] new external state [OFFLINE] old value: [INTERMEDIATE] on node001 label = []
2012-07-03 07:45:11.331: [ CRSPE][1357756736] Resource Resource Instance ID[ora.crsd 1 1]. Values:
STATE=OFFLINE
TARGET=ONLINE
LAST_SERVER=node001
CURRENT_RCOUNT=1
LAST_RESTART=1341292505
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=29
STATE_CHANGE_VERS=29
LAST_FAULT=1341292504
DEGREE_ID=1
ID=ora.crsd 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:none
ReadLocks:|STATE INITED||INITIAL CHECK DONE| has failed!
2012-07-03 07:45:11.331: [ CRSPE][1357756736] Processing unplanned state change for [ora.crsd 1 1]
2012-07-03 07:45:11.331: [ CRSPE][1357756736] ora.crsd 1 1: uptime exceeds uptime threshold , resetting restart count
2012-07-03 07:45:11.331: [ CRSPE][1357756736] Local Recovery picked for [ora.crsd 1 1]
2012-07-03 07:45:11.333: [ CRSPE][1357756736] Sending message to agfw: id = 23260
2012-07-03 07:45:11.333: [ AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:11.333: [ CRSPE][1357756736] CRS-2672: Attempting to start 'ora.crsd' on 'node001'
2012-07-03 07:45:11.333: [ AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23260 to the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.349: [ AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.350: [ AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:12.350: [ CRSPE][1357756736] Received reply to action [Start] message ID: 23260
2012-07-03 07:45:12.350: [ AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.crsd 1 1] ID 4098:23261 from the agent /opt/grid/product/11gR2/grid/bin/orarootagent_root
2012-07-03 07:45:12.350: [ AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.crsd 1 1] ID 4098:23260
2012-07-03 07:45:12.350: [ CRSPE][1357756736] Received reply to action [Start] message ID: 23260
2012-07-03 07:45:12.351: [ CRSPE][1357756736] RI [ora.crsd 1 1] new external state [INTERMEDIATE] old value: [OFFLINE] on node001 label = []
2012-07-03 07:45:12.351: [ CRSPE][1357756736] CRS-2676: Start of 'ora.crsd' on 'node001' succeeded
2012-07-03 07:45:12.351: [ CRSPE][1357756736] PE Command [ Resource State Change (ora.crsd 1 1) : 0x1c284d40 ] has completed
2012-07-03 07:45:12.351: [ AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23269
2012-07-03 07:45:12.351: [ AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23269
2012-07-03 07:45:12.351: [ AGFW][1347250496] Agfw received reply from PE for resource state change for ora.crsd 1 1
2012-07-03 07:53:17.220: [ AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:25353
2012-07-03 07:53:17.221: [ AGFW][1347250496] Received state change for ora.diskmon 1 1 [old state = ONLINE, new state = PLANNED_OFFLINE]
2012-07-03 07:53:17.221: [ AGFW][1347250496] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:23497]
2012-07-03 07:53:17.221: [ AGFW][1347250496] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:25353
2012-07-03 07:53:17.221: [ CRSPE][1357756736] State change received from node001 for ora.diskmon 1 1
2012-07-03 07:53:17.221: [ CRSPE][1357756736] Processing PE command id=140. Description: [Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0]
2012-07-03 07:53:17.221: [ CRSPE][1357756736] RI [ora.diskmon 1 1] new external state [OFFLINE] old value: [ONLINE] on node001 label = []
2012-07-03 07:53:17.221: [ CRSPE][1357756736] RI [ora.diskmon 1 1] new target state: [OFFLINE] old value: [ONLINE]
2012-07-03 07:53:17.222: [ CRSOCR][1351452992] Multi Write Batch processing...
2012-07-03 07:53:17.222: [ CRSPE][1357756736] Resource Resource Instance ID[ora.diskmon 1 1]. Values:
STATE=OFFLINE
TARGET=OFFLINE
LAST_SERVER=node001
CURRENT_RCOUNT=0
LAST_RESTART=1341294297
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=
INCARNATION=2
STATE_CHANGE_VERS=5
LAST_FAULT=1341242723
DEGREE_ID=1
ID=ora.diskmon 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:START of [ora.cssd 1 1] on [node001] : 0x2aaab000ba10
ReadLocks:|STATE INITED| has failed!
2012-07-03 07:53:17.222: [ CRSPE][1357756736] Processing unplanned state change for [ora.diskmon 1 1]
2012-07-03 07:53:17.222: [ CRSPE][1357756736] Target is not ONLINE, not recovering [ora.diskmon 1 1]
2012-07-03 07:53:17.222: [ CRSPE][1357756736] PE Command [ Resource State Change (ora.diskmon 1 1) : 0x2aaab00318a0 ] has completed
2012-07-03 07:53:17.222: [ AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23501
2012-07-03 07:53:17.222: [ AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23501
2012-07-03 07:53:17.223: [ AGFW][1347250496] Agfw received reply from PE for resource state change for ora.diskmon 1 1
2012-07-03 07:53:17.223: [ CRSOCR][1351452992] Multi Write Batch done.
2012-07-03 07:54:56.523: [ AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.523: [ AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
2012-07-03 07:54:56.524: [ CRSPE][1357756736] Received reply to action [Start] message ID: 23220
2012-07-03 07:54:56.524: [ AGFW][1347250496] Received the reply to the message: RESOURCE_START[ora.cssd 1 1] ID 4098:23242 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.524: [ CRSPE][1357756736]Created alert : (:CRSPE00111:) : Start action timed out!
2012-07-03 07:54:56.524: [ CRSPE][1357756736] Start action failed with error code: 3
2012-07-03 07:54:56.524: [ AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.cssd 1 1] ID 4098:23220
2012-07-03 07:54:56.524: [ CRSPE][1357756736] Received reply to action [Start] message ID: 23220
2012-07-03 07:54:56.524: [ AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:54:56.524: [ AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:54:56.524: [ AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:54:56.524: [ CRSPE][1357756736] CRS-2674: Start of 'ora.cssd' on 'node001' failed
2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2674: Start of 'ora.cssd' on 'node001' failed]
MSGTYPE:
TextMessage[1]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
2012-07-03 07:54:56.525: [ CRSPE][1357756736] Sending message to agfw: id = 23555
2012-07-03 07:54:56.525: [ CRSPE][1357756736] CRS-2679: Attempting to clean 'ora.cssd' on 'node001'
2012-07-03 07:54:56.525: [ AGFW][1347250496] Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:54:56.525: [ AGFW][1347250496] Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is being stopped. Will be restarted when agent stops completly
2012-07-03 07:54:56.525: [ AGFW][1347250496] Could not forward message [RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555] to agent. /opt/grid/product/11gR2/grid/bin/cssdagent_root is not running
2012-07-03 07:54:56.525: [ AGFW][1347250496] Can not start Agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id root is waiting to be restarted.
2012-07-03 07:54:56.525: [UiServer][1361959232] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2679: Attempting to clean 'ora.cssd' on 'node001']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752] Client disconnected.
2012-07-03 07:55:01.627: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 14
2012-07-03 07:55:01.628: [ CRSCOMM][1340946752] IPC listener connection to member 14 has been removed
2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:14|Type:3}
2012-07-03 07:55:01.628: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:14|Type:3}
2012-07-03 07:55:01.628: [ CRSPE][1357756736] Disconnected from server:
2012-07-03 07:55:01.628: [ AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
2012-07-03 07:55:01.628: [ AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
2012-07-03 07:55:01.628: [ AGFW][1347250496] Restarting the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.628: [ AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10188] stopped!
2012-07-03 07:55:01.628: [ CRSCOMM][1347250496] removeConnection: Member 14 does not exist.
2012-07-03 07:55:01.628: [ AGFW][1347250496] Starting the agent: /opt/grid/product/11gR2/grid/bin/cssdagent with user id: root and incarnation:7
2012-07-03 07:55:01.632: [ AGFW][1347250496] Starting the HB [Interval = 30000, misscount = 10kill allowed=0] for agent: /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New IPC Member:{Relative|Node:0|Process:15|Type:3}:AGENT
2012-07-03 07:55:01.819: [CLSFRAME][1340946752] New process connected to us ID:{Relative|Node:0|Process:15|Type:3} Info:AGENT
2012-07-03 07:55:01.837: [ AGFW][1347250496] Agfw Proxy Server received the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-07-03 07:55:01.837: [ AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root with pid:10530 connected to server.
2012-07-03 07:55:01.837: [ AGFW][1347250496] Agfw Proxy Server sending message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.838: [ AGFW][1347250496] Agfw Proxy Server sending message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.838: [ AGFW][1347250496] Agfw Proxy Server forwarding the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555 to the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.839: [ AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_HANDSHAKE[Proxy] ID 20484:14
2012-07-03 07:55:01.851: [ AGFW][1347250496] Received the reply to the message: RESTYPE_ADD[ora.cssd.type] ID 8196:23591 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.854: [ AGFW][1347250496] Received the reply to the message: RESOURCE_ADD[ora.cssd 1 1] ID 4356:23593 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.861: [ AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.862: [ AGFW][1347250496] Agfw Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:55:01.862: [ AGFW][1347250496] Received the reply to the message: RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23594 from the agent /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.862: [ CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
2012-07-03 07:55:01.862: [ AGFW][1347250496] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.cssd 1 1] ID 4100:23555
2012-07-03 07:55:01.863: [ AGFW][1347250496] Agfw Proxy Server received the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:55:01.863: [ CRSPE][1357756736] Received reply to action [Clean] message ID: 23555
2012-07-03 07:55:01.863: [ AGFW][1347250496] Suicide request received from /opt/grid/product/11gR2/grid/bin/cssdagent_root
2012-07-03 07:55:01.863: [ AGFW][1347250496] Agfw Proxy Server replying to the message: AGENT_SUICIDE[Proxy] ID 20486:34
2012-07-03 07:55:01.863: [ CRSPE][1357756736] CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded
2012-07-03 07:55:01.863: [ CRSPE][1357756736] Sequencer for [ora.cssd 1 1] has completed with error: CRS-0215: Could not start resource 'ora.cssd'.
2012-07-03 07:55:01.863: [UiServer][1361959232] Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2681: Clean of 'ora.cssd' on 'node001' succeeded]
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.cssd]
WAIT:
TextMessage[0]
2012-07-03 07:55:01.864: [ CRSPE][1357756736] PE Command [ Start Resource : 0x2aaab0009d00 ] has completed
2012-07-03 07:55:01.864: [ AGFW][1347250496] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:23606
2012-07-03 07:55:01.864: [ CRSPE][1357756736] UI Command [Start Resource : 0x2aaab0009d00] is replying to sender.
2012-07-03 07:55:01.864: [ AGFW][1347250496] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:23606
2012-07-03 07:55:01.864: [UiServer][1361959232] Container [ Name: UI_DATA
ora.cssd:
TextMessage[215]
2012-07-03 07:55:01.864: [UiServer][1361959232] Done for ctx=0x2aaab004ece0
2012-07-03 07:55:01.869: [UiServer][1364060480] Closed: remote end failed/disc.
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Couldnt clscreceive message, no message: 11
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] Client disconnected.
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752][FFAIL] Listener got clsc error 11 for memNum. 15
2012-07-03 07:55:03.791: [ CRSCOMM][1340946752] IPC listener connection to member 15 has been removed
2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Removing IPC Member:{Relative|Node:0|Process:15|Type:3}
2012-07-03 07:55:03.791: [CLSFRAME][1340946752] Disconnected from AGENT process: {Relative|Node:0|Process:15|Type:3}
2012-07-03 07:55:03.792: [ CRSPE][1357756736] Disconnected from server:
2012-07-03 07:55:03.792: [ AGFW][1347250496] Agfw Proxy Server received process disconnected notification, count=1
2012-07-03 07:55:03.792: [ AGFW][1347250496] /opt/grid/product/11gR2/grid/bin/cssdagent_root disconnected.
2012-07-03 07:55:03.792: [ AGFW][1347250496] Agent /opt/grid/product/11gR2/grid/bin/cssdagent_root[10530] stopped!
2012-07-03 07:55:03.792: [ CRSCOMM][1347250496] removeConnection: Member 15 does not exist. -
Why root.sh fails in the second node?
Hi
After successful install of oracle11 grid in 2 nodes and running root.sh on
node1, root.sh on node2 fails:
[root@vmorarac2 dev]#
/u01/app/product/11.2.0/oracle/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/product/11.2.0/oracle
Enter the full pathname of the local bin directory:
[/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed
by
Database Configuration Assistant when a database is
created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2013-07-17 08:37:10: Parsing the host name
2013-07-17 08:37:10: Checking for super user
privileges
2013-07-17 08:37:10: User has super user privileges
Using configuration parameter file:
/u01/app/product/11.2.0/oracle/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been
started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but
found an active CSS daemon on node vmorarac1, number 1, and is terminating
CRS-2673: Attempting to stop 'ora.cssdmonitor' on
'vmorarac2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'vmorarac2'
succeeded
An active cluster was found during exclusive startup,
restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on
'vmorarac2'
CRS-2676: Start of 'ora.mdnsd' on 'vmorarac2'
succeeded
CRS-2672: Attempting to start 'ora.gipcd' on
'vmorarac2'
CRS-2676: Start of 'ora.gipcd' on 'vmorarac2'
succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on
'vmorarac2'
CRS-2676: Start of 'ora.gpnpd' on 'vmorarac2'
succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on
'vmorarac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'vmorarac2'
succeeded
CRS-2672: Attempting to start 'ora.cssd' on
'vmorarac2'
CRS-2672: Attempting to start 'ora.diskmon' on
'vmorarac2'
CRS-2676: Start of 'ora.diskmon' on 'vmorarac2'
succeeded
CRS-2674: Start of 'ora.cssd' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.cssd' on
'vmorarac2'
CRS-2681: Clean of 'ora.cssd' on 'vmorarac2' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on
'vmorarac2'
CRS-2677: Stop of 'ora.diskmon' on 'vmorarac2'
succeeded
CRS-4000: Command Start failed, or completed with
errors.
CRS-2672: Attempting to start 'ora.cssd' on
'vmorarac2'
CRS-2672: Attempting to start 'ora.diskmon' on
'vmorarac2'
CRS-2674: Start of 'ora.diskmon' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.diskmon' on
'vmorarac2'
CRS-5016: Process
"/u01/app/product/11.2.0/oracle/bin/diskmon" spawned by agent
"/u01/app/product/11.2.0/oracle/bin/orarootagent.bin" for action "clean" failed:
details at "(:CLSN00010:)" in
"/u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/orarootagent_root.log"
CRS-2681: Clean of 'ora.diskmon' on 'vmorarac2'
succeeded
CRS-2674: Start of 'ora.cssd' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.cssd' on
'vmorarac2'
CRS-2681: Clean of 'ora.cssd' on 'vmorarac2' succeeded
CRS-4000: Command Start failed, or completed with
errors.
Command return code of 1 (256) from command:
/u01/app/product/11.2.0/oracle/bin/crsctl start resource ora.ctssd -init -env
USR_ORA_ENV=CTSS_REBOOT=TRUE
Start of resource "ora.ctssd -init -env
USR_ORA_ENV=CTSS_REBOOT=TRUE" failed
Failed to start CTSS
Failed to start Oracle Clusterware stack
[root@vmorarac2 dev]#
[root@vmorarac2 dev]#
try again:
[root@vmorarac2 bin]# ./crsctl start resource ora.ctssd
-init -env USR_ORA_ENV=CTSS_REBOOT=TRUE
CRS-2672: Attempting to start 'ora.cssd' on
'vmorarac2'
CRS-2672: Attempting to start 'ora.diskmon' on
'vmorarac2'
CRS-2674: Start of 'ora.diskmon' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.diskmon' on
'vmorarac2'
CRS-5016: Process
"/u01/app/product/11.2.0/oracle/bin/diskmon" spawned by agent
"/u01/app/product/11.2.0/oracle/bin/orarootagent.bin" for action "clean" failed:
details at "(:CLSN00010:)" in
"/u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/orarootagent_root.log"
CRS-2681: Clean of 'ora.diskmon' on 'vmorarac2'
succeeded
CRS-2674: Start of 'ora.cssd' on 'vmorarac2' failed
CRS-2679: Attempting to clean 'ora.cssd' on
'vmorarac2'
CRS-2681: Clean of 'ora.cssd' on 'vmorarac2' succeeded
CRS-4000: Command Start failed, or completed with
errors.
[root@vmorarac2 bin]# ps -ef | grep u01
root 8913 1 0 08:37 ? 00:00:06
/u01/app/product/11.2.0/oracle/bin/ohasd.bin reboot
oracle 10507 1 0 08:39 ? 00:00:02
/u01/app/product/11.2.0/oracle/bin/oraagent.bin
oracle 10522 1 0 08:39 ? 00:00:00
/u01/app/product/11.2.0/oracle/bin/mdnsd.bin
oracle 10534 1 0 08:39 ? 00:00:00
/u01/app/product/11.2.0/oracle/bin/gipcd.bin
oracle 10548 1 0 08:39 ? 00:00:39
/u01/app/product/11.2.0/oracle/bin/gpnpd.bin
root 11723 1 0 11:00 ? 00:00:03
/u01/app/product/11.2.0/oracle/bin/cssdmonitor
[oracle@vmorarac2 bin]$ ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready
Services
CRS-4530: Communications failure contacting Cluster
Synchronization Services daemon
CRS-4534: Cannot communicate with Event Managerlog:
Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2013-07-17 08:38:04.485: [ AGFW][3481860944] Starting the agent: /u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/
2013-07-17 08:38:04.485: [ AGENT][3481860944] Agent framework initialized, Process Id = 10319
2013-07-17 08:38:04.487: [ USRTHRD][3481860944] Utils::getCrsHome crsHome /u01/app/product/11.2.0/oracle
2013-07-17 08:38:04.487: [ USRTHRD][3481860944] Process::convertPidToString pid = 10319
2013-07-17 08:38:04.488: [ AGFW][3481860944] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
2013-07-17 08:38:04.488: [CLSFRAME][3481860944] Inited lsf context 0x317e9e0
2013-07-17 08:38:04.488: [CLSFRAME][3481860944] Initing CLS Framework messaging
2013-07-17 08:38:04.488: [CLSFRAME][3481860944] New Framework state: 2
2013-07-17 08:38:04.488: [CLSFRAME][3481860944] M2M is starting...
2013-07-17 08:38:04.490: [ CRSCOMM][3481860944] m_pClscCtx=0x31d1bd0m_pUgblm=0x31d5720
2013-07-17 08:38:04.490: [ CRSCOMM][3481860944] Starting send thread
2013-07-17 08:38:04.490: [ CRSCOMM][1119435072] clsIpc: sendWork thread started.
2013-07-17 08:38:04.491: [ CRSCOMM][1129924928] IPC Client thread started listening
2013-07-17 08:38:04.491: [ CRSCOMM][1129924928] init data sent from server
2013-07-17 08:38:04.491: [CLSFRAME][3481860944] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:vmorarac2
2013-07-17 08:38:04.491: [CLSFRAME][3481860944] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:vmorarac2
2013-07-17 08:38:04.492: [CLSFRAME][3481860944] Starting thread model named: MultiThread
2013-07-17 08:38:04.492: [CLSFRAME][3481860944] Starting thread model named: SingleThread
2013-07-17 08:38:04.492: [CLSFRAME][3481860944] Starting thread model named: SingleThreadT
2013-07-17 08:38:04.492: [CLSFRAME][3481860944] New Framework state: 3
2013-07-17 08:38:04.493: [ AGFW][3481860944] Agent Framework started successfully
2013-07-17 08:38:04.493: [ AGFW][1182374208] Agfw engine module has enabled...
2013-07-17 08:38:04.493: [CLSFRAME][1182374208] Module Enabling is complete
2013-07-17 08:38:04.493: [CLSFRAME][1182374208] New Framework state: 6
2013-07-17 08:38:04.493: [ AGFW][1182374208] Agent is started with userid: root , expected user: root
2013-07-17 08:38:04.493: [ AGFW][1182374208] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
2013-07-17 08:38:04.505: [ AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.crs.type] ID 8196:358
2013-07-17 08:38:04.506: [ AGFW][1182374208] Added new restype: ora.crs.type
2013-07-17 08:38:04.506: [ AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.crs.type] ID 8196:358
2013-07-17 08:38:04.506: [ AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.ctss.type] ID 8196:360
2013-07-17 08:38:04.506: [ AGFW][1182374208] Added new restype: ora.ctss.type
2013-07-17 08:38:04.507: [ AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.ctss.type] ID 8196:360
2013-07-17 08:38:04.516: [ AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.diskmon.type] ID 8196:362
2013-07-17 08:38:04.516: [ AGFW][1182374208] Added new restype: ora.diskmon.type
2013-07-17 08:38:04.517: [ AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.diskmon.type] ID 8196:362
2013-07-17 08:38:04.519: [ AGFW][1182374208] Agent received the message: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:364
2013-07-17 08:38:04.520: [ AGFW][1182374208] Added new restype: ora.drivers.acfs.type
2013-07-17 08:38:04.520: [ AGFW][1182374208] Agent sending last reply for: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:364
2013-07-17 08:38:04.521: [ AGFW][1182374208] Agent received the message: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:366
2013-07-17 08:38:04.521: [ AGFW][1182374208] Added new resource: ora.diskmon 1 1 to the agfw
2013-07-17 08:38:04.522: [ AGFW][1182374208] Agent sending last reply for: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:366
2013-07-17 08:38:04.522: [ AGFW][1182374208] Agent received the message: RESOURCE_START[ora.diskmon 1 1] ID 4098:367
2013-07-17 08:38:04.522: [ AGFW][1182374208] Preparing START command for: ora.diskmon 1 1
2013-07-17 08:38:04.522: [ AGFW][1182374208] ora.diskmon 1 1 state changed from: UNKNOWN to: STARTING
2013-07-17 08:38:04.526: [ AGFW][1161394496] Executing command: start for resource: ora.diskmon 1 1
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] clsn_agent::start {
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DaemonAgent{
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] }DaemonAgent
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DiskmonAgent::DiskmonAgent {
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] InitAttrs {
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] __IS_HASD_AGENT=TRUE
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] }InitAttrs
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DiskmonAgent::DiskmonAgent }
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] DiskmonAgent::start {
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Arg Value = -d
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Arg Value = -f
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Total Count of Environment Variables = 3
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Adding Environment Variables _ORA_AGENT_ACTION=TRUE
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Adding Environment Variables __IS_HASD_AGENT=
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Adding Environment variable from USR_ORA_ENV ORACLE_USER=oracle
2013-07-17 08:38:04.527: [ora.diskmon][1161394496] [start] Utils:execCmd action = 1 flags = 5 ohome = (null) cmdname = diskmon.
2013-07-17 08:38:04.528: [ora.diskmon][1161394496] [start] getOracleHomeAttrib: oracle_home = /u01/app/product/11.2.0/oracle
2013-07-17 08:38:04.528: [ora.diskmon][1161394496] [start] Utils:execCmd Running the binary from /u01/app/product/11.2.0/oracle/bin/diskmon
2013-07-17 08:38:04.531: [CRSTIMER][1091324224] Timer Thread Starting.
2013-07-17 08:38:04.533: [ora.diskmon][1161394496] [start] execCmd ret = 0
2013-07-17 08:38:04.533: [ora.diskmon][1161394496] [start] }DaemonAgent::start
2013-07-17 08:38:10.534: [ora.diskmon][1161394496] [start] DiskmonAgent::connect {
2013-07-17 08:38:10.534: [ora.diskmon][1161394496] [start] Process::convertPidToString pid = 10319
2013-07-17 08:38:10.535: [ora.diskmon][1161394496] [start] DiskmonAgent::connect }
2013-07-17 08:38:10.535: [ora.diskmon][1161394496] [start] DiskmonAgent::start }
2013-07-17 08:38:10.535: [ora.diskmon][1161394496] [start] clsn_agent::start }
2013-07-17 08:38:10.535: [ AGFW][1161394496] Command: start for resource: ora.diskmon 1 1 completed with status: SUCCESS
2013-07-17 08:38:10.535: [ AGFW][1182374208] Agent sending reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:367
2013-07-17 08:38:10.537: [ AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:38:10.538: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
2013-07-17 08:38:10.538: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 0
2013-07-17 08:38:10.538: [ AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:38:10.538: [ AGFW][1182374208] ora.diskmon 1 1 state changed from: STARTING to: ONLINE
2013-07-17 08:38:10.538: [ AGFW][1182374208] Started implicit monitor for:ora.diskmon 1 1
2013-07-17 08:38:10.538: [ AGFW][1182374208] Agent sending last reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:367
2013-07-17 08:38:30.543: [ AGFW][1182374208] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:38:30.544: [ AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:38:30.544: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
2013-07-17 08:38:30.544: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 0
2013-07-17 08:38:30.545: [ AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:38:50.550: [ AGFW][1182374208] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:38:50.551: [ AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:38:50.551: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
2013-07-17 08:38:50.551: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 0
2013-07-17 08:38:50.551: [ AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:39:02.782: [ AGFW][1182374208] Agent received the message: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:547
2013-07-17 08:39:02.782: [ AGFW][1182374208] Preparing STOP command for: ora.diskmon 1 1
2013-07-17 08:39:02.782: [ AGFW][1182374208] ora.diskmon 1 1 state changed from: ONLINE to: STOPPING
2013-07-17 08:39:02.783: [ AGFW][1161394496] Executing command: stop for resource: ora.diskmon 1 1
2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] clsn_agent::stop {
2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] DiskmonAgent::stop {
2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] DiskmonAgent::stop }
2013-07-17 08:39:02.783: [ora.diskmon][1161394496] [stop] clsn_agent::stop }
2013-07-17 08:39:02.783: [ AGFW][1161394496] Command: stop for resource: ora.diskmon 1 1 completed with status: SUCCESS
2013-07-17 08:39:02.784: [ AGFW][1161394496] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:39:02.784: [ora.diskmon][1161394496] [check] DiskmonAgent::check {
2013-07-17 08:39:02.784: [ora.diskmon][1161394496] [check] DiskmonAgent::check } 2
2013-07-17 08:39:02.784: [ AGFW][1161394496] check for resource: ora.diskmon 1 1 completed with status: PLANNED_OFFLINE
2013-07-17 08:39:02.784: [ AGFW][1182374208] Agent sending reply for: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:547
2013-07-17 08:39:02.785: [ AGFW][1182374208] ora.diskmon 1 1 state changed from: STOPPING to: PLANNED_OFFLINE
2013-07-17 08:39:02.785: [ AGFW][1182374208] Agent sending last reply for: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:547
2013-07-17 08:39:02.785: [ AGFW][1182374208] Agent has no resources to be monitored.Sending suicide request.
2013-07-17 08:39:02.786: [ AGFW][1182374208] Agent sending message to PE: AGENT_SUICIDE[Proxy] ID 20486:72
2013-07-17 08:39:02.789: [ AGFW][1182374208] Agent is commiting suicide.
2013-07-17 08:39:02.790: [ AGFW][1182374208] Agent is exiting with exit code: 1
2013-07-17 08:39:12.176: [ AGFW][1664629584] Starting the agent: /u01/app/product/11.2.0/oracle/log/vmorarac2/agent/ohasd/orarootagent_root/
2013-07-17 08:39:12.176: [ AGENT][1664629584] Agent framework initialized, Process Id = 10581
2013-07-17 08:39:12.178: [ USRTHRD][1664629584] Utils::getCrsHome crsHome /u01/app/product/11.2.0/oracle
2013-07-17 08:39:12.178: [ USRTHRD][1664629584] Process::convertPidToString pid = 10581
2013-07-17 08:39:12.178: [ AGFW][1664629584] SERVER IPC CONNECT STR: (ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))
2013-07-17 08:39:12.178: [CLSFRAME][1664629584] Inited lsf context 0xd9309e0
2013-07-17 08:39:12.179: [CLSFRAME][1664629584] Initing CLS Framework messaging
2013-07-17 08:39:12.179: [CLSFRAME][1664629584] New Framework state: 2
2013-07-17 08:39:12.179: [CLSFRAME][1664629584] M2M is starting...
2013-07-17 08:39:12.180: [ CRSCOMM][1664629584] m_pClscCtx=0xd983bd0m_pUgblm=0xd987720
2013-07-17 08:39:12.180: [ CRSCOMM][1664629584] Starting send thread
2013-07-17 08:39:12.181: [ CRSCOMM][1115052352] clsIpc: sendWork thread started.
2013-07-17 08:39:12.181: [ CRSCOMM][1125542208] IPC Client thread started listening
2013-07-17 08:39:12.181: [ CRSCOMM][1125542208] init data sent from server
2013-07-17 08:39:12.181: [CLSFRAME][1664629584] New IPC Member:{Relative|Node:0|Process:0|Type:2}:OHASD:vmorarac2
2013-07-17 08:39:12.181: [CLSFRAME][1664629584] New process connected to us ID:{Relative|Node:0|Process:0|Type:2} Info:OHASD:vmorarac2
2013-07-17 08:39:12.182: [CLSFRAME][1664629584] Starting thread model named: MultiThread
2013-07-17 08:39:12.182: [CLSFRAME][1664629584] Starting thread model named: SingleThread
2013-07-17 08:39:12.182: [CLSFRAME][1664629584] Starting thread model named: SingleThreadT
2013-07-17 08:39:12.182: [CLSFRAME][1664629584] New Framework state: 3
2013-07-17 08:39:12.182: [ AGFW][1664629584] Agent Framework started successfully
2013-07-17 08:39:12.182: [ AGFW][1177991488] Agfw engine module has enabled...
2013-07-17 08:39:12.183: [CLSFRAME][1177991488] Module Enabling is complete
2013-07-17 08:39:12.183: [CLSFRAME][1177991488] New Framework state: 6
2013-07-17 08:39:12.183: [ AGFW][1177991488] Agent is started with userid: root , expected user: root
2013-07-17 08:39:12.183: [ AGFW][1177991488] Agent sending message to PE: AGENT_HANDSHAKE[Proxy] ID 20484:14
2013-07-17 08:39:12.192: [ AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.crs.type] ID 8196:886
2013-07-17 08:39:12.192: [ AGFW][1177991488] Added new restype: ora.crs.type
2013-07-17 08:39:12.192: [ AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.crs.type] ID 8196:886
2013-07-17 08:39:12.198: [ AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.ctss.type] ID 8196:888
2013-07-17 08:39:12.198: [ AGFW][1177991488] Added new restype: ora.ctss.type
2013-07-17 08:39:12.199: [ AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.ctss.type] ID 8196:888
2013-07-17 08:39:12.204: [ AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.diskmon.type] ID 8196:890
2013-07-17 08:39:12.204: [ AGFW][1177991488] Added new restype: ora.diskmon.type
2013-07-17 08:39:12.204: [ AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.diskmon.type] ID 8196:890
2013-07-17 08:39:12.209: [ AGFW][1177991488] Agent received the message: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:892
2013-07-17 08:39:12.209: [ AGFW][1177991488] Added new restype: ora.drivers.acfs.type
2013-07-17 08:39:12.210: [ AGFW][1177991488] Agent sending last reply for: RESTYPE_ADD[ora.drivers.acfs.type] ID 8196:892
2013-07-17 08:39:12.210: [ AGFW][1177991488] Agent received the message: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:894
2013-07-17 08:39:12.210: [ AGFW][1177991488] Added new resource: ora.diskmon 1 1 to the agfw
2013-07-17 08:39:12.210: [ AGFW][1177991488] Agent sending last reply for: RESOURCE_ADD[ora.diskmon 1 1] ID 4356:894
2013-07-17 08:39:12.210: [ AGFW][1177991488] Agent received the message: RESOURCE_START[ora.diskmon 1 1] ID 4098:895
2013-07-17 08:39:12.211: [ AGFW][1177991488] Preparing START command for: ora.diskmon 1 1
2013-07-17 08:39:12.211: [ AGFW][1177991488] ora.diskmon 1 1 state changed from: UNKNOWN to: STARTING
2013-07-17 08:39:12.216: [ AGFW][1167501632] Executing command: start for resource: ora.diskmon 1 1
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] clsn_agent::start {
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] DaemonAgent{
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] }DaemonAgent
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] DiskmonAgent::DiskmonAgent {
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] InitAttrs {
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] __IS_HASD_AGENT=TRUE
2013-07-17 08:39:12.216: [ora.diskmon][1167501632] [start] }InitAttrs
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] DiskmonAgent::DiskmonAgent }
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] DiskmonAgent::start {
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Arg Value = -d
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Arg Value = -f
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Total Count of Environment Variables = 3
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Adding Environment Variables _ORA_AGENT_ACTION=TRUE
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Adding Environment Variables __IS_HASD_AGENT=
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Adding Environment variable from USR_ORA_ENV ORACLE_USER=oracle
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Utils:execCmd action = 1 flags = 5 ohome = (null) cmdname = diskmon.
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] getOracleHomeAttrib: oracle_home = /u01/app/product/11.2.0/oracle
2013-07-17 08:39:12.217: [ora.diskmon][1167501632] [start] Utils:execCmd Running the binary from /u01/app/product/11.2.0/oracle/bin/diskmon
2013-07-17 08:39:12.220: [CRSTIMER][1198971200] Timer Thread Starting.
2013-07-17 08:39:12.220: [ora.diskmon][1167501632] [start] execCmd ret = 0
2013-07-17 08:39:12.220: [ora.diskmon][1167501632] [start] }DaemonAgent::start
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] DiskmonAgent::connect {
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] Process::convertPidToString pid = 10581
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] DiskmonAgent::connect }
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] DiskmonAgent::start }
2013-07-17 08:39:18.222: [ora.diskmon][1167501632] [start] clsn_agent::start }
2013-07-17 08:39:18.222: [ AGFW][1167501632] Command: start for resource: ora.diskmon 1 1 completed with status: SUCCESS
2013-07-17 08:39:18.223: [ AGFW][1177991488] Agent sending reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:895
2013-07-17 08:39:18.223: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:39:18.223: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:39:18.224: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:39:18.224: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:39:18.225: [ AGFW][1177991488] ora.diskmon 1 1 state changed from: STARTING to: ONLINE
2013-07-17 08:39:18.225: [ AGFW][1177991488] Started implicit monitor for:ora.diskmon 1 1
2013-07-17 08:39:18.225: [ AGFW][1177991488] Agent sending last reply for: RESOURCE_START[ora.diskmon 1 1] ID 4098:895
2013-07-17 08:39:38.231: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:39:38.232: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:39:38.232: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:39:38.232: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:39:38.232: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:39:58.237: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:39:58.238: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:39:58.238: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:39:58.238: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:39:58.238: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:40:12.107: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:966
2013-07-17 08:40:18.243: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:40:18.244: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:40:18.244: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:40:18.244: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:40:18.244: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:40:38.250: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:40:38.251: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:40:38.251: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:40:38.251: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:40:38.251: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:40:42.116: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:982
2013-07-17 08:40:58.245: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:40:58.246: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:40:58.247: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:40:58.247: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:40:58.247: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:41:12.125: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:998
2013-07-17 08:41:18.252: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:41:18.252: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:41:18.253: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:41:18.253: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:41:18.253: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:41:38.259: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:41:38.260: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:41:38.260: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:41:38.260: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:41:38.260: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:41:58.255: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:41:58.256: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:41:58.256: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:41:58.256: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:41:58.256: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:42:12.134: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1026
2013-07-17 08:42:18.261: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:42:18.262: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:42:18.262: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:42:18.262: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:42:18.262: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:42:38.268: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:42:38.269: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:42:38.269: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:42:38.269: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:42:38.269: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:42:58.265: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:42:58.266: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:42:58.266: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:42:58.267: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:42:58.267: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:43:12.144: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1054
2013-07-17 08:43:18.272: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:43:18.272: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:43:18.273: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:43:18.273: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:43:18.273: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:43:38.278: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:43:38.278: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:43:38.279: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:43:38.279: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:43:38.279: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:43:42.154: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1070
2013-07-17 08:43:58.284: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:43:58.285: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:43:58.285: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:43:58.285: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:43:58.285: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:44:12.154: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1086
2013-07-17 08:44:18.291: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:44:18.292: [ AGFW][1167501632] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:44:18.292: [ora.diskmon][1167501632] [check] DiskmonAgent::check {
2013-07-17 08:44:18.292: [ora.diskmon][1167501632] [check] DiskmonAgent::check } 0
2013-07-17 08:44:18.292: [ AGFW][1167501632] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:44:38.296: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:44:38.297: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:44:38.297: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:44:38.297: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:44:38.297: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:44:42.163: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1102
2013-07-17 08:44:58.302: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:44:58.303: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:44:58.303: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:44:58.304: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:44:58.304: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:45:12.174: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1118
2013-07-17 08:45:18.309: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:45:18.309: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:45:18.310: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:45:18.310: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:45:18.310: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:45:38.315: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:45:38.316: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:45:38.316: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:45:38.316: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:45:38.316: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:45:42.183: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1134
2013-07-17 08:45:58.312: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:45:58.313: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:45:58.313: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:45:58.313: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:45:58.313: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:46:12.192: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1150
2013-07-17 08:46:18.318: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:46:18.319: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:46:18.319: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:46:18.319: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:46:18.319: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:46:38.325: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:46:38.326: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:46:38.326: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:46:38.326: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:46:38.326: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:46:42.203: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1166
2013-07-17 08:46:58.332: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:46:58.333: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:46:58.333: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:46:58.333: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:46:58.333: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:47:12.203: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1182
2013-07-17 08:47:18.338: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:47:18.339: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:47:18.339: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:47:18.339: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:47:18.340: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:47:38.345: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:47:38.345: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:47:38.346: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:47:38.346: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:47:38.346: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:47:42.211: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1198
2013-07-17 08:47:58.351: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:47:58.352: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:47:58.352: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:47:58.352: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:47:58.352: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:48:12.220: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1214
2013-07-17 08:48:18.358: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:48:18.359: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:48:18.359: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:48:18.359: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:48:18.359: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:48:38.365: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:48:38.366: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:48:38.366: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:48:38.366: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:48:38.366: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:48:42.230: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1230
2013-07-17 08:48:58.370: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:48:58.371: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:48:58.371: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:48:58.371: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:48:58.371: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:49:12.238: [ AGFW][1177991488] Agent received the message: AGENT_HB[Engine] ID 12293:1246
2013-07-17 08:49:18.377: [ AGFW][1177991488] CHECK initiated by timer for: ora.diskmon 1 1
2013-07-17 08:49:18.378: [ AGFW][1157011776] Executing command: check for resource: ora.diskmon 1 1
2013-07-17 08:49:18.378: [ora.diskmon][1157011776] [check] DiskmonAgent::check {
2013-07-17 08:49:18.379: [ora.diskmon][1157011776] [check] DiskmonAgent::check } 0
2013-07-17 08:49:18.379: [ AGFW][1157011776] check for resource: ora.diskmon 1 1 completed with status: ONLINE
2013-07-17 08:49:23.262: [ AGFW][1177991488] Agent received the message: RESOURCE_STOP[ora.diskmon 1 1] ID 4099:1310
2013-07-17 0
Maybe you are looking for
-
This web site is so very contusing I don't even know how I posted the first question. "Ask a question" does not take me to a place to POST a question." It only takes me to articles. Click on "Get community Support" https://support.mozilla.org/en-US/p
-
Exporting DV back to a Samsung VP-D351
Hi, Could anyone tell me if they have been successful in exporting DV back to a Samsung VP-D351 camera within iMovie? I can transfer DV from the D351 to the iMac but after editing the movie I cannot copy it back to the camera, Ive tried all the sugg
-
Create new portal account in EJB
Hi All, I have a stateless EJB whose work is to update a database and create a portal account (by the service of UserManager Bean). These two actions use two different Tx Data Sources( myDataSource and commercePool ) respectively. Everytime I invoke
-
Form size increases with F4 help
Hello All, I am facing a performance issue in an interactive form. We have developed an adobe form and linked it to portal through WebDynpro. We have around 10 fields with F4 help on the screen. We achieved the F4 help with a push button next to the
-
HT204368 Cannot pair any device, phone like Nokia or else to transfer data, it appears Not pair.
Cannot pair to any cell phon like Nokia or else. It appears Not pair. Even to any other bt device.