今天早上数据库出现一个怪问题,

Question

早上业务部分和我说数据库连不上，首先ssh连接至数据库
sqlplus / as sysdba习惯性登入，卡住，无反应
再去查看alert.log
Mon Nov 19 08:33:08 2012
ksvcreate: Process(m000) creation failed
Mon Nov 19 08:35:16 2012
ksvcreate: Process(m000) creation failed
Mon Nov 19 08:51:37 2012
kkjcre1p: unable to spawn jobq slave process
Mon Nov 19 08:51:37 2012
Errors in file /opt/oracle/admin/nqora/bdump/nqora_cjq0_5029.trc:
Mon Nov 19 08:54:02 2012
kkjcre1p: unable to spawn jobq slave process
Mon Nov 19 08:54:02 2012
Errors in file /opt/oracle/admin/nqora/bdump/nqora_cjq0_5029.trc:
Mon Nov 19 09:02:24 2012
ksvcreate: Process(m001) creation failed
Mon Nov 19 09:04:29 2012
kkjcre1p: unable to spawn jobq slave process
Mon Nov 19 09:04:29 2012
Errors in file /opt/oracle/admin/nqora/bdump/nqora_cjq0_5029.trc:
Mon Nov 19 09:05:13 2012
Errors in file /opt/oracle/admin/nqora/bdump/nqora_pmon_5013.trc:
查看/opt/oracle/admin/nqora/bdump/nqora_cjq0_5029.trc:，日志信息是这样的
*** 2012-11-19 08:51:26.386
Waited for process J000 to initialize for 110 seconds
*** 2012-11-19 08:51:26.386
Dumping diagnostic information for J000:
OS pid = 3025
loadavg : 0.29 0.13 0.11
memory info: free memory = 0.00M
swap info: free = 0.00M alloc = 0.00M total = 0.00M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S oracle 3025 1 0 75 0 - 4645209 semtim 08:49 ? 00:00:00 ora_j000_nqora
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread 0x2b8b6f595c40 (LWP 3025)]
0x00000034d1ed4477 in semop () from /lib64/libc.so.6
#0 0x00000034d1ed4477 in semop () from /lib64/libc.so.6
#1 0x0000000003be9dfa in sskgpwwait ()
#2 0x0000000003bccdf0 in skgpwwait ()
#3 0x00000000007a0d9a in kslges ()
#4 0x00000000007a0918 in kslgetl ()
#5 0x00000000007ab0be in ksfglt ()
#6 0x0000000003b69b6d in kghfrunp ()
#7 0x0000000003b6880d in kghfnd ()
#8 0x0000000003b66fcb in kghalo ()
#9 0x0000000000783412 in ksp_param_handle_alloc ()
#10 0x0000000000782b14 in kspcrec ()
#11 0x000000000080550a in ksucre ()
#12 0x000000000236b3f0 in kkjrdp ()
#13 0x0000000002f4da4c in opirip ()
#14 0x000000000132b016 in opidrv ()
#15 0x0000000001eb3146 in sou2o ()
#16 0x0000000000723245 in opimai_real ()
#17 0x00000000007230fc in main ()
The program is running. Quit anyway (and detach it)? (y or n) [answered Y; input not from terminal]
*** 2012-11-19 08:51:27.214
Killing process (ospid 3025): (reason=x4 error=0)
... and the process is still alive after kill!
*** 2012-11-19 08:52:57.936
Waited for process J000 to initialize for 60 seconds
空闲内存和swap都为0了，但是原因查不到
/opt/oracle/admin/nqora/bdump/nqora_pmon_5013.trc信息是这样的
*** 2012-11-19 08:51:28.327
KGX cleanup...
KGX Atomic Operation Log 0x4c3bce9e0
Mutex 0x286a15bd0(1012, 0) idn 7adb03e oper GET_EXCL
Cursor Pin uid 1105 efd 5 whr 11 slp 16383
=================================================
KGL Child 5 Level 0 latch clean-up information:
=================================================
Operation = 12
kgllcpt1 = 3619f8d8 kgllcpt2 = 0
kgllcpt3 = 0 kgllcpt4 = 0
kgllcpt5 = 88bd0138 kgllcub4 = 0
*** 2012-11-19 08:54:31.599
KGX cleanup...
KGX Atomic Operation Log 0x4c3bce9e0
Mutex 0x286a15bd0(1012, 0) idn 7adb03e oper GET_EXCL
Cursor Pin uid 1105 efd 5 whr 11 slp 16383
=================================================
KGL Child 5 Level 0 latch clean-up information:
=================================================
Operation = 12
kgllcpt1 = 3619f8d8 kgllcpt2 = 0
kgllcpt3 = 0 kgllcpt4 = 0
kgllcpt5 = 88bd0138 kgllcub4 = 0

Answer 1

OS pid = 3025
loadavg : 0.29 0.13 0.11
memory info: free memory = 0.00M
swap info: free = 0.00M alloc = 0.00M total = 0.00M怀疑与内存使用情况有关， 是否有OS监控 top ps ?输出？ 找出耗尽内存的 进程 是首要任务 ， AWR ash alert.log os log都是可以参考的对象
其次任何提问 数据库版本信息都是必要的