Ora-3136 then ksvcreate: Process(m000) creation failed and db hangs

Question

Recently upgraded a large database from 10.2.0.4.0 to 10.2.0.5.0.
The database recevies many thousand short-lived connections per minute.
Since the upgrade we're repeatedly running into this problem
Symptoms are that the incoming sessions will start to time out with ora-3136, then the alert log will show
ksvcreate: Process(m000) creation failed
then finally existying connections will freeze and no more connections can be established. I can't even get in to shut the db down normally - i have to kill pmon to get to the point where i can restart it.
Typical alert log pattern:
Thu Feb 02 02:26:51 PST 2012
WARNING: inbound connection timed out (ORA-3136)
Thu Feb 02 02:31:21 PST 2012
ksvcreate: Process(m000) creation failed
Thu Feb 02 02:33:29 PST 2012
ksvcreate: Process(m000) creation failed
Thu Feb 02 02:35:36 PST 2012
kkjcre1p: unable to spawn jobq slave process
Thu Feb 02 02:35:36 PST 2012
Errors in file /scratch/jdevadf/oracle/product/10.2.0/admin/srg102/bdump/srg102_cjq0_5920.trc:
If i look in that file, i get this:
/scratch/jdevadf/oracle/product/10.2.0/admin/srg102/bdump/srg102_cjq0_5920.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /scratch/jdevadf/oracle/product/10.2.0
System name: Linux
Node name: dadvil0075
Release: 2.6.32-100.26.2.el5
Version: #1 SMP Tue Jan 18 20:11:49 EST 2011
Machine: x86_64
Instance name: srg102
Redo thread mounted by this instance: 1
Oracle process number: 10
Unix process pid: 5920, image: oracle@dadvil0075 (CJQ0)
*** 2012-02-01 21:34:08.175
*** SERVICE NAME:(SYS$BACKGROUND) 2012-02-01 21:34:08.149
*** SESSION ID:(3992.1) 2012-02-01 21:34:08.149
ssd_unwind_bp: unhandled instruction at 0x2fb3962 instr=f
*** 2012-02-02 02:34:29.355
Waited for process J001 to initialize for 60 seconds
*** 2012-02-02 02:34:29.362
Process diagnostic dump for J001, OS id=25477
loadavg : 5.22 3.75 3.36
Memory (Avail / Total) = 755.21M / 11706.12M
Swap (Avail / Total) = 1904.80M / 2047.34M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S jdevadf 25477 1 0 80 0 - 680831 semtim 02:33 ? 00:00:00 ora_j001_srg102
Stack:
[Thread debugging using libthread_db enabled]
0x00000038556d5497 in semop () from /lib64/libc.so.6
#0 0x00000038556d5497 in semop () from /lib64/libc.so.6
#1 0x0000000003d91322 in sskgpwwait ()
#2 0x0000000003d73d7a in skgpwwait ()
#3 0x00000000007acf87 in kslges ()
#4 0x00000000007ac8f0 in kslgetl ()
#5 0x0000000000777a37 in kstipg ()
#6 0x0000000000813aaa in ksucrp ()
#7 0x0000000002498644 in kkjrdp ()
#8 0x00000000030b98c7 in opirip ()
#9 0x00000000013a4a2b in opidrv ()
#10 0x0000000001f9e66e in sou2o ()
#11 0x000000000072d2c5 in opimai_real ()
#12 0x000000000072d17c in main ()
A debugging session is active.
Inferior 1 [process 25477] will be detached.
Quit anyway? (y or n) [answered Y; input not from terminal]
Process diagnostic dump actual duration=1.190000 sec
(max dump time=30.000000 sec)
*** 2012-02-02 02:34:30.544
*** 2012-02-02 02:34:40.555
Waited for process J001 to initialize for 70 seconds
*** 2012-02-02 02:34:40.555
Process diagnostic dump for J001, OS id=25477
loadavg : 4.80 3.71 3.35
Memory (Avail / Total) = 728.61M / 11706.12M
Swap (Avail / Total) = 1904.81M / 2047.34M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S jdevadf 25477 1 0 80 0 - 680831 semtim 02:33 ? 00:00:00 ora_j001_srg102
Stack:
[Thread debugging using libthread_db enabled]
0x00000038556d5497 in semop () from /lib64/libc.so.6
#0 0x00000038556d5497 in semop () from /lib64/libc.so.6
#1 0x0000000003d91322 in sskgpwwait ()
#2 0x0000000003d73d7a in skgpwwait ()
#3 0x00000000007acf87 in kslges ()
#4 0x00000000007ac8f0 in kslgetl ()
#5 0x0000000000777a37 in kstipg ()
#6 0x0000000000813aaa in ksucrp ()
#7 0x0000000002498644 in kkjrdp ()
#8 0x00000000030b98c7 in opirip ()
#9 0x00000000013a4a2b in opidrv ()
#10 0x0000000001f9e66e in sou2o ()
#11 0x000000000072d2c5 in opimai_real ()
#12 0x000000000072d17c in main ()
A debugging session is active.
Inferior 1 [process 25477] will be detached.
Quit anyway? (y or n) [answered Y; input not from terminal]
[repeated for several processes]
I can upload the files or they are available on the internal network.
Initially i thought it was bug 8426816, but that's supposed to be included in the 10.2.0.5.0 patchset.

Answer 1

Hi;
Symptoms are that the incoming sessions will start to time out with ora-3136, then the alert log will show
ksvcreate: Process(m000) creation failedDid you check note Troubleshooting Guide (TSG) - ksvcreate: Process(xxxx) creation failed [ID 1379200.1] ?
Regard
Helios