Received Platform error, process stuck in Trace?

Question

On Friday, one of my linux 2.6.11 systems running JRockit triggered a "platform error." After receiving the error, one of the processes is stuck in Trace mode (as reflected by top, ps, etc), which means none of the processes will die using "kill" either (which means the system won't cleanly reboot).
The error was:
[JRockit] Platform error is: Cannot allocate memory (12)
===================
JROCKIT FATAL ERROR
Failure to map memory (mmap) (70)
===================
The process that triggered the failure (from /proc/X/status):
Name: java
State: T (stopped)
SleepAVG: 93%
Tgid: 25143
Pid: 25143
PPid: 25142
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 256
Groups: 4 10 20 32 101 1000
VmSize: 2196916 kB
VmLck: 0 kB
VmRSS: 2177784 kB
VmData: 2190740 kB
VmStk: 84 kB
VmExe: 56 kB
VmLib: 5488 kB
VmPTE: 2156 kB
Threads: 1
SigPnd: 0000000000000100
ShdPnd: 0000000000004101
SigBlk: 0000000000000000
SigIgn: 8000000000000000
SigCgt: 1000004380405a8f
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
Running:
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-b64)
BEA WebLogic JRockit(R) (build dra-38972-20041208-2001-linux-ia32, R25.0.0-75, GC: System optimized over throughput (initial strategy singleparpar))
If I see this again, what's the best way to remedy it?

Answer 1

When a process dies its parent gets a notification. However, the parent
must explicitly poll for this notification using wait().
If a process (JRockit) dies and its parent (whatever) doesn't do wait(),
the JRockit process will become "zombie". This means that the process
won't ever do anything else, but it doesn't go away either. It won't go
away until the parent wait()s for it or the parent dies.
If by "traced" you mean that JRockit's process state is "T", it means
somebody (most probably gdb or strace) has called ptrace() on JRockit.
That turns the ptrace() caller into JRockit's parent. And JRockit won't
go away until the parent either dies or calls wait().
So next time (if there is one), try killing JRockit's parent process.
That said, I have seen cases where zombie processes have init (the first
process) for parent, and I don't have any explanation for that.
Also, if JRockit turns the whole system to mush, that's (at least) the
kernel's fault. The kernel shouldn't let that happen under any
circumstances, regardless of what JRockit does (either by design or by
being broken).
Regards //Johan
Nicole Pauls wrote:
It's a custom 2.6.11 kernel, but it is debian (and based on debian kernel sources). Good guess :)
The "not responding to -9" was definitely weird. I would expect the system to not respond to signals very quickly in a high load situation, but it's strange that after I passed many signals of differing value (ending with 23) that it suddenly died. It had sat for about 12 hours after severla -9s before I got around to passing the sequential signals to it.
I have seen OutOfMemoryErrors and other inside-the-JVM problems and been able to send -9 signals, so it was definitely related to the state the "Platform Error" put the system in.
Thanks for your thoughts -- I've asked a few other linux people out there for theirs as well and so far come up with nothing other than "must have been coincidence that it responded to -9 at the same time I was passing other signals" :)
Also, I am testing and then planning on migrating to 1.5.0_02, but since this is the first time I've seen the issue, I don't expect to see it again. If the system does reproduce the issue in a timely fashion, I'll update the JVM and see if that resolves the strangeness.