"not responding" means what, exactly?

Question

As we all know, Application Monitor (as well as the Force-Quit dialog, and I think other places) shows those nifty "(not responding)" suffixes highlghted in red when a process is working ovetime or simply hung.  I'm curious what triggers that, and how it can be captured from unix.
Here's what I think I know (from a lot of time spent on google and looking at man pages):
the 'not responding' cue means that the app is not reponding to system calls for roughly 2 seconds (with no explanation of what system calls are being tried or how they fail)
that Application Monitor mirrors the unix top utility (though there's nothing in top that correlates to 'not responding' - 'stuck' has a different meaning)
that nothing in the ps utility corresponds to 'not responding'
that this is one of those things that's going to drive me batty until I figure it out.
So what does 'not responding' in activity monitor *actually* mean, in unix terms?  I'm not looking for a way to kill processes (I know how to do that), and the obvious things like top's 'stuck' indicator are wrong, and not what I'm after.  I just want to know how I can leverage/mimic/replicate that 'not responding' thing that seems so simple in Application Monitor.  Where-oh-where in unix (or anywhere else) can this result be produced?
DankeDankeDanke... 

Answer 1

"Not Responding" is NOT a Unix concept.  Because if the process can be killed it is (Forced Quit), then it is still alive as far as Unix is concerned.
It is my guess that "Not Responding" means the application is not pulling items from its "Event Queue", so it will not see the "Quit" event, or any "Key Clicks", "Button Clicks", etc....
From a Unix perspective there are only 2 kinds of processes that cannot be killed.  Those that are already dead (Zombies), and a process that is stuck in the kernel, and the kernel code is not checking to see if any kill signals have been issued against the process.
A Zombie is a process "In Name Only", as all its resources have been released, and it just has a small process structure that is waiting for its parent process to collect the process' termination status.  A Zombie process CANNOT consume CPU time, it has not really memory to speak of, nor do any I/O.  If there are Zombie processes, they are generally the result of a program that was not paying attention and did not issue Unix 'wait()' calls to collect the status from an subprocess they created.  If the parent process exits without collecting their child process completion status, all the child processes get inherited by the Mac OS X /sbin/launchd process (PID 1) and /sbin/launchd will issue the wait() calls (on other Unix systems PID 1 is called 'init', but it provides the same orphaned child status collection services).  So to have a Zombie, the parent must still exist, but not care enough to collect the child process completion status.
The stuck in a kernel process is a full process that most likely has tripped over some kind of kernel bug.  It can consume CPU (its most likely talent),  tie up RAM, and cannot be killed via Force-Quit (kill -9, which cannot be ignored).  However, kill signals are delivered as a process is leaving kernel space and returning to user space.  So if a process is stuck in kernel space, and does not check to see if a signal has been delivered (the only way signals are honored in kernel code), and does not return to user space, then it cannot be killed.  However, this also signifies a bug in the kernel, as properly written kernel code should either quickly do its thing, and return, or if it needs to wait for something, check for signals and gracefully back-out.
If you have an unkillable process, the only way to cure that is to reboot.
A "Not Responding" app can generally be killed, so it neither a Zombie nor an unkillable Unix process.  Thus my conclusion that "Not Responding" is not a Unix concept, and why I suspect it is an application not checking its "Event Queue".
An Event Queue is a user mode GUI framework concept.  And has nothing to do with Unix, which is why there is no 'ps', no 'top', no anything Unixie that will report about "Not Responding".
I've been writing Unix kernel code since '95 for different Unix kernels, and I've been a Unix applications developer since '85.  "Not Responding" is a higher level abstraction that Apple has implemented, and I think it is related to their GUI frameworks and "Event Queues".  But it is all a guess based on knowing it is not Unix.