System hanging when it runs out of memory

Question

Hello,
my system has a finite amount of RAM and swap (it doesn't matter, to my purpose, if it's 16GB or 128MB, I'm not interested in increasing it anyway).
Sometimes my apps completely use all the available memory. I would expect that in these cases the kernel kills some apps to keep working correctly. The OOM Killer exists just for this, doesn't it?
What happens, instead, is that the system hangs. Even the mouse stops working. Sometimes it manages to get back to life in a few seconds/minutes, other times hours pass by and nothing changes.
I don't want to add more memory, I just want that the kernel kills some application when it's running out of memory.
Why isn't this happening?
Right now I'm writing a bash script that will kill the most memory-hungry process when available memory gets below 10%, because I'm sick of freezing my machine.
But why do I have to do this? Why do I need an user space tool polling memory usage and sentencing applications according to a cheap policy? What the hell is wrong with my OOM Killer, why isn't it doing is job?!

Answer 1

Alright, you won, now quit pointing out my ignorance
Your awkish oom killer is a lot cooler than mine, switching to it, thanks!
I did some testing (initially just to test the oom killing script) and found out that if a program tries to allocate all the memory it can, it gets eventually killed by linux's oom killer. If instead it stops allocating new memory when there are less than 4MB of free memory (or similar values) the OOM won't do anything, and the system will get stuck, like if a forkbomb was running.
That's it, this program with MINSIZE=1 will be killed, while with MINSIZE=4MB will force me to hard reboot:
#include <string.h>
#include <stdlib.h>
#define MINSIZE (1024*1024*4) // 4MB
int main( )
int block = 1024*1024*1024; // 1GB
void *p;
while( 1 ) {
p = malloc( block );
if( p ) {
memset( p, 85, block );
else if( block > MINSIZE ) {
block /= 2;
return 0;
Guess I'd need to go deeper to understand why Linux' oom killer works like that, but I won't (assuming the oom killing script will behave).