High unexplained ram usage by jvm on linux

Question

I am experiencing the following issue.
I have a java server app on a very busy server, running many threads during high load periods (up to a couple thousands, I know I have to switch to nio).
The heap size is set with a max of 2GB, and never goes above that.
I also use some direct memory buffers, which at any time take a max of 3.5GB (imposed by app). The maxdirectmem is set to 7GB to avoid out of mem, but it never reaches that far.
I assume the app itself would take about 1GB max.
This gives a total of 6.5GB used, which is what I see a couple hours after server starts up.
Now, after a couple days, I realize that the app starts taking a huge amount of ram space.
The result of pmap shows a 15GB private space, which definitely tells that something is up.
mapped: 15301724K writeable/private: 14957284K shared: 2784K
As a result, my fs cache gets ridiculously small, and the server is struggling.
The different reasons I can think of are:
1) the high number of threads being allocated take a lot of memory. But this seems to be ruled out since a server just restarted shows almost as many threads, but does not have high memory usage. The threads are stored in a threadpool.
2) the direct buffers dont get released, java lets app use up to the max 7GB before trying to give any mem back to the OS. That would be really annoying for me, because reducing the size means risk of out of mem. Plus it would be a stupid design. I am currently testing with a max of 5GB, see if it helps.
3) somehow the hotspot JIT compilation feature, or some other jvm feature, takes too much ram.
I am trying to analyze the pmap output to understand what takes it all.
There is no big item that stands out, but I see many many lines like the following:
00002aab6f900000    3072 rwx-- 00002aab6f900000 000:00000   [ anon ]
00002aab6fc42000    1024 rwx-- 00002aab6fc42000 000:00000   [ anon ]
00002aab6fd63000    6160 rwx-- 00002aab6fd63000 000:00000   [ anon ]
00002aab70404000    1552 rwx-- 00002aab70404000 000:00000   [ anon ]
00002aab705aa000    1032 rwx-- 00002aab705aa000 000:00000   [ anon ]
00002aab7077f000    5120 rwx-- 00002aab7077f000 000:00000   [ anon ]
00002aab70cc3000    4104 rwx-- 00002aab70cc3000 000:00000   [ anon ]
00002aab710ed000    4120 rwx-- 00002aab710ed000 000:00000   [ anon ]
00002aab71500000    1020 rwx-- 00002aab71500000 000:00000   [ anon ]
00002aab715ff000       4-----
00002aab715ff000 000:00000   [ anon ]
00002aab71600000    1032 rwx-- 00002aab71600000 000:00000   [ anon ]
00002aab717f9000    8256 rwx-- 00002aab717f9000 000:00000   [ anon ]
00002aab72100000     988 rwx-- 00002aab72100000 000:00000   [ anon ]
00002aab721f7000      36-----
00002aab721f7000 000:00000   [ anon ]
00002aab7220d000    8192 rwx-- 00002aab7220d000 000:00000   [ anon ]
00002aab72a6b000   12312 rwx-- 00002aab72a6b000 000:00000   [ anon ]
00002aab73693000    3072 rwx-- 00002aab73693000 000:00000   [ anon ]
etc...There are like 10k lines like these, and they all get added in the private mapping of the process.
Does anyone know what that memory is used for?
Is there any java tool I can attach to a running process that would give me a complete view of the memory (not just the heap).
This is kind of related to the followind thread that never got answered:
http://forum.java.sun.com/thread.jspa?threadID=5172403
thanks in advance
Edited by: AJG on Mar 25, 2008 6:17 PM

Answer 1

When looking at the pmap, i see one big 2GB anon mapping before the end of libraries mapping:
00002aaaadfe0000 2048000 rwx-- 00002aaaadfe0000 000:00000   [ anon ]I assume this is the heap area, and it's fine.
The problematic part I think is the 10k lines I see after the last library mapping, each of them taking 1-2MB or more of space.