Observing extreme overhead due to the collector

Question

Hi all,
I've been using the collector for quite q while to chase down bottlenecks in a program with heavy I/O and pthreads-based blocking, and usually the profiler has a negligible overhead or even speeds things up slightly. However, just recently profiling has started to really slow things down, sometimes cutting performance by 90% or even 99%
Has anyone else seen anything like this? Are there any (not-so-)well-known things (or combinations of them) that I might be doing to trip up the collector?
Symptoms include:
- spawning a few dozen threads takes tens of seconds
- dtrace shows calls to pread() taking 0.33 seconds each when reading from tmpfs (a 10000x increase). Other syscalls see the same kind of slowdown but are called far less often and don't affect performance -- fstat is the worst, at over a second per call.
- the profiler results assign nearly 30% of program execution time (in the "other" category) to an innocent function which happens to be called soon after control returns from pthread_cond_wait
- top shows an average of only 1-2 threads active at any given time (normally 38-40)
Using dbx, I can toggle the symptoms by enabling and disabling the collector on a running instance of the program.
At first I thought it was the new patches I'd recently installed, but reverting the patches, then updating the OS version, then moving to a machine that's way behind on its patches (and working fine for other users) all made no difference.
Other details that probably don't matter: I link with -lmtmalloc and uses 'popen()' at regular intervals to call 'fstat' and parse its output; there's nothing special about my signal handlers other than a SIGINT handler that sets a single atomic flag and returns immediately (long-running functions watch for it as a signal to error out early).
Thanks!

Answer 1

Sorry... should have posted the versions before. I am compiling with Sun CC, running within dbx, and attaching/detaching the collector from there. My primary machine just got a fresh install of SS12 today:
SunOS 5.10 Generic_138888-03 sun4v sparc SUNW,SPARC-Enterprise-T5220 Solaris
Sun C++ 5.9 SunOS_sparc Patch 124863-01 2007/07/25
Sun Dbx Debugger 7.6 SunOS_sparc Patch 124872-01 2007/07/12
Sun Analyzer 7.6 SunOS_sparc 2007/05/03
and the secondary machine has:
SunOS 5.10 Generic_127111-09 sun4v sparc SUNW,SPARC-Enterprise-T5220
Sun C++ 5.9 SunOS_sparc 2007/05/03
Sun Dbx Debugger 7.6 SunOS_sparc 2007/05/03
Sun Analyzer 7.6 SunOS_sparc 2007/05/03
Part of the problem is that it's transient -- sometimes the profiler works perfectly, often it's somewhat slow (-20% perf), and once in a while the 99% performance hit comes. The first time this came up zfs and tmpfs were fighting over memory (with a large swap partition on a zvol); the next couple of times somebody fired up db2 and opened a database connection. Another time a colleague had accidentally overfilled /export/home with a large core file. Fixing the trigger, reinstalling software (or Solaris!), or rebooting doesn't seem to fix the problem... but sometimes, if you just run long enough, it fixes itself.
Case in point: I fired up the collector again to run dtrace and profiler numbers, and over about 30 minutes of profiling runs each measurement behaved better than the last until the problems completely went away. The long syscalls were the first to go, followed by a performance jump (only -20% vs non-profiled), then performance edged back up to normal, and finally thread creation time went back to near-instant. Last of all (and long after performance improved!) top and vmstat started reporting significant user time again. For quite a while there they were showing that 0.2% user and 2.1% kernel (64 contexts) achieved the work of 10 threads going full bore.
Then, I tried again with 40 threads instead of 10 and some issues came back: several minutes to spawn the threads, sleep(10) took 14 seconds to return, but performance not too terrible.
As it happens, most of this transition got captured by a single long experiment; I can't post it publicly, but did ran er_archive on the resulting test.er if that would be any help.
BTW, SIGINT is used only as an 'emergency stop' and dtrace reported no signals besides SIGPROF.