Seeking runtime mutex analysis tool

Question

I am using Sun C++ 5.5 on Solaris 8, on an Ultra 10.
Does there exist a "known reliable" tool for tracking (logging, analyzing, reporting) a Solaris application's RUN TIME operationis on Solaris mutexes?
I've written such a tool, myself, but if I am to believe its current reports there are serious flaws in the Solaris mutex mechanism on Solaris 8. I conclude, instead, that my tool is broken and that I should try to find an equivalent tool that actually WORKS.
So, I would appreciate either pointers to such (a) tool(s) -- OR confirmation that there are, indeed, serious flaws in the Solaris mutex mechanism on Solaris 8 and that my homebrewed tool is truthful... ;-)
Thanks in advance,
Chris

Answer 1

Yes, I use the -mt option on all builds -- actually I use an "in-house standard" Makefile that sets this for me, every time.
It is not clear what luck I will have with the thread-library patches -- I am not in a position to use them myself, but must talk a separate sysadmin department into handling it for me -- but thank you, nonetheless, for the suggestion. It's also possible that the patches in question are already installed; only "they" will know.
I will also definitely try the "optional new thread library." Had not heard about that one. Is there something, somewhere, that I can read that will tell me what differences in behavior I might expect to see?
Details of my observations are ... difficult... to convey, due to sheer volume; I'll do my best to summarize here, but I apologize for the length of what follows.
First, truss of a small "test app," running for about 20 minutes (super-slow due to truss itself; equivalent to about two minutes at "normal" speed) produces over 1.5 million lines (62 Mbytes) of output. Heavy postprocessing to select just the operations associated with a single mutex produces a 47Kb file, most of which is inoccuous (sp?). Here and there, however, I observe sequences along the lines of the following (the parenthesized numbers are mine, for the sake of reference; the rest is pretty much exactly what truss puts out):
(1)   /4 -> mutex_lock(0xc68cb, ... (other args here)...)
(2)   /4 <- mutex_lock(0xc68cb, ... (other args here)...) = 0
(3)   /5 -> mutex_lock(0xc68cb, ... (other args here)...)
(4)   /5 <- mutex_lock(0xc68cb, ... (other args here)...) = 0
(5)   /5 -> mutex_unlock(0xc68cb, ... (other args here)...)
(6)   /5 <- mutex_unlock(0xc68cb, ... (other args here)...) = 0
      .  (repeat thread-5 lock-unlock pairs, numerous times, here)
(7)   /4 -> mutex_unlock(0xc68cb, ... (other args here)...)
(8)   /4 <- mutex_unlock(0xc68cb, ... (other args here)...) = 0Taking this at face value, my interpretation is as follows:
- at line 1, thread 4 requests a lock on the mutex.
- at line 2, thread 4 is granted the requested lock on the mutex
- at line 3, thread 5 requests a lock on the mutex. I would expect this request to block, due to thread 4's existing lock, such that we would not see the completion of this request until after an unlock performed by thread 4. However...
- at line 4, thread 5 is granted a lock on the mutex! If truss is telling us the bald-faced truth about the sequence of operations here, this lock should not have been granted, because thread 4 already held a lock on this mutex!
- in lines 5 and 6, we see thread 5 unlocking the mutex (which it "should never have been able to lock in the first place");
- between lines 6 and 7, but omitted here for the sake of brevity, I see ten or twenty duplicates of the sequence of operations in lines 3 through 6, namely, thread 5 locking-and-unlocking the mutex, entirely successfully (return value of 0) and without blocking (no other threads doing anything in between thread 5's activities)
- finally, in lines 7 and 8,, we see thread 4 unlocking the mutex that it locked at line(s) 1 (and 2).
The only explanations I can come up with, for the observed sequence -- which is seen in several other places in the log file, for this and several other mutexes, and on every run of the test program -- are these:
1) truss is 100% truthful and accurate in its reporting of mutex operations, in particular with regard to their exact sequencing, and the mutex facility does, in fact, occasionally allow thread 5 to lock a mutex already locked by thread 4. (I know for a fact that the facility allows e.g. thread 5 to unlock a mutex held by thread 4; there's another post somewhere in this forum about this being done deliberately in a piece of questionable code. I consider it somewhat of a flaw that the mutex facility allows this at all, but other people here argue that there "may be compelling reasons" (i.e. deep in the theoretical department at Sun) for "things being the way they are" in this respect. Anybody know for sure?)
2) Multi-threading being what it is -- a complex, subtle, and oft-unwieldy beast -- truss "gets confused" in some subtle way, and ends up reporting events in a slightly different sequence than they actually occurred. In the particular case outlined here, for example, perhaps thread 4 did unlock the mutex "right after it locked it," but truss, for whatever reasons, didn't report it right away -- namely, not until after thread 5 had perfectly "legally" locked-and-unlocked the mutex a few times. (Alas, it's not clear why truss would get around to reporting thread 5's activities, before getting around to reporting thread 4's activities previously completed. It almost seems to require large amounts of latency on all threads, interacting in a complex manner. I can just manage to leave mental "elbow room" for this, because I don't know exactly how truss works.. It's harder, though, to shrug off the essentially-identical results produced by code of my own, whose inner workings I do know, and which I would have trusted implicitly if it hadn't started delivering reports like these in the first place.
So, folks, the burning question is, which, if either, of these two conclusions can we confirm? If it's the former, we have serious problems, but should face them squarely and fix them. If it's the latter, I would desperately love to hear it -- with believable supporting details -- from Someone Who Really Knows, so that I can simply shrug off the disturbing "evidence" and write off the whole issue of realtime mutex-tracing as "one of those things that can never quite be nailed down."
Thanks for listening.