Help finding location of Memory corruption using libumem and mdb

Question

Hi there,
I am debugging an application, an rpc based server that runs on a solaris 10 box. This is a server that has a footprint of about 2.3 gigs (while using libumem debug options, its close to 3.5 gigs). Most of the size is because of caching. When certain modifications are done to objects in the cache, a copy of the cache is made by the modifying thread, and then the older copy is freed when no other thread is using it.
One of our customers is using libumem as the memory manager. They started seeing crashes every now and then when the copy of cache is being freed. Obviously, it doesnt crash everytime, and each time the crash is with a different object all together. libumem raises a signal 6 abort, and complains that a piece of memory that is being freed is invalid or corrupt memory. Needless to say, countless hours of code walkthrough, and trying to reproduce the problem in house has failed. So we turned on libumem debugging and logging. Using watchmalloc at this site is not an option because of performance considerations. This is the setup we are using.
LD_PRELOAD=${LD_PRELOAD}:libumem.so.1
export LD_PRELOAD
UMEM_DEBUG=default;export UMEM_DEBUG
UMEM_LOGGING=transaction;export UMEM_LOGGING
We have a core from this crash. I am using mdb to analyze the core.
::umem_status tells me that the address being freed is invalid or corrupt memory. It even shows me where it was initially allocated. I want to find out where this piece of memory got corrupted.
::umalog shows the log of all memory allocation and freeing transactions. But because this is a huge app, it has a ton of information. I tried using address::umalog and that doesnt seem to narrow down the log.
So my question is, given the core generated using these settings, what mdb commands do I use to find out what operation/code corrupted the memory. I would expect to see three pieces of information for the address whose free failed.
1. Where it was initially allocated
2. Where the error/crash occured
3. What operation corrupted the memory, whether its a overwrite or freeing (leading to double free)
I ll be glad to provide more information if needed.
Any help is deeply appreciated.

Answer 1

If you have the address that is corrupt, try finding addresses that are just before the corrupt address. If your corrupt address is 0xF028, for example, look for addresses like 0xF020 or 0XF018. Most likely what's happening is some code is writing too far into a block, corrupting what's in the next block.