Kxs crashes spontaneously with a big cache

Question

Hello,
we are running iPlanet 6.0 sp2 on Solaris 8 in a clustered environment.
we are using iPlanet results caching C++ API feature to cache results
generated by applogics
running on C++ engine (kcs).
the caching criteria look like
"arg1=*,arg2=*,arg3=*" or "QUERY_STRING=*" or ""
the results cache can get quite big, so the KXS process size in memory
quickly goes up to
about 450M and from that "leaks" slightly several 2 to 10 Megs a day.
The problem is, we are experiencing spontaneous KXS crashes (core dumps from
SIGBUS),
which happen some time after it has built up the bulk of the cache,
according to stack traces it
happens in just about any part of the KXS work cycle:
trying to add a cache entry,
trying to find a cache entry,
trying to get some memory for some "DCCheck" - whatever that is
(you may see sample stack traces below)
And yes, we already tried to reduce the cache size, before we ran at about
800M KXS memory,
now we reduced the number of cache entries for each of our apps, so the KXS
memory after building
up cache is 450-500M.
Any ideas what we should look at? or is it just a buggy synchronization so
when the hash map
grows large & lookup takes longer it pukes? and we just need to go to yet
smaller sizes?
we never seen KXS behaving like that before, but we did not utilize result
caching to that extent
either...
Thanks,
Alexei
========================================================================
=>[1] GXHash2Table::Lookup(0x0, 0x1002d658, 0x74, 0x2fa9e0, 0x2555,
0x2f636769), at 0xff369b94
[2] GXRepositEl::FindInCache(0x2fa938, 0x2fa9e0, 0x2fa968, 0xfe06b494,
0x3be6ebd8, 0x0), at 0xfe05609c
[3] GXReposit::Get(0x2320c8, 0x2fa938, 0x232140, 0xf8e01b28, 0xfe06b494,
0xf8e01b37), at 0xfe057800
[4] GXReqStep_DCCheck::ExecStep(0xd00f968, 0x0, 0xff355c9c, 0xff37c348,
0xfde49bbc, 0x24fc48), at 0xfde368b4
[5] GXReqMgr::Main(0x2, 0xc2a888, 0x0, 0x1, 0x244eb8, 0xd00f968), at
0xfdf3b09c
[6] GXRequestProc(0x0, 0x244eb8, 0x800b0009, 0xc2a888, 0x1, 0x27c067d), at
0xfdf38af0
[7] GXExecThread(0xc2a888, 0xfdf38a6c, 0xff1ae000, 0x8, 0x9651608, 0x0),
at 0xfe3d4038
[8] ptroot(0x9651608, 0xff20c24c, 0x1, 0x5, 0x1, 0xfe401000), at
0xff1f2ad0
===================================================================
=>[1] t_delete(0x160cbc38, 0xfefb8018, 0x723e3c62, 0x160cb878, 0xefba3c0,
0x3b8), at 0xfef41f94
[2] realfree(0x160cbc30, 0xfefbe8b0, 0xfefb8018, 0x160cb878, 0x3b9,
0x160cb880), at 0xfef41af0
[3] mallocunlocked(0x40, 0x0, 0xfefb8018, 0x40, 0x160cb830, 0x0), at
0xfef41654
[4] malloc(0x40, 0x0, 0x80110000, 0xfe06b494, 0x3bdb3a46, 0x0), at
0xfef413ec
[5] operator new(0x40, 0xff05fc18, 0xff064d40, 0x0, 0xfe06b494,
0xfbe01b37), at 0xff021e0c
[6] GXReqStep_DCCheck::ExecStep(0x2dacb00, 0x0, 0xff355c9c, 0xff37c348,
0xfde49bbc, 0x24eb40), at 0xfde36a88
[7] GXReqMgr::Main(0x2, 0x2bf3b0, 0x0, 0x1, 0x244f48, 0x2dacb00), at
0xfdf3b09c
[8] GXRequestProc(0x0, 0x244f48, 0x800b0009, 0x2bf3b0, 0x1, 0x2c0ac1), at
0xfdf38af0
[9] GXExecThread(0x2bf3b0, 0xfdf38a6c, 0xff1ae000, 0x8, 0x2bf470, 0x0), at
0xfe3d4038
[10] ptroot(0x2bf470, 0xff20c24c, 0x1, 0x5, 0x1, 0xfe401000), at
0xff1f2ad0
======================================================================
=>[1] t_splay(0x4916230, 0x76178, 0x0, 0x0, 0x0, 0x0), at 0xfef42088
[2] t_delete(0x4916230, 0xfefb8018, 0x6861726c, 0x4915e70, 0x296a9f90,
0x3b8), at 0xfef41eec
[3] realfree(0x4916228, 0xfefbe8b0, 0xfefb8018, 0x4915e70, 0x3b9,
0x4915e78), at 0xfef41af0
[4] mallocunlocked(0x40, 0x0, 0xfefb8018, 0x40, 0x4915e28, 0x0), at
0xfef41654
[5] malloc(0x40, 0x0, 0x80110000, 0xfe06b494, 0x3bcc9f93, 0x0), at
0xfef413ec
[6] operator new(0x40, 0xff05fc18, 0xff064d40, 0x0, 0xfe06b494,
0xfc601b37), at 0xff021e0c
[7] GXReqStep_DCCheck::ExecStep(0x2a6c2f08, 0x0, 0xff355c9c, 0xff37c348,
0xfde49bbc, 0x250c30), at 0xfde36a88
[8] GXReqMgr::Main(0x2, 0x2bf210, 0x0, 0x1, 0x246c60, 0x2a6c2f08), at
0xfdf3b09c
[9] GXRequestProc(0x0, 0x246c60, 0x800b0009, 0x2bf210, 0x1, 0x2be7b9), at
0xfdf38af0
[10] GXExecThread(0x2bf210, 0xfdf38a6c, 0xff1ae000, 0x8, 0x2bf2d0, 0x0),
at 0xfe3d4038
[11] ptroot(0x2bf2d0, 0xff20c24c, 0x1, 0x5, 0x1, 0xfe401000), at
0xff1f2ad0
====================================================================
=>[1] GXBufferChain::~GXBufferChain(0x18214d30, 0x3, 0xff2f06a8, 0x1a, 0x19,
0x1), at 0xff2f0714
[2] GXBufferChain::Release(0x0, 0x18214d30, 0xd, 0xfefb8018,
0x15d423d0, 0x1), at 0xff2f0884
[3] GXVALClear(0x181626b0, 0x0, 0x388, 0x0, 0x0, 0x0), at 0xff2fbfd8
[4] GXValListEl::~GXValListEl(0x18162698, 0x3, 0xff2feaf0, 0xff318bdc,
0x15d423d0, 0x28), at 0xff2feb38
[5] GXHashTable::~GXHashTable(0x17ae98e8, 0x2, 0x34, 0xd, 0x0,
0x4d58), at 0xff368798
[6] GXValList::~GXValList(0x17ae98a8, 0x3, 0xff2fe02c, 0xfef42278,
0x6c65, 0x18244569), at 0xff2fe104
[7] GXValList::Release(0x0, 0x17ae98a8, 0x69, 0xfe06b494, 0x6c,
0xff369a68), at 0xff2fce7c
[8] GXCacheEntry::~GXCacheEntry(0x1814bab8, 0x3, 0x0, 0x1, 0x17dce5f8,
0x0), at 0xfe055604
[9] GXRepositEl::AddToCache(0x2d8f20, 0x2d8f50, 0x2d8fc8, 0xfe06b494,
0x1814bab8, 0x178c68d8), at 0xfe055dc8
[10] GXReposit::Add(0x237a90, 0x237b08, 0xfe06be1c, 0x2d8f20,
0x8024000d, 0xfe057884), at 0xfe057dfc
[11] GXReqStep_DCCache::ExecStep(0x0, 0xfde4a2ec, 0xff37c348,
0xfde49bbc, 0x24ec18, 0x24ec18), at 0xfde36fe4
[12] GXReqMgr::Main(0x0, 0x2bbf70, 0x0, 0x1, 0x244f48, 0x166e7138), at
0xfdf3b09c
[13] GXRequestProc(0x0, 0x244f48, 0x800b0009, 0x2bbf70, 0x1,
0x2b9d11), at 0xfdf38af0
[14] GXExecThread(0x2bbf70, 0xfdf38a6c, 0xff1ae000, 0x8, 0x2bc030,
0x0), at 0xfe3d4038
[15] ptroot(0x2bc030, 0xff20c24c, 0x1, 0x5, 0x1, 0xfe401000), at
0xff1f2ad0

Answer 1

It crashes after adding (adding, not converting) 300 clips approximatively.
Sometimes more, sometimes less.
I have 7.0.2 version.
I'm using the plugin, yes.
I have a 3x1To RAID1 internal SATA (spinpoint F1)
Message was edited by: chle