Contention on index block splits consuming significant database time

Question

Hi Guys,
can anybody suggest on how to remove Contention on index block splits,this is giving so many issues on my production DB,the CPU usage shots up and application hangs for few minutes.
DB is 10.2.0.3 and OS is IBM AIX 5.3

Answer 1

I found this.. it might be useful
One possibility is that this is caused by shared CBC latching peculiarities:
1) during normal selects your index root block can be examined under a
shared cache buffers chains latch.
So as long as everybody is only reading the index root block, everybody can
do it concurrently (without pinning the block). The "current holder count"
in the CBC latch structure is just increased by one for every read only
latch get and decreased by one on every release. 0 value means that nobody
has this latch taken currently.
Nobody has to wait for others for reading index root block in all read only
case. That greatly helps to combat hot index root issues.
2) Now if a branch block split happens a level below the root block, the
root block has to be pinned in exclusive mode for reflecting this change in
it. In order to pin a block you need to get the corresponding CBC latch in
exclusive mode.
If there are already a bunch of readers on the latch, then the exclusive
latch getter will just flip a bit in the CBC latch structure - stating it's
interest for exclusive get.
Every read only latch get will check for this bit, if it's set, then the
getters will just spin instead, waiting this bit to be cleared (they may
yield or sleep immediately as well, I haven't checked). Now the exclusive
getter has to spin/wait until all the shared getters have released the latch
and the "current holder count" drops to zero. Once it's zero (and the getter
manager to get on to CPU) it can get the latch, do its work and release the
latch.
During all that time starting from when the "exclusive interest" bit was
set, nobody could access this indexes root block except the processes which
already had the latch in shared mode. Depending on latch spin/sleep strategy
for this particular case and OSD implementation, this could mean that all
those "4000 readers per second" start just spinning on that latch, causing
heavy spike in CPU usage and they all queue up.
How do diagnose that:
You could sample v$latch_misses to see whether the number of "kcbgtcr:
kslbegin shared" nowaitfails/sleeps counter takes an exceptional jump up
once you observe this hiccup.
How to fix that once diagnosed:
The usual stuff, like partitioning if possible or creating a single table
hash cluster instead.
If you see that the problem comes from excessive spinning, think about
reducing the spinning overhead (by reducing spincount for example). This
could affect your other database functions though..
If you can't do the above - then if you have off-peak time, then analyse
indexes (using treedump for start) and if you see a block split coming in a
branch below root block, then force the branch block to split during
off-peak time by inserting carefully picked values into the index tree,
which go exactly in the range which cause the proper block to split. Then
you can just roll back your transaction - the block splits are not rolled
back nor coalesced somehow, as this is done in a separate recursive
transaction.
And this
With indexes, the story is more complicated since you can't just insert a
row into any free block available like with tables. Multiple freelists with
tables help us to spread up inserts to different datablocks, since every
freelist has its distinct set of datablocks in it. With indexes, the
inserted key has to go exactly to the block where the structure of b?tree
index dictates, so multiple freelists can't help to spread contention here.
When any of the index blocks has to split, a new block has to be allocated
from the freelist (and possibly unlinked from previous location in index),
causing an update to freelist entry in segment header block. Now if you had
defined multiple freelists for your segment, they'd still remain in the
single segment header block and if you'd have several simultaneous block
splits, the segment header would become the bottleneck.
You could relieve this by having multiple freelist groups (spreading up
freelists into multiple blocks after segment header), but this approach has
it's problems as well - like a server process which maps to freelist group 1
doesn't see free blocks in freelist group 2, thus possibly wasting space in
some cases...
So, if you have huge contention on regular index blocks, then you should
rethink the design (avoid right hand indexes for example), or physical
design (partition the index), increasing freelists won't help here.
But if you have contention on index segment's header block because of block
splits/freelist operations, then either partition the index or have multiple
freelist groups, adding freelists again won't help here. Note that adding
freelist groups require segment rebuild.