Excessive system CPU time on Solaris 10 host with multiple zones

Question

We current have three T2000s running Solaris 10 with all of the latest patches installed. Each machine is identically configured with a single, 1.2GHz 8-core CPU & 32GB RAM. Two of the three are in our production environment and have three zones serving users' needs (Oracle app servers, to be specific).
The third server is our test environment, and it hosts 5 zones. Four of the five zones are similar to those in our production environment (running Oracle or other J2EE app servers). The fifth zone is running eight Oracle RDBMS instances - seven at 9.2, one at 10.2. As this is a test environment, those instances are configured to use a modest amount of system resources.
We are seeing an odd behavior that at first blush appeared to imply that we had put too much on this single server. However, after looking into it more closely, I'm now thinking that what we are seeing may in fact be some sort of OS issue.
Specifically, when monitoring the server load in the global zone, we will see sudden spikes in the load factor, jumping above 20.00 and staying there for a minute or two, then dropping down to 3.00-6.00. During the time that the load is very high, vmstat reveals that an inordinate amount of CPU time is being spent in the kernel.
For example:
kthr      memory            page            disk          faults      cpu
r b w   swap  free  si  so pi po fr de sr m1 m1 m1 m2   in   sy   cs us sy id
0 0 0 62970776 1092640 0 0  0  3  2  0  0  0  0  0  0 3056 21548 5902 4  3 93
0 0 0 62974184 1094632 0 0  0  8  6  0  0  0  0  0  0 3000 17155 5755 3  1 95
0 0 0 62971736 1091760 0 0  0  3  2  0  0  0  0  0  0 2982 20254 5754 4  2 94
0 0 0 62974080 1094424 0 0  0  3  2  0  0  0  0  0  0 3041 18316 5842 4  1 95
0 0 0 62967808 1088256 0 0  0  3  3  0  0  0  0  0  0 3250 18788 6118 4 11 85
0 0 0 62880504 1081528 0 0  0  2  2  0  0  0  0  0  0 3673 14910 6549 3 31 66
0 0 0 62899936 1078272 0 0  0  0  0  0  0  0  0  0  0 3415 14216 6069 4 33 63
0 0 0 62928744 1075224 0 0  0  5  5  0  0  0  0  0  0 4100 14889 7592 4 50 46
1 0 0 62870280 1068096 0 0  0  0  0  0  0  0  0  0  0 4388 12581 8186 5 74 21
5 0 0 62860552 1062064 0 0 51  0  0  0  0  0  0  0  0 4856 12904 9275 5 79 17
11 0 0 62837472 1054064 0 0 0  5  5  0  0  0  0  0  0 4350 11576 9084 4 89  7
15 0 0 62860376 1045088 0 0 0  0  0  0  0  0  0  0  0 4491 10718 8954 4 93  3
19 0 0 62828208 1034744 0 0 0  2  2  0  0  0  0  0  0 4392 10194 9308 4 93  3
5 0 0 62884880 1029232 0 0  0  5  5  0  0  0  0  0  0 4860 10864 9716 5 93  2
0 0 0 62888088 1026552 0 0  0  0  0  0  0  0  0  0  0 4551 11987 8814 5 88  7
0 0 0 62861944 1014688 0 0  0  0  0  0  0  0  0  0  0 4612 13246 8972 5 91  4
1 0 0 62866912 1009992 0 0  2  5  5  0  0 11 11 11  0 4551 15213 9024 6 91  3
0 0 0 62911632 1043184 0 0 88  3  3  0  0  4  2  2  0 4105 42573 7913 12 42 46
0 0 0 62962560 1082128 0 0 13  2  2  0  0  5  3  2  0 3107 19107 5853 4  2 94Note that when the load first spikes, there are very few jobs in the run queue. Of course, as the kernel monopolizes more and more of the CPU time, the number of jobs in the run queue builds until such time as the kernel relinquishes the CPU and the user jobs are serviced.
I have never seen this kind of vmstat output in the 20 years that I've been administering Sun servers, though I must admit that these three T2ks are the first multi-core, zoned machines that I have experience with.
So do I have something configured wrong, do we have too many services configured for this one machine to handle, or is there indeed and OS issue involved here?
Thanks,
Bill

Answer 1

Darren,
Thanks for the feedback. I ran the hotkernel script on both a low-loaded and a higher loaded system (only saw my sys time reach about 60% compared to the 90+ I was seeing yesterday). Here are the 10 most frequently called kernel functions on the normal system:
SUNW,UltraSPARC-T1`bcopy                                 1016   0.1%
unix`mutex_vector_enter                                  1059   0.1%
unix`disp_getwork                                        1093   0.1%
unix`page_freelist_coalesce                              1512   0.2%
zfs`fletcher_2_native                                    1519   0.2%
SUNW,UltraSPARC-T1`copyin                                1546   0.2%
SUNW,UltraSPARC-T1`copyout                               1602   0.2%
unix`page_trylock                                        1789   0.2%
unix`mutex_enter                                         1848   0.2%
unix`cpu_halt                                          762334  95.3%Here's the top 10 when the system is getting pounded:
unix`disp_getwork                                         495   0.1%
unix`page_freelist_coalesce                               525   0.2%
unix`page_geti_contig_pages                              6637   2.0%
unix`page_unlock_noretire                               10124   3.0%
unix`mutex_exit                                         10652   3.1%
unix`page_trylock_contig_pages                          11461   3.4%
unix`mutex_vector_enter                                 12785   3.8%
unix`mutex_enter                                        14907   4.4%
unix`page_trylock                                       50461  14.9%
unix`cpu_halt                                          211391  62.2%Any thoughts as to what this implies?
Thanks,
Bill