Some problems in measuring system performance

Question

Dear all
I'm a new in Solaris world. Recently, my team is doing some performance tests on the Solaris 10 platform. But I find that it puzzles me about how Solaris system will measure CPU load
for example, I use command prstat -L -p <pid> to determine the CPU load of each threads for a process and get the result like:
[zhf@SunOS@whale]/export/home/zhf/PCS_Rel/conf> prstat -L -p 12685
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
12685 zhf 58M 34M sleep 52 0 0:00:06 3.8% pcs/4
12685 zhf 58M 34M sleep 42 0 0:00:05 3.7% pcs/6
12685 zhf 58M 34M sleep 59 0 0:00:05 3.6% pcs/5
12685 zhf 58M 34M sleep 59 0 0:00:02 1.4% pcs/8
12685 zhf 58M 34M sleep 59 0 0:00:00 0.5% pcs/15
12685 zhf 58M 34M sleep 59 0 0:00:00 0.2% pcs/16
12685 zhf 58M 34M sleep 59 0 0:00:00 0.1% pcs/7
12685 zhf 58M 34M sleep 59 0 0:00:01 0.1% pcs/1
12685 zhf 58M 34M sleep 59 0 0:00:00 0.0% pcs/3
12685 zhf 58M 34M sleep 59 0 0:00:00 0.0% pcs/2
12685 zhf 58M 34M sleep 59 0 0:00:00 0.0% pcs/14
12685 zhf 58M 34M sleep 59 0 0:00:00 0.0% pcs/13
12685 zhf 58M 34M sleep 59 0 0:00:00 0.0% pcs/12
12685 zhf 58M 34M sleep 59 0 0:00:00 0.0% pcs/11
12685 zhf 58M 34M sleep 59 0 0:00:00 0.0% pcs/10
12685 zhf 58M 34M sleep 59 0 0:00:00 0.0% pcs/9
and prstat -mL -p <pid> to determine the microstate of each thread for a process. the example like:
[zhf@SunOS@whale]/export/home/zhf/PCS_Rel/conf> prstat -mL -p 12685
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
12685 zhf 28 0.4 0.0 0.0 0.0 72 0.0 0.0 377 15 762 0 pcs/4
12685 zhf 24 0.3 0.0 0.0 0.0 75 0.0 0.0 332 16 666 0 pcs/6
12685 zhf 21 0.3 0.0 0.0 0.0 78 0.0 0.0 290 8 584 0 pcs/5
12685 zhf 4.8 0.6 0.0 0.0 0.0 95 0.0 0.0 501 4 4K 0 pcs/8
12685 zhf 2.4 0.3 0.0 0.0 0.0 97 0.0 0.1 1K 3 2K 0 pcs/15
12685 zhf 0.9 0.3 0.0 0.0 0.0 0.0 99 0.0 503 10 1K 0 pcs/16
12685 zhf 0.3 0.2 0.0 0.0 0.0 0.0 99 0.0 501 0 1K 0 pcs/7
12685 zhf 0.1 0.1 0.0 0.0 0.0 0.0 100 0.1 501 2 501 0 pcs/3
12685 zhf 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 77 0 47 0 pcs/2
12685 zhf 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 pcs/14
12685 zhf 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 0 0 0 0 pcs/13
12685 zhf 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 0 0 0 0 pcs/12
12685 zhf 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 pcs/11
12685 zhf 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 pcs/10
12685 zhf 0.0 0.0 0.0 0.0 0.0 100 0.0 0.0 0 0 0 0 pcs/9
12685 zhf 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 0 0 0 0 pcs/1
Let's look at thread thread 4, I can see from -L result that thread 4 occupies 3.8% CPU time. But in -mL result shows that thread 4 user part only take 28% time and 72% is waiting for locks.
My question is, is the "waitig for locks" also calculated into CPU load? That's to say, if the 3.8% CPU load includes lock time, is that means the real processing time is 3.8%*28%? or the 3.8% CPU load not includes lock time, So, the 3.8% CPU load is the real cost of this thread (which is 28% user processing). I wish my explanation will not mess you :)
For my colleagues have many arguments on this, but no one could be sure. So I ask the experts here to seek the answers.
many many thanks in advance
Cheers
Shen
Message was edited by:
lishen

Answer 1

#1. The first display you have (without the -m) is not an immediate display. The CPU figures are long-term averages, so they can lie considerably.
Take an otherwise idle machine and run a CPU intensive program. It will take 100% of one CPU immediately, but 'top' and 'prstat' will take many seconds to reflect that.
#2. Whether 'waiting on a lock' takes CPU time probably depends on how it's wating. Solaris has adaptive locks, so sometimes the wait will take CPU time and other times it sleeps. (Going to sleep and waking up again has an overhead associated with it. So if the lock is going to release "quickly", then it makes sense to just spin on the CPU and do nothing for a few cycles until the lock is released. However if it's going to take "a while" for the lock to release, then it's better to release the CPU and let other processes have it while we wait for the lock.)
In most circumstances (and almost certainly in your example) the processes are sleeping while waiting for the lock. However there might be other situations where that is not true.
Darren