How to find the right kstat info & interpret it ?

Question

Having found out how to read the kstat structures I am
still struggling with finding the right information which
makes sense!
I am interested in the following statistics:
0. Swap Space - similar to that reported by swap
1. Scan rate similar to that reported by vmstat
2. Handspread page (see Adrian Cockcrofts' performance monitoring articles)
3. Disk Space Usage - similar to df -k
4. Process memory usage -- ps
5. System Error Messages -- this may not be possible to read at all
from kstat. Probably here I will have to make do with reading /var/adm/messages
6. Disk Errors -- similar to iostat -E (Solaris 2.6 onwards)
7. Ethernet stats -- like netstat -I le0
I found the following article on SunSolve Online: FAQ 1230 'Three Virtual Memory Performance Monitors' which directs me to some of the kstat cell
types.
I started with swap space. The above article plus a few other pointers seemed to indicate the the
correct kstat struct and field to use here was vminfo.swap_avail
When I compared the figures dumped from this structure with those reported by vmstat & swap they
did not compare! eg.
kstat: vminfo.swap_avail 321393372929
kstat: vminfo.swap_free 33339822544
vmstat unix tool:
swap avail (KB) 19932
Furthermore sysinfo.h indicates that vminfo.swap_avail is expressed in pages.
1 page = 4.096 KB, so the kstat figures should be multiplied by 4 which makes
the comparison even worse. Whilst I don't expect the figures to be
an exact match -- I was hoping they would be in the same ball-park,
Next I tried page in/page out rate
Here I compared:
kstat: cpu_vminfo.pgpgin 473224
kstat: cpu_vminfo.pgpgout 14554
vmstat unix tool:
pi (page in) 4
po (page out) 0
These figures sort of match up, if you take the most sig digit in
vminfo.
I was wondering if anyone has a definitive list on what are the best
kstat fields to use -- and if any wierd and wonderful calculations must
be performed on any of them to get figures close to that supplied by
the familiar unix tools.

Answer 1

Hi!
I wrote a quick program to read the vminfo stats and like you saw completely crazy figures. I looked further into this and found that every second, the system clock routine adds the freemem, swap and so on to a cumulative total.
To put it another way, the figure you see for free memory is the sum total of the value of freemem sampled every second since your system booted.
Thus, to get the average freemem in pages since boot, you need to read the value of freemem and divide by the number of seconds since boot.
You can derive the number of seconds in two ways from the kstats:
(1) read lbolt (incremented every 100th sec) and divide by 100 (hz)
or (2) read sysinfo.updates (which is incremented every time the kstats are updated).
I have attached a sample program that demonstrates both of these and reports average freemem since boot in K (like vmstat).
Similarly, to get a 5 second average, take two readings of freemem and sysinfo.updates five seconds apart , then calculate
(freemem2-freemem1)/(updates2-updates1)
It seems most, if not all, of the system counters are cumulative in this fashion.
Hope that helps.
Ralph
SUN Developer Technical Support
<pre>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <kstat.h>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/sysinfo.h>
#include <stdio.h>
main()
kstat_ctl_t *kc;
kstat_t *vminfo_ksp;
kstat_t *system_misc_ksp;
kstat_t *sysinfo_ksp;
kstat_named_t *lbolt_knp;
vminfo_t vminfo;
sysinfo_t sysinfo;
int hz;
int ltime;
ulong_t updates;
if ((kc = kstat_open()) == NULL)
perror("kstat_open failed"); exit(1);
/* get lbolt and divide by hz to get secs since boot */
system_misc_ksp = kstat_lookup(kc, "unix", 0, "system_misc");
kstat_read(kc, system_misc_ksp, NULL);
lbolt_knp = kstat_data_lookup(system_misc_ksp, "lbolt");
hz = sysconf(_SC_CLK_TCK);
ltime = lbolt_knp->value.l/hz;
/* get number of statistic updates so far (=secs since boot) */
sysinfo_ksp = kstat_lookup(kc, "unix", 0, "sysinfo");
kstat_read(kc, sysinfo_ksp, &sysinfo);
updates=sysinfo.updates;
printf("lbolt time %d\n",ltime);
printf("updates %d\n",updates);
/* get vminfo */
vminfo_ksp = kstat_lookup(kc, "unix", -1, "vminfo");
kstat_read(kc, vminfo_ksp, &vminfo);
printf("freemem %lld\n",(vminfo.freemem*4)/updates);
printf("freemem %lld\n",(vminfo.freemem*4)/ltime);
</pre>