How to find reason for system crash?

Question

Hello,
I have a test system running in VirtualBox environment and its crashing every now and then, sometimes two times a day but sometimes it lasts for 3 days before crash happens.
system info
root>cat /etc/*release*
Solaris 10 10/09 s10x_u8wos_08a X86
root>uname -a
SunOS 5.10 Generic_142910-17 i86pc i386 i86pc
dump files
root>ls -l vmdump.*
-rw-r--r-- 1 root root 116064256 Jan 13 05:35 vmdump.0
-rw-r--r-- 1 root root 108003328 Jan 13 10:15 vmdump.1
-rw-r--r-- 1 root root 112852992 Jan 14 18:53 vmdump.2
-rw-r--r-- 1 root root 129236992 Jan 17 08:41 vmdump.3
-rw-r--r-- 1 root root 122486784 Jan 17 16:39 vmdump.4
started to use mdb
root@cs5vs01>mdb -k unix.4 vmcore.4
Loading modules: [ unix krtld genunix specfs dtrace cpu.generic uppc pcplusmp ufs ipc ip hook neti sctp arp usba fctl nca lofs zfs nfs random md cpc fcip sppp ]
::stack
vpanic()
kadmin+0x517()
uadmin+0xc7()
sys_syscall+0x17b()
$C
fffffe800100ee60 vpanic()
fffffe800100eeb0 kadmin+0x517()
fffffe800100ef00 uadmin+0xc7()
fffffe800100ef10 sys_syscall+0x17b()I wonder how I can continue from this point? Any support would be greatly appreciated!
Anyone knows about a good step by step debug guide? For example how to logically track down the root cause, good example commands and so on?
Thank you!
BR
Daniel

Answer 1

We can see from the following stack that the panic was caused by someone or some app calling uadmin(1M)
$C
fffffe800100ee60 vpanic()
fffffe800100eeb0 kadmin+0x517()
fffffe800100ef00 uadmin+0xc7()
fffffe800100ef10 sys_syscall+0x17b()
So the question is who/what. ::ptree will give you the process tree and you should see from that the execname responsible. What apps do you have running in this VBOX environment? Oracle RAC uses the uadmin interface because it doesn't have a failfast driver of it's own. There are several other applications who also use the uadmin syscall for similar reasons. Normally they panic the node/system because something has timed out and they need to restore cluster/system stability and integrity.
With regards to the step-by-step guide, no such document exists because it's impossible to write one. Each crash dump is different, each person does things slightly differently. What you need to do is understand how to use the tools (typically mdb or scat) and then start poking about in crash dumps. In fact you may find scat (Solaris Crash Analysis Tool) more user friendly and easier to learn than mdb. SCAT can be downloaded from https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_SMI-Site/en_US/-/USD/ViewProductDetail-Start?ProductRef=SCAT-5.2-G-F@CDS-CDS_SMI
There's some information for mdb in the man page and at the following locations:
http://dlc.sun.com/osol/docs/content/MODDEBUG/intro-1.html
http://download.oracle.com/docs/cd/E19253-01/816-5041/index.html
Unfortunately Crash Dump Analysis (CDA) isn't something that can easily be done via the forums. If you raise a service request it'll come to the Kernel group and we can then have a look for you.
Regards,
Steve