Solaris 10 hopelessly unreliable - random hangs

Question

Hi,
I have several systems which over recent months have just hung. They are all X86 systems running a variety of versions of Solaris 10 and 11. Latest system Solaris 10u5 fully patched.
There are no messages in /var/adm/messages. The systems have zfs filesystems active at the time that they hang (might or might not be significant). The latest crash occurred during a zfs receive operation, the only other activity on the system was firefox web browser. On another system which has hung three times in two days, we see lockd running with 53% of the cpu when the system starts to creak, cpu is 95-98% idle, the io is minimal but the system becomes unresponsive. A little later all network services become unresponsive and the console dies.
The only option is a power off.
We also have another solaris 10u5 box which randomly reboots, possibly even powers off and on again (yes its on a UPS and the other boxes on the same UPS are fine), again with no messages, thats a new sunfire x4100 box.
Anybody have any idea how to solve this one? I can't get into the machines at all do perform any diagnostics when they hang.
I need to upgrade our oracle server but if solaris 10 is this unreliable I can't.

Answer 1

bbrain wrote:
Your right, I am referring to the x session. With all io operations apparently blocked I wonder if the serial port console will still operate? any thoughts.if ping responds, it'll most likely work. However, that's not what I meant by "console." I'm was talking about a non-X text display (you can select it from the login window). X does do file io at times (loading fonts and cursors, doing logging), so it can hang if the filesystem goes south.
Failing that I could run the system from the console. I looked for kadb just after my last post and noticed it had changed (just shows how long it has been since I last had cause to use it: :-< ).luckily it isn't too different. %-)
I did have x window displays with zpool iostat 5 and vmstat 5 and top running but when the hang occurs they look fine but stop updating.It's always a little tricky since X seized whether or not this is significant. I don't know what your free memory #s usually are - do know that zfs can get cranky rather quickly if your free drops below 100M.
Looks like we are on the trail of the 4100 power off, when the ups gets activated through a test or loss of mains everything in the rack stays up apart from the 4100. Looks like it's got a sensitive power supply we are going to place a support call to see if sun can shed any light on the problem.HW problems always adds spice to the mix...
As for the other machine that hung three times in as many days, well it's been running happily again for over a week we offloaded some of its shared filesystems but didn't change anything else.Hmmmm. Perhaps you have a bad spot on the disk? I just had a raptor die which caused the machine to appear as if it locked up. It hung the sata bus at the point of error, and the ata driver is too persistent in trying to read the block. You might try using format and do a analyze/read (non-destructive scan) to check this out.
-r