Ultra 10 Panic

Question

Hi All,
I have one Ultra 10 server running Solaris 8. For last some days it has started giving panics and core dumps. I used the 'mdebug' utility to analyse the core dumps and I found following messages in the outfile.
Panic String
0x2a100045488: ![AFT1] Corrected Memory Error on CPU0 at TL=0, errID 0x0000005e.f823c7ae
AFSR 0x00000000.00100000<CE> AFAR 0x00000000.10253808
AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC 0x1014b690
UDBH 0x027b<UE> UDBH.ESYND 0x7b UDBL 0x0000 UDBL.ESYND 0x00
** Panic due to CE bits not set in the UDBs
Has anyone seen this before ? What can be the reason of the crash ?
FYI: The memory of this server is changed very recently.
Thanks

Answer 1

Your system experienced a correctable memory error trap, but when the kernel read out the various
error registers to figure out how to handle the error, it observed that the error that had latched in to
the UltraSPARC Data Buffer (UDB) was in fact an Uncorrectable Error (UE) and not a correctable one.
The UDB acts as a buffer between the older US-I and US-II chips and memory and its state is updated
separately from the CPU so in rare cases you may have the CPU start to take an exception and then
the in-flight memory transactions will cause the UDB to see a more serious problem by the time the
CPU starts executing the trap handler for the original error.
Given that you say you very recently changed memory, that is the first obvious suspect. I would
recommend reseating the DIMMs you changed and making sure they are properly and firmly
connected, and booting your machine with diag-level=max or use SunVTS to test everything carefully.
If you find a problem, your new DIMM may be physically damaged.
As an aside, in Solaris 10, we're introducing the new Predictive Self-Healing technology to make
these situations much easier to deal with. Rather than printing out these engineering-level messages,
your system will provide automated diagnosis for these types of problems as well as self-healing
capabilities like offlining bad processors and regions of memory. You can look at
http://sun.com/bigadmin/content/selfheal/ for more information.
-Mike