Hardware error? umem: heap corruption detected

Question

hi all,
on one of our servers (t5240, ldom) just began occuring few days ago in 50% of creating zone attempts following error:
[NOTICE: Zone rebooting]
SunOS Release 5.10 Version Generic_141444-09 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: web0
Reading ZFS config: done.
umem allocator: boundary tag corrupted
bcp ^ bxstat = a110c8ed, should be f4eef4ee
buffer=c28720  bufctl=c26960  cache: umem_alloc_32
previous transaction on buffer c28720:
thread=ca  time=T-0.003097460  slab=b99b30  cache: umem_alloc_32
libumem.so.1'umem_cache_alloc+0x210
libumem.so.1'umem_alloc+0x60
libumem.so.1'malloc+0x28
libuutil.so.1'uu_zalloc+0x4
libuutil.so.1'uu_msprintf+0x34
svc.startd'?? (0x2e14c)
svc.startd'method_run+0x520
svc.startd'method_thread+0x15c
libc.so.1'?? (0xff1c8a20)
umem: heap corruption detected
stack trace:
libumem.so.1'?? (0xff29c64c)
libumem.so.1'?? (0xff29d6bc)
libumem.so.1'umem_cache_alloc+0x210
libumem.so.1'umem_alloc+0x60
libumem.so.1'malloc+0x28
libuutil.so.1'uu_zalloc+0x4
libscf.so.1'scf_pg_create+0x4
librestart.so.1'restarter_store_contract+0x4c
svc.startd'?? (0x2dae0)
svc.startd'method_run+0x590
svc.startd'method_thread+0x15c
libc.so.1'?? (0xff1c8a20)
Feb 15 13:29:23 svc.startd[5990]: restarting after interruption
Feb 15 13:29:25 svc.startd[5990]: svc:/system/cron:default: Method "/lib/svc/method/svc-cron" failed with exit status 99.
Feb 15 13:29:25 svc.startd[5990]: system/cron:default failed: transitioned to maintenance (see 'svcs -xv' for details).. i rollbacked the global zones' root to 100% healthy snapshot and did /reconfigure reboot, i also rollbacked a ztemplate zones' (the zone from which are other zones cloned) to initial snapshot .. but this still occurs randomly in 50% of cases ..
any idea what it is ??
thanks and regards
ave, daniel
ps. i have also investigatated using fmadm, ipmitool and SP> for some HW error, but with no success ..
Edited by: dpecka on Feb 16, 2010 12:43 AM

Answer 1

Hi George2,
The error message appears in the popup "Microsoft VC++ Debug Library" because you probably have Just-In-Time (JIT) debugging enabled in Visual Studio.  Please follow the linked instructions to disable debugging so that we can see where the crash occurs.  Once that's complete and you obtain theLabVIEW crash log, we can continue attacking the issue.  Goodnight!
Peter K.
National Instruments