ZFS write hangs / DMA reset? Can it be fixed?

Question

My Solaris 10 (5/08) box with 3 Seagate 1TB drives hooked to a PCI-X SiI3124
randomly stops responding to ZFS IO for up to a minute, during alternate long
(100+MB) file reads and writes.
Disabling NCQ may eliminate the problem during reads, but not during writes:
/etc/system added line: "set sata:sata_func_enable = 0x4"
I don't get any kern.info messages telling me what is going on when this event occurs.
# zpool status
(blocks right before the config: line until ZFS is responsive again)
Here is the output of lockstat when ZFS is hung over ssh, with a pending Samba / network write process:
(NFS gives similar results)
# lockstat -kgIW -D 60 sleep 10
Profiling interrupt: 978 events in 10.074 seconds (97 events/sec)
Count genr cuml rcnt nsec Hottest CPU+PIL Caller
960 98% ---- 0.00 1514 cpu[0] thread_start
939 96% ---- 0.00 1486 cpu[0] idle
939 96% ---- 0.00 1486 cpu[0] cpu_halt
16 2% ---- 0.00 1383 cpu[0] scan_memory
11 1% ---- 0.00 3087 cpu[0]+2 dosoftint
5 1% ---- 0.00 7012 cpu[0] taskq_thread
4 0% ---- 0.00 8499 cpu[0] as_fault
4 0% ---- 0.00 8499 cpu[0] segvn_fault
4 0% ---- 0.00 8499 cpu[0] segvn_faultpage
4 0% ---- 0.00 8499 cpu[0] pagefault
4 0% ---- 0.00 8499 cpu[0] trap
4 0% ---- 0.00 8499 cpu[0] cmntrap
3 0% ---- 0.00 8906 cpu[0] anon_private
3 0% ---- 0.00 9393 cpu[0] vmem_free
3 0% ---- 0.00 9393 cpu[0] vmem_xfree
3 0% ---- 0.00 9393 cpu[0] kmem_depot_ws_reap
3 0% ---- 0.00 9393 cpu[0] kmem_magazine_destroy
3 0% ---- 0.00 9393 cpu[0] kmem_slab_free
3 0% ---- 0.00 9393 cpu[0] kmem_slab_destroy
3 0% ---- 0.00 9393 cpu[0] segkmem_free
3 0% ---- 0.00 9393 cpu[0] segkmem_free_vn
3 0% ---- 0.00 8828 cpu[0] hat_unload
3 0% ---- 0.00 8828 cpu[0] hat_pte_unmap
2 0% ---- 0.00 8630 cpu[0] seg_free
2 0% ---- 0.00 8630 cpu[0] segvn_unmap
2 0% ---- 0.00 8630 cpu[0] segvn_free
2 0% ---- 0.00 3442 cpu[0] kmem_cache_reap
2 0% ---- 0.00 8699 cpu[0] page_numtopp_nolock
2 0% ---- 0.00 8755 cpu[0] hwblkpagecopy
2 0% ---- 0.00 7333 cpu[0] mutex_enter
2 0% ---- 0.00 8637 cpu[0] hat_kmap_unload
2 0% ---- 0.00 8630 cpu[0] sys_sysenter
1 0% ---- 0.00 11412 cpu[0] elfexec
1 0% ---- 0.00 7278 cpu[0] fop_getpage
1 0% ---- 0.00 5848 cpu[0] as_unmap
1 0% ---- 0.00 7278 cpu[0] anon_zero
1 0% ---- 0.00 11412 cpu[0] as_free
1 0% ---- 0.00 11412 cpu[0] anon_decref
1 0% ---- 0.00 11412 cpu[0] anon_free
1 0% ---- 0.00 7278 cpu[0] swap_getapage
1 0% ---- 0.00 7278 cpu[0] swap_getpage
1 0% ---- 0.00 11412 cpu[0] exec_args
1 0% ---- 0.00 11412 cpu[0] gexec
1 0% ---- 0.00 11412 cpu[0] exece
1 0% ---- 0.00 11412 cpu[0] exec_common
1 0% ---- 0.00 10905 cpu[0] page_destroy
1 0% ---- 0.00 1631 cpu[0]+2 av_dispatch_softvect
1 0% ---- 0.00 1631 cpu[0]+2 cyclic_softint
1 0% ---- 0.00 11412 cpu[0] relvm
1 0% ---- 0.00 5848 cpu[0] munmap
1 0% ---- 0.00 5848 cpu[0] kmem_cache_free
1 0% ---- 0.00 9085 cpu[0] x86pte_invalidate_pfn
1 0% ---- 0.00 9085 cpu[0] x86pte_inval_func
1 0% ---- 0.00 9209 cpu[0] hat_unload_callback
1 0% ---- 0.00 3762 cpu[0] htable_reap
1 0% ---- 0.00 1631 cpu[0]+2 cbe_low_level
1 0% ---- 0.00 1631 cpu[0]+2 dtrace_state_clean
1 0% ---- 0.00 1631 cpu[0]+2 dtrace_speculation_clean
1 0% ---- 0.00 2738 cpu[0] (usermode)
1 0% ---- 0.00 3122 cpu[0] nfs4_reclaim
Edited by: mdurller on Jun 5, 2008 8:36 AM

Answer 1

If you are using a si3124, the driver WILL hang (zfs or otherwise) because sun is glacially slow in bring out the fix that I gave them almost a year ago (CR 6566207 - more info here http://bugs.opensolaris.org/view_bug.do?bug_id=6566207 ). So, if you want a working driver for 05/08, you have the following choices:
1) Find a version of sxce where the driver was fixed but still compatible with s10 (the putback was in Aug, says it's in snv71)
2) Bug sun for the IDR that has this fix
3) Ask someone nicely who has the fix already to send the patched si3124 drivers to you (like me :-) )
Note that the patched driver won't work in failsafe mode do to symbol conflicts even if you copy them into the failsafe image.
-r