ZFS pool frequently going offline

Question

I am setting up some servers with ZFS raids and and finding that all of them are suffering from I/O errors that cause the pool to go offline (and when that happens everything freezes and I have to power cycle... then everything boots up fine).
T1000, V245, and V240 systems all exhibit the same behavior.
Root is mirrored ZFS.
The raid is configure as one big LUN (3 to 8 TB depending on system) and that lun is the entire pool. In other words, there is no ZFS redundancy. My thinking was I would let the raid handle that.
Based on some searches I decided to try setting
set sd:sd_max_throttle=20
in /etc/system and rebooting, but that made no difference.
My sense is that the troubles start when there is a lot of activity. I ran these for many days with light activity and no problems. Once I started migrating the data over from the old systems did the problems start. Here is a typical error log:
Jun 6 16:13:15 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1 (mpt3):
Jun 6 16:13:15 newserver Connected command timeout for Target 0.
Jun 6 16:13:15 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1 (mpt3):
Jun 6 16:13:15 newserver Target 0 reducing sync. transfer rate
Jun 6 16:13:16 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:16 newserver SCSI transport failed: reason 'reset': retrying command
Jun 6 16:13:19 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:19 newserver Error for Command: read(10) Error Level: Retryable
Jun 6 16:13:19 newserver scsi: Requested Block: 182765312 Error Block: 182765312
Jun 6 16:13:19 newserver scsi: Vendor: IFT Serial Number: 086A557D-00
Jun 6 16:13:19 newserver scsi: Sense Key: Unit Attention
Jun 6 16:13:19 newserver scsi: ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0
Jun 6 16:13:19 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:19 newserver incomplete read- retrying
Jun 6 16:13:20 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:20 newserver incomplete write- retrying
Jun 6 16:13:20 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:20 newserver incomplete write- retrying
Jun 6 16:13:20 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:20 newserver incomplete write- retrying
<... ~80 similar lines deleted ...>
Jun 6 16:13:21 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:21 newserver incomplete read- retrying
Jun 6 16:13:21 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:21 newserver incomplete read- giving up
At this point everything is hung and I am forced to power cycle.
I'm very confused on how to proceed with this.... since this is happening on all three systems I an reluctant to blame the hardware.
I would be very grateful to any suggestions on how to get out from under this!
Thanks,
David C

Answer 1

which s10 are you running? You could try increasing the timeout value and see if that helps (see mpt(7d) - mpt-on-bus-time). It could be that when the raid controller is busy, it may take longer to service something that it is trying to correct. I've seen drives just go out to lunch for a while (presumably, the SMART firmware is doing something) and comes back fine (but the delay in response causes problems).