Extremely Disappointed and Frustrated with ZFS

Question

I feel like crying. In the last 2 years of my life ZFS has caused me more professional pain and grief than anything else. At the moment I'm anxiously waiting for a resilver to complete, I expect that my pool will probably fault and I'll have to rebuild it.
I work at a small business, and I have two Solaris 11 servers which function as SAN systems primarly serving virtual machine disk images to Proxmox VE cluster. Each is fitted with an Areca 12 port SATA controller put in JBOD mode, and populated with WD Caviar Black 2TB drives (probably my first and biggest mistake, was in not using enterprise class drives). One system is configured as a ZFS triple mirror, and the other a double mirror, both have 3 hot spares.
About a year ago I got CKSUM errors on one of the arrays I promptly ran zpool scrub on the array, and stupidly I decided to scrub the other array at the same time. The scrub quickly turned into resilvers on both arrays as more CKSUM errors were uncovered, and as the resilver continued the CKSUM count rose until both arrays faulted and were irrecoverable. Irrecoverable metadata corruption was the error I got from ZFS. After 20+ hours of attempted recovery, trying to play back the ZIL I had to destroy them both and rebuild everything. I never knew for certain what the cause was, but I suspected disk write caching on the controller, and/or the use of a non-enterprise class flash drive for ZIL.
In the aftermath did extremely thorough checking of all devices. I checked each backplane port, each drive for bad sectors, SATA controller onboard RAM, main memory, ran extended burn-in testing, etc. I then rebuilt the arrays without controller write caching and no seperate ZIL device. I also scheduled weekly scrubbing, and scripted ZFS alerting.
Yesterday I got an alert from the controller on my array with the triple mirror about read errors on one of the ports. I ran a scrub which completed and then I proceeded to replace the drive I was getting read errors on. I offlined the old drive and inserted a brand new drive and ran zfs replace. Re-silver started fast and then the rate quickly dropped down to 1.0MB/s, my controller began spitting out a multitude of SATA command timeout errors on the port of the newly inserted drive. Since the whole array had essential froze up, I popped the drive out and everything ran back at full speed, resilvering against one of the hot spares. Now the resilver soon started uncovering CKSUM errors similar to the disaster I had last year. Error counts rose and now my system dropped another drive off in the same mirror set and is resilvering 2 drives in the same set, with the third drive in the set showing 6 CKSUM errors. I'm afraid I'm going to lose the whole array again, as the only drive left in the set is showing errors as well. WTF?!?!?!?!?!?!
So I suspect I have a bad batch of disks, however, why the heck did zfs scrub complete and show no errors? What is the point of ZFS scrub if it doesn't accuratly uncover errors? I'm so frustrated that these types of errors seem to show up only during resilvers. I'm beginning to think ZFS isn't as robust as advertised....

Answer 1

FMA does notify admins automatically through the smtp-notify service via mail notification to root. You
can customize this service to send notification to your own email account on any system.
The poster said he has scripted for ZFS, but I don't know if that means reviewing zpool status or
FMA data.
With ongoing hardware problems, you need to review FMA data as well. See the example below.
Rob Johnston has a good explanation of this smtp-notify service, here:
https://blogs.oracle.com/robj/entry/fma_and_email_notifications
For the system below, I had to enable sendmail to see the failure notice in root's mail,
but that was it.
Thanks, Cindy
I failed a disk in a pool:
# zpool status -v tank
pool: tank
state: DEGRADED
status: One or more devices are unavailable in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or 'fmadm repaired', or replace the device
with 'zpool replace'.
scan: resilvered 944M in 0h0m with 0 errors on Mon Dec 17 10:30:05 2012
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c3t1d0 ONLINE 0 0 0
c3t2d0 UNAVAIL 0 0 0
device details:
c3t2d0 UNAVAIL cannot open
status: ZFS detected errors on this device.
The device was missing.
see: http://support.oracle.com/msg/ZFS-8000-LR for recovery
errors: No known data errors
Check root's email:
# mail
From [email protected] Mon Dec 17 10:48:54 2012
Date: Mon, 17 Dec 2012 10:48:54 -0700 (MST)
From: No Access User <[email protected]>
Message-Id: <[email protected]>
Subject: Fault Management Event: tardis:ZFS-8000-LR
To: [email protected]
Content-Length: 751
SUNW-MSG-ID: ZFS-8000-LR, TYPE: Fault, VER: 1, SEVERITY: Major
EVENT-TIME: Mon Dec 17 10:48:53 MST 2012
PLATFORM: SUNW,Sun-Fire-T200, CSN: 11223344, HOSTNAME: tardis
SOURCE: zfs-diagnosis, REV: 1.0
EVENT-ID: c2cfa39b-71f4-638e-fb44-9b223d9e0803
DESC: ZFS device 'id1,sd@n500000e0117173e0/a' in pool 'tank' failed to open.
AUTO-RESPONSE: An attempt will be made to activate a hot spare if available.
IMPACT: Fault tolerance of the pool may be compromised.
REC-ACTION: Use 'fmadm faulty' to provide a more detailed view of this event. Run 'zpool status -lx' for more information. Please refer to the associated reference document at http://support.oracle.com/msg/ZFS-8000-LR for the latest service procedures and policies regarding this diagnosis.