Solaris 10 / ZFS Parameter for Continuous Write Loads

Question

I have been spending some time working on tuning continuous write load tuning on a Solaris 10/ZFS based system. One of the issues I observed is a drop in throughput every several seconds. Removing the fsync() calls from JE (you would never want to do this normally) smoothed out the dips in throughput which pointed to IO issues.
My colleague Sam pointed me at this discuss-zfs entry. And indeed, adding
set zfs:zfs_write_limit_override = 0x2000000
to /etc/system (with the fsync() calls put back into JE) does in fact seem to smooth things out (0x2000000 = 33MB). Of course, you'll want to adjust that parameter based on what size of on-board disk cache you have.
One way to get a rough indication of whether this is a potential problem is to use iostat and see if there are IO spikes.
Charles Lamb

Answer 1

# prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t2d0s2
fmthard: New volume table of contents now in place.Most likely the rpool detection on a boot disk is probably less than optimal. I'd start over, do a dd if=/dev/zero of=/dev/rdsk/c1t2d0s2 count=100,
then do a format -e and add partition data that matches c1t0d0s2 (I know fmthard is supposed to do this, but zfs boot is a strange beast that straddles the old fdisk and new EFI labeling, and fmthard may not have been updated correctly). Then try the zpool attach. If this succeeds, make sure you do an installgrub on the second disk so it will boot.