Start-up / shut-down (and other) problems with ZFS over iSCSI

Question

Hi,
I've had a limited time in which to try some concepts on an evaluation x4500 server. This is unfortunately my last day, so by the time anyone can reply I will probably not be able to to further tests, but maybe someone will be able to reproduce the issues.
I'm using the evaluation server to export some iSCSI targets, and I'm connecting to them from my laptop, which has a fresh installation of SXDE 01/08. I was able to attach to the targets with
iscsiadm add static-config \
iqn.1986-03.com.sun:02:f4281081-c3fc-e448-c8f0-943d8861c9e8,192.168.15.62
iscsiadm add static-config \
iqn.1986-03.com.sun:02:371c6fe7-f0c8-e02c-c865-baef90fb71ce,192.168.15.62
iscsiadm add static-config \
iqn.1986-03.com.sun:02:ee74d5de-3046-6295-f35e-afe60e13db23,192.168.15.62
iscsiadm modify discovery -s enableThis made the appropriate entries appear in /dev/dsk so I could then set up a simple zpool and zfs filesystem with:
zpool create -m none dPool c3t010000144FA70C1400002A0047C2BF73d0 \
c3t010000144FA70C1400002A0047C2BF81d0 c3t010000144FA70C1400002A0047C2BF8Bd0
zfs create -o mountpoint=/data -o sharenfs=on dPool/dataThe general concept I'm testing is having a ZFS-based server using an IP SAN as a growable source of storage, and making the data available to clients over NFS/CIFS or other services. In principle this solution should also allow failover to another server, since all the ZFS data and metadata is in the IP SAN, not on the server. Although not done in this example it should also be possible to run raidz across multiple iSCSI disk arrays. However, it's not been a bed of roses. I've had a lot of errors in dmesg like the following, which I think are causing zfs/zpool commands to stall at times:
Feb 28 14:30:39 F4060 iscsi: [ID 866572 kern.warning] WARNING: iscsi connection(ffffff014f6b6b78) protocol error - received an unsupported opcode:0x41
Feb 28 14:30:41 F4060 iscsi: [ID 158826 kern.warning] WARNING: iscsi connection(10) login failed - failed to receive login response
Feb 28 14:30:41 F4060 scsi_vhci: [ID 734749 kern.warning] WARNING: vhci_scsi_reset 0x1
Feb 28 14:30:41 F4060 iscsi: [ID 339442 kern.notice] NOTICE: iscsi connection failed to set socket optionTCP_NODELAY, SO_RCVBUF or SO_SNDBUF
Feb 28 14:30:41 F4060 iscsi: [ID 933263 kern.notice] NOTICE: iscsi connection(13) unable to connect to target iqn.1986-03.com.sun:02:ee74d5de-3046-6295-f35e-afe60e13db23
Feb 28 14:30:41 F4060 iscsi: [ID 339442 kern.notice] NOTICE: iscsi connection failed to set socket optionTCP_NODELAY, SO_RCVBUF or SO_SNDBUF
Feb 28 14:30:41 F4060 iscsi: [ID 933263 kern.notice] NOTICE: iscsi connection(7) unable to connect to target iqn.1986-03.com.sun:02:f4281081-c3fc-e448-c8f0-943d8861c9e8Does anyone know why a Solaris iSCSI target would send an unsupported opcode (0x41) to a Solaris iSCSI initiator? Surely they should be talking the same language!
The main problems however are with shutdown and start-up. On occasions, I suspect that the ordering of ZFS, iSCSI and network services gets a bit out of sync. On one occasion the laptop even refused to complete the shutdown because it was reporting a continuous stream of console messages like
Feb 27 18:26:37 F4060 iscsi: [ID 933263 kern.notice] NOTICE: iscsi connection(13) unable to connect to target iqn.1986-03.com.sun:02:ee74d5de-3046-6295-f35e
-afe60e13db23
Feb 27 18:26:37 F4060 iscsi: [ID 933263 kern.notice] NOTICE: iscsi connection(10) unable to connect to target iqn.1986-03.com.sun:02:371c6fe7-f0c8-e02c-c865
-baef90fb71ce
Feb 27 18:26:37 F4060 iscsi: [ID 933263 kern.notice] NOTICE: iscsi connection(7) unable to connect to target iqn.1986-03.com.sun:02:f4281081-c3fc-e448-c8f0-
943d8861c9e8I also get these on start-up, where it looks like ZFS tries to load the zpool configuration before iSCSI has found the disks, and even worse, iSCSI is starting up before nwamd has time to do its network auto-magic, and complains that the devices are unavailable.
If these problems sorted themselves out after everything came up, I wouldn't really mind some temporary complaints in the log file, but what I get after a reboot is a working zpool but an unmounted ZFS filesystem! Here is what I have today:
bash-3.2# zpool status
  pool: dPool
state: ONLINE
scrub: scrub completed with 0 errors on Thu Feb 28 14:33:43 2008
config:
        NAME                                     STATE     READ WRITE CKSUM
        dPool                                    ONLINE       0     0     0
          c3t010000144FA70C1400002A0047C2BF73d0  ONLINE       0     0     0
          c3t010000144FA70C1400002A0047C2BF81d0  ONLINE       0     0     0
          c3t010000144FA70C1400002A0047C2BF8Bd0  ONLINE       0     0     0
errors: No known data errors
bash-3.2# zfs list
NAME         USED  AVAIL  REFER  MOUNTPOINT
dPool        480M  28.9G     1K  none
dPool/data   480M  28.9G   480M  /dataThis all looks fine, and you can see that I was even able to scrub the pool data with no problems. But where are the 480MB of data I have put in the /data mountpoint:
bash-3.2# ls /data
bash-3.2# df -h /data
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1d0s0         15G   4.4G    11G    30%    /As you can see, /data is unmounted, causing df to revert to the / filesystem containing the empty /data mountpoint, instead of showing the zpool mount.
Since zfs is supposed to take care of its own mounts rather than using vfstab, I can't use "mount /data" to force this to mount. The only workaround I've found is to export and import the zpool. Then I get the filesystem to reappear:
bash-3.2# df -h /data
Filesystem             size   used  avail capacity  Mounted on
dPool/data              29G   480M    29G     2%    /dataDoes anyone know if these are known issues with snv_79b, and is there a fix available or in the works?
TIA,
Graham

Answer 1

EasyE, Welcome to the discussion area!
(a) Call Apple and get your iMac G5 fixed since it is in the repair extension group. Don't waste your time doing anything else.
(b) This area is for discussing the iMac G4. Since you have an iMac G5 in the future you should post in the iMac G5 discussion area.