Zfs pool I/O failures

Hello,
Been using an external SAS/SATA tray connected to a t5220 using a SAS cable as storage for a media library. The weekly scrub cron failed last week with all disks reporting I/O failures:
zpool status
pool: media_NAS
state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: http://www.sun.com/msg/ZFS-8000-HC
scan: scrub in progress since Thu Apr 30 09:43:00 2015
    2.34T scanned out of 9.59T at 14.7M/s, 143h43m to go
    0 repaired, 24.36% done
config:
        NAME        STATE     READ WRITE CKSUM
        media_NAS   UNAVAIL 10.6K    75     0 experienced I/O failures
          raidz2-0 UNAVAIL 21.1K    10     0 experienced I/O failures
            c6t0d0 UNAVAIL    212     6     0 experienced I/O failures
            c6t1d0 UNAVAIL    216     6     0 experienced I/O failures
            c6t2d0 UNAVAIL    225     6     0 experienced I/O failures
            c6t3d0 UNAVAIL    217     6     0 experienced I/O failures
            c6t4d0 UNAVAIL    202     6     0 experienced I/O failures
            c6t5d0 UNAVAIL    189     6     0 experienced I/O failures
            c6t6d0 UNAVAIL    187     6     0 experienced I/O failures
            c6t7d0 UNAVAIL    219    16     0 experienced I/O failures
            c6t8d0 UNAVAIL    185     6     0 experienced I/O failures
            c6t9d0 UNAVAIL    187     6     0 experienced I/O failures
The console outputs this repeated error:
SUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major
EVENT-TIME: 20
PLATFORM: SUNW,SPARC-Enterprise-T5220, CSN: -, HOSTNAME: t5220-nas
SOURCE: zfs-diagnosis, REV: 1.0
EVENT-ID: e935894e-9ab5-cd4a-c90f-e26ee6a4b764
DESC: The number of I/O errors associated with a ZFS device exceeded acceptable levels.
AUTO-RESPONSE: The device has been offlined and marked as faulted. An attempt will be made to activate a hot spare if available.
IMPACT: Fault tolerance of the pool may be compromised.
REC-ACTION: Use 'fmadm faulty' to provide a more detailed view of this event. Run 'zpool status -x' for more information. Please refer to the associated reference document at http://sun.com/msg/ZFS-8000-FD for the latest service procedures and policies regarding this diagnosis.
Chassis | major: Host detected fault, MSGID: ZFS-8000-FD
/var/adm/messages has an error message for each disk in the data pool, this being the error for sd7:
May 3 16:24:02 t5220-nas scsi: [ID 107833 kern.warning] WARNING: /pci@0/pci@0/p
ci@9/scsi@0/disk@2,0 (sd7):
May 3 16:24:02 t5220-nas       Error for Command: read(10)                Error
Level: Fatal
May 3 16:24:02 t5220-nas scsi: [ID 107833 kern.notice]         Requested Block:
1815064264                Error Block: 1815064264
Have tried rebooting the system and running zpool clear as the zfs link in the console errors suggest. Sometimes the system will reboot fine, other times it requires issuing a break from LOM, because the shutdown command is still trying after more than an hour.   The console usually outputs more messages, as the reboot is completing, basically saying the faulted hardware has been restored, and no additional action is required. A scrub is recommended in the console message. When I check the pool status the previously suspended scrub starts back where it left off:
zpool status
pool: media_NAS
state: ONLINE
scan: scrub in progress since Thu Apr 30 09:43:00 2015
    5.83T scanned out of 9.59T at 165M/s, 6h37m to go
    0 repaired, 60.79% done
config:
        NAME        STATE     READ WRITE CKSUM
        media_NAS   ONLINE       0     0     0
          raidz2-0 ONLINE       0     0     0
            c6t0d0 ONLINE       0     0     0
            c6t1d0 ONLINE       0     0     0
            c6t2d0 ONLINE       0     0     0
            c6t3d0 ONLINE       0     0     0
            c6t4d0 ONLINE       0     0     0
            c6t5d0 ONLINE       0     0     0
            c6t6d0 ONLINE       0     0     0
            c6t7d0 ONLINE       0     0     0
            c6t8d0 ONLINE       0     0     0
            c6t9d0 ONLINE       0     0     0
errors: No known data errors
Then after an hour or two all the disks go back into an I/O error state.   Thought it might be the SAS controller card, PCI slot, or maybe the cable, so tried using the other PCI slot in the riser card first (don't have another cable available).   Now the system is back online and again trying to complete the previous scrub:
zpool status
pool: media_NAS
state: ONLINE
scan: scrub in progress since Thu Apr 30 09:43:00 2015
    5.58T scanned out of 9.59T at 139M/s, 8h26m to go
    0 repaired, 58.14% done
config:
        NAME        STATE     READ WRITE CKSUM
        media_NAS   ONLINE       0     0     0
          raidz2-0 ONLINE       0     0     0
            c6t0d0 ONLINE       0     0     0
            c6t1d0 ONLINE       0     0     0
            c6t2d0 ONLINE       0     0     0
            c6t3d0 ONLINE       0     0     0
            c6t4d0 ONLINE       0     0     0
            c6t5d0 ONLINE       0     0     0
            c6t6d0 ONLINE       0     0     0
            c6t7d0 ONLINE       0     0     0
            c6t8d0 ONLINE       0     0     0
            c6t9d0 ONLINE       0     0     0
errors: No known data errors
the zfs file systems are mounted:
bash# df -h|grep media
media_NAS               14T   493K   6.3T     1%    /media_NAS
media_NAS/archive       14T   784M   6.3T     1%    /media_NAS/archive
media_NAS/exercise      14T    42G   6.3T     1%    /media_NAS/exercise
media_NAS/ext_subs      14T   3.9M   6.3T     1%    /media_NAS/ext_subs
media_NAS/movies        14T   402K   6.3T     1%    /media_NAS/movies
media_NAS/movies/bluray    14T   4.0T   6.3T    39%    /media_NAS/movies/bluray
media_NAS/movies/dvd    14T   585K   6.3T     1%    /media_NAS/movies/dvd
media_NAS/movies/hddvd    14T   176G   6.3T     3%    /media_NAS/movies/hddvd
media_NAS/movies/mythRecordings    14T   329K   6.3T     1%    /media_NAS/movies/mythRecordings
media_NAS/music         14T   347K   6.3T     1%    /media_NAS/music
media_NAS/music/flac    14T    54G   6.3T     1%    /media_NAS/music/flac
media_NAS/mythTV        14T    40G   6.3T     1%    /media_NAS/mythTV
media_NAS/nuc-celeron    14T   731M   6.3T     1%    /media_NAS/nuc-celeron
media_NAS/pictures      14T   5.1M   6.3T     1%    /media_NAS/pictures
media_NAS/television    14T   3.0T   6.3T    33%    /media_NAS/television
but the format command is not seeing any of the disks:
format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
       0. c1t0d0 <SEAGATE-ST9146803SS-0006 cyl 65533 alt 2 hd 2 sec 2187>
          /pci@0/pci@0/pci@2/scsi@0/sd@0,0
       1. c1t1d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@0/pci@0/pci@2/scsi@0/sd@1,0
       2. c1t2d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@0/pci@0/pci@2/scsi@0/sd@2,0
       3. c1t3d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848> solaris
          /pci@0/pci@0/pci@2/scsi@0/sd@3,0
Before moving the card into the other slot in the riser card format saw each disk in the zfs pool.    Not sure why the disks are not seen in format but the zfs pool seems to be available to the OS.    The disks in the attached tray were setup for Solaris to see using the Sun StorageTek RAID Manager, they were passed as 2TB raid0 components to Solaris, and format saw them as available 2TB disks.    Any suggestions as to how to proceed if the scrub completes with the SAS card in the new I/O slot?    Should I force a reconfigure of devices on the next reboot? If the disks fault out again with I/O errors in this slot, the next steps were to try a new SAS card and/or cable. Does that sound reasonable?
Thanks,

Was the system online (and the ZFS pool) too when you moved the card? That might explain why the disks are confused. Obviously, this system is experiencing some higher level problem like a bad card or cable because disks generally don't fall over at the same time. I would let the scrub finish, if possible, and shut the system down. Bring the system to single-user mode, and review the zpool import data around the device enumeration. If the device info looks sane, then import the pool. This should re-read the device info. If the device info is still not available during the zpool import scan, then you need to look at a higher level.
Thanks, Cindy

Similar Messages

Unable to destroy ZFS pool

Hello everyone,
is there any way how to remove suspended ZFS pool when underlying storage has been removed from the OS?
# zpool status test
pool: test
state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: http://www.sun.com/msg/ZFS-8000-HC
scan: none requested
config:
NAME STATE READ WRITE CKSUM
test UNAVAIL 0 0 0 experienced I/O failures
c2t50060E8016068817d2 UNAVAIL 0 0 0 experienced I/O failures
All the zpool operations hang on the system :
# ps -ef |grep zpool
root 5 0 0 May 16 ? 151:42 zpool-rpool
root 19747 1 0 Jun 02 ? 0:00 zpool clear test
root 12714 1 0 Jun 02 ? 0:00 zpool destroy test
root 9450 1 0 Jun 02 ? 0:00 zpool history test
root 13592 1 0 Jun 02 ? 0:00 zpool destroy test
root 19684 1 0 May 30 ? 0:00 zpool destroy -f test
root 9166 0 0 May 30 ? 0:07 zpool-test
root 18514 1 0 Jun 02 ? 0:00 zpool destroy -f test
root 3327 0 0 May 30 ? 4:25 zpool-OScopy
root 7332 1 0 May 30 ? 0:00 zpool clear test
root 5016 1 0 Jun 02 ? 0:00 zpool online test c2t50060E8016068817d2
root 25080 1 0 Jun 01 ? 0:00 zpool clear test
root 23451 1 0 01:26:57 ? 0:00 zpool destroy test
Disk is not more visible on the system:
# ls -la /dev/dsk/c2t50060e8016068817d2*
/dev/dsk/c2t50060e8016068817d2*: No such file
Any suggestions how to remove the pool without preforming reboot?
Thanks in advance for any help

I had the same issue recently (solaris 11.1 system) where I deleted a LUN from the SAN before destroying a zpool on it. The pool was suspended and all operations on it failed. I also tried a zpool clear but that did not work and additionally, all other operations on other zpools were also hanging after that. The "workaround" was to delete /etc/zpool.cache and reboot the system.
I raised an SR and a feature request for this but to my knowledge, nothing has been done yet. There is note 1457074.1 on MOS that describes this for Solaris 10 (including a bug and patch) and claims that solaris 11 is not affected.
good luck
bjoern

SFTP chroot from non-global zone to zfs pool

Hi,
I am unable to create an SFTP chroot inside a zone to a shared folder on the global zone.
Inside the global zone:
I have created a zfs pool (rpool/data) and then mounted it to /data.
I then created some shared folders: /data/sftp/ipl/import and /data/sftp/ipl/export
I then created a non-global zone and added a file system that loops back to /data.
Inside the zone:
I then did the ususal stuff to create a chroot sftp user, similar to: http://nixinfra.blogspot.com.au/2012/12/openssh-chroot-sftp-setup-in-linux.html
I modifed the /etc/ssh/sshd_config file and hard wired the ChrootDirectory to /data/sftp/ipl.
When I attempt to sftp into the zone an error message is displayed in the zone -> fatal: bad ownership or modes for chroot directory /data/
Multiple web sites warn that folder ownership and access privileges is important. However, issuing chown -R root:iplgroup /data made no difference. Perhaps it is something todo with the fact the folders were created in the global zone?
If I create a simple shared folder inside the zone it works, e.g. /data3/ftp/ipl......ChrootDirectory => /data3/ftp/ipl
If I use the users home directory it works. eg /export/home/sftpuser......ChrootDirectory => %h
FYI. The reason for having a ZFS shared folder is to allow separate SFTP and FTP zones and a common/shared data repository for FTP and SFTP exchanges with remote systems. e.g. One remote client pushes data to the FTP server. A second remote client pulls the data via SFTP. Having separate zones increases security?
Any help would be appreciated to solve this issue.
Regards John

sanjaykumarfromsymantec wrote:
Hi,
I want to do IPC between inter-zones ( commnication between processes running two different zones). So what are the different techniques can be used. I am not interested in TCP/IP ( AF_INET) sockets.Zones are designed to prevent most visibility between non-global zones and other zones. So network communication (like you might use between two physical machines) are the most common method.
You could mount a global zone filesystem into multiple non-global zones (via lofs) and have your programs push data there. But you'll probably have to poll for updates. I'm not certain that's easier or better than network communication.
Darren

Solaris 10 upgrade and zfs pool import

Hello folks,
I am currently running "Solaris 10 5/08 s10x_u5wos_10 X86" on a Sun Thumper box where two drives are mirrored UFS boot volume and the rest is used in ZFS pools. I would like to upgrade my system to "10/08 s10x_u6wos_07b X86" to be able to use ZFS for the boot volume. I've seen documentation that describes how to break the mirror, create new BE and so on. This system is only being used as iSCSI target for windows systems so there is really nothing on the box that i need other then my zfs pools. Could i simply pop the DVD in and perform a clean install and select my current UFS drives as my install location, basically telling Solaris to wipe them clean and create an rpool out of them. Once the installation is complete, would i be able to import my existing zfs pools ?
Thank you very much

Sure. As long as you don't write over any of the disks in your ZFS pool you should be fine.
Darren

Large number of Transport errors on ZFS pool

This is sort of a continuation of thread:
Issues with HBA and ZFS
But since it is a separate question thought I'd start a new thread.
Because of a bug in 11.1, I had to downgrade to 10_U11. Using an LSI 9207-8i HBA (SAS2308 chipset). I have no errors on my pools but i consistently see errors when trying to read from the disks. They are always Retryable or Reset. All in all the system functions but as I started testing I am seeing a lot of errors in IOSTAT.
bash-3.2# iostat -exmn
extended device statistics ---- errors ---
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot device
0.1 0.2 1.0 28.9 0.0 0.0 0.0 41.8 0 1 0 0 1489 1489 c0t5000C500599DDBB3d0
0.0 0.7 0.2 75.0 0.0 0.0 21.2 63.4 1 1 0 1 679 680 c0t5000C500420F6833d0
0.0 0.7 0.3 74.6 0.0 0.0 20.9 69.8 1 1 0 0 895 895 c0t5000C500420CDFD3d0
0.0 0.6 0.4 75.5 0.0 0.0 26.7 73.7 1 1 0 1 998 999 c0t5000C500420FB3E3d0
0.0 0.6 0.4 75.3 0.0 0.0 18.3 68.7 0 1 0 1 877 878 c0t5000C500420F5C43d0
0.0 0.0 0.2 0.7 0.0 0.0 0.0 2.1 0 0 0 0 0 0 c0t5000C500420CE623d0
0.0 0.6 0.3 76.0 0.0 0.0 20.7 67.8 0 1 0 0 638 638 c0t5000C500420CD537d0
0.0 0.6 0.2 74.9 0.0 0.0 24.6 72.6 1 1 0 0 638 638 c0t5000C5004210A687d0
0.0 0.6 0.3 76.2 0.0 0.0 20.0 78.4 1 1 0 1 858 859 c0t5000C5004210A4C7d0
0.0 0.6 0.2 74.3 0.0 0.0 22.8 69.1 0 1 0 0 648 648 c0t5000C500420C5E27d0
0.6 43.8 21.3 96.8 0.0 0.0 0.1 0.6 0 1 0 14 144 158 c0t5000C500420CDED7d0
0.0 0.6 0.3 75.7 0.0 0.0 23.0 67.6 1 1 0 2 890 892 c0t5000C500420C5E1Bd0
0.0 0.6 0.3 73.9 0.0 0.0 28.6 66.5 1 1 0 0 841 841 c0t5000C500420C602Bd0
0.0 0.6 0.3 73.6 0.0 0.0 25.5 65.7 0 1 0 0 678 678 c0t5000C500420D013Bd0
0.0 0.6 0.3 76.5 0.0 0.0 23.5 74.9 1 1 0 0 651 651 c0t5000C500420C50DBd0
0.0 0.6 0.7 70.1 0.0 0.1 22.9 82.9 1 1 0 2 1153 1155 c0t5000C500420F5DCBd0
0.0 0.6 0.4 75.3 0.0 0.0 19.2 58.8 0 1 0 1 682 683 c0t5000C500420CE86Bd0
0.0 0.0 0.2 0.7 0.0 0.0 0.0 1.9 0 0 0 0 0 0 c0t5000C500420F3EDBd0
0.1 0.2 1.0 26.5 0.0 0.0 0.0 41.9 0 1 0 0 1511 1511 c0t5000C500599E027Fd0
2.2 0.3 133.9 28.2 0.0 0.0 0.0 4.4 0 1 0 17 1342 1359 c0t5000C500599DD9DFd0
0.1 0.3 1.1 29.2 0.0 0.0 0.2 34.1 0 1 0 2 1498 1500 c0t5000C500599DD97Fd0
0.0 0.6 0.3 75.6 0.0 0.0 22.6 71.4 0 1 0 0 677 677 c0t5000C500420C51BFd0
0.0 0.6 0.3 74.8 0.0 0.1 28.6 83.8 1 1 0 0 876 876 c0t5000C5004210A64Fd0
0.6 43.8 18.4 96.9 0.0 0.0 0.1 0.6 0 1 0 5 154 159 c0t5000C500420CE4AFd0
Mar 12 2013 17:03:34.645205745 ereport.fs.zfs.io
nvlist version: 0
     class = ereport.fs.zfs.io
     ena = 0x114ff5c491a00c01
     detector = (embedded nvlist)
     nvlist version: 0
          version = 0x0
          scheme = zfs
          pool = 0x53f64e2baa9805c9
          vdev = 0x125ce3ac57ffb535
     (end detector)
     pool = SATA_Pool
     pool_guid = 0x53f64e2baa9805c9
     pool_context = 0
     pool_failmode = wait
     vdev_guid = 0x125ce3ac57ffb535
     vdev_type = disk
     vdev_path = /dev/dsk/c0t5000C500599DD97Fd0s0
     vdev_devid = id1,sd@n5000c500599dd97f/a
     parent_guid = 0xcf0109972ceae52c
     parent_type = mirror
     zio_err = 5
     zio_offset = 0x1d500000
     zio_size = 0xf1000
     zio_objset = 0x12
     zio_object = 0x0
     zio_level = -2
     zio_blkid = 0x452
     __ttl = 0x1
     __tod = 0x513fa636 0x26750ef1
I know all of these drives are not bad and I have confirmed they are all running the latest firmware and correct sector size, 512 (ashift 9). I am thinking it is some sort of compatibility with this new HBA but have no way of verifying. Anyone have any suggestions?
Edited by: 991704 on Mar 12, 2013 12:45 PM

There must be something small I am missing. We have another system configured nearly the same (same server and HBA, different drives) and it functions. I've gone through the recommended storage practices guide. The only item I have not been able to verify is
"Confirm that your controller honors cache flush commands so that you know your data is safely written, which is important before changing the pool's devices or splitting a mirrored storage pool. This is generally not a problem on Oracle/Sun hardware, but it is good practice to confirm that your hardware's cache flushing setting is enabled."
How can I confirm this? As far as I know these HBAs are simply HBAs. No battery backup. No on-board memory. The 9207 doesn't even offer RAID.
Edited by: 991704 on Mar 15, 2013 12:33 PM

Create ZONE in ZFS pool solaris10

Hi Gurus,
I'm reading some solaris 10 tutorials about ZFS and Zones. Is it possible to create a new storage pool using my current hard disk in which I installed solaris???
I'm a bit new in Solaris, I have a SPARC box in which I'm learnin about solaris 10. I have installed Solaris 10 using ZFS file system. I think my box only have 1 disk but not sure. I see 46 GB of free space running "df -kh " command
I run "format" command, this is the output
root@orclidm # format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@1,0
Specify disk (enter its number):
zpool list "display this:"
root@orclidm # zpool list
NAME SIZE ALLOC FREE CAP HEALTH ALTROOT
rpool 68G 13.1G 54.9G 19% ONLINE -
zfs list "display this:"
root@orclidm # zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 21.3G 45.6G 106K /rpool
rpool/ROOT 11.6G 45.6G 31K legacy
rpool/ROOT/s10s_u10wos_17b 11.6G 45.6G 11.6G /
rpool/dump 1.50G 45.6G 1.50G -
rpool/export 66K 45.6G 32K /export
rpool/export/home 34K 45.6G 34K /export/home
rpool/swap 8.25G 53.9G 16K -
I read in a tutorial that when you create a zpool you need to specify an empty hard disk, is that correct?
Please point me on the best approach to create zones using zfs pools.
Regards

manin21 wrote:
Hi Gurus,
I'm reading some solaris 10 tutorials about ZFS and Zones. Is it possible to create a new storage pool using my current hard disk in which I installed solaris???IF you have a spare partition you may use that.
>
I'm a bit new in Solaris, I have a SPARC box in which I'm learnin about solaris 10. I have installed Solaris 10 using ZFS file system. I think my box only have 1 disk but not sure. I see 46 GB of free space running "df -kh " command
I run "format" command, this is the output
root@orclidm # format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@1,0
Specify disk (enter its number):
This shows two disks. In a production setup you might mirror this.
zpool list "display this:"
root@orclidm # zpool list
NAME SIZE ALLOC FREE CAP HEALTH ALTROOT
rpool 68G 13.1G 54.9G 19% ONLINE -
The command:
zpool status
would show you what devices you are using
zfs list "display this:"
root@orclidm # zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 21.3G 45.6G 106K /rpool
rpool/ROOT 11.6G 45.6G 31K legacy
rpool/ROOT/s10s_u10wos_17b 11.6G 45.6G 11.6G /
rpool/dump 1.50G 45.6G 1.50G -
rpool/export 66K 45.6G 32K /export
rpool/export/home 34K 45.6G 34K /export/home
rpool/swap 8.25G 53.9G 16K -
I read in a tutorial that when you create a zpool you need to specify an empty hard disk, is that correct?
No.
You can use partions/slices instead. A zone storage pool is composed of one or more devices; each device can be a a whole disk, disk slice or even a file if i remember correctly ( .... but you really dont want to use a file normally).
Please point me on the best approach to create zones using zfs pools.
RegardsYour storage rpool is 68GB in size on a 72GB disk .... therefore the disk is full up and their is no space for another zfs pool. If zpool status shows your disk is mirrored by zfs that is that. Otherwise you may choose to create a storage pool on the other disk (not best production practice).
often one creates a zfs filesystem out of an existing filesystem.
zfs create -o mountpoint=/zones rpool/zones
zfs create rpool/zones/myzone
Then use zonepath=/zones/myzone creating the zone.
- I was googling to cross check my answer ... the following blog has an example but it is a little old and may be opensolaris orientated.
https://blogs.oracle.com/DanX/entry/solaris_zfs_and_zones_simple
Authorative information is at http://docs.oracle.com, notably:
http://docs.oracle.com/cd/E23823_01/index.html
http://docs.oracle.com/cd/E23823_01/html/819-5461/index.html
http://docs.oracle.com/cd/E18752_01/html/817-1592/index.html

ISCSI array died, held ZFS pool. Now box han

I was doing some iSCSI testing and, on an x86 EM64T server running an out-of-the box install of Solaris 10u5, created a ZFS pool on two RAID-0 arrays on an IBM DS300 iSCSI enclosure.
One of the disks in the array died, the DS300 got really flaky, and now the Solaris box gets hung in boot. It looks like it's trying to mount the ZFS filesystems. The box has two ZFS pools, or had two, anyway. The other ZFS pool has some VirtualBox images filling it.
Originally, I got a few iSCSI target offline messages on the console, so I booted to failsafe and tried to run iscsiadm to remove the targets, but that wouldn't work. So I just removed the contents of /etc/iscsi and all the iSCSI instances in /etc/path_to_inst on the root drive.
Now the box hangs with no error messages.
Anyone have any ideas what to do next? I'm willing to nuke the iSCSI ZFS pool as it's effectively gone anyway, but I would like to save the VirtualBox ZFS pool, if possible. But they are all test images, so I don't have to save them. The host itself is a test host with nothing irreplaceable on it, so I could just reinstall Solaris. But I'd prefer to figure out how to save it, even if only for the learning experience.

Try this. Disconnect the iSCSI drives completely, then boot. My fallback plan on zfs if things get screwed up is to physically disconnect the zfs drives so that solaris doesn't see them on boot. It marks them failed and should boot. Once it's up, zpool destroy the pools WITH THE DRIVES DISCONNECTED so that it doesn't think there's a pool anymore. THEN reconnect the drives and try to do a "zpool import -f".
The pools that are on intact drives should be still ok. In theory :)
BTW, if you removed devices, you probably should do a reconfiguration boot (create a /a/reconfigure in failsafe mode) and make sure the devices gets reprobed. Does the thing boot in single user ( pass -s after the multiboot line in grub )? If it does, you can disable the iscsi svcs with "svcadm disable network/iscsi_initiator; svcadm disable iscsitgt".

Replace FC Card and ZFS Pools

I have to replace a Qlogic ISP2200 dual port Fibre Channel card with a new card in a V480 server. I have 2 ZFS Pools that mount via that card. Would I have to export and import the ZFS pools when replacing the card? I've read you have to when moving the pools to a different server.
Naturally the World Wide Number (WWN) would be different on the new FC card and other than changing my SAN switch zone information I'm not sure how ZFS would deal with this situation. The storage itself would not change.
Any ideas are welcome.
Running Solaris 10 (11/06) with kernel patch 125100-07
Thanks,
Chris

I have to replace a Qlogic ISP2200 dual port Fibre Channel card with a new card in a V480 server. I have 2 ZFS Pools that mount via that card. Would I have to export and import the ZFS pools when replacing the card? I've read you have to when moving the pools to a different server.
Naturally the World Wide Number (WWN) would be different on the new FC card and other than changing my SAN switch zone information I'm not sure how ZFS would deal with this situation. The storage itself would not change.
Any ideas are welcome.
Running Solaris 10 (11/06) with kernel patch 125100-07
Thanks,
Chris

[Solved] Can't Import ZFS Pool as /dev/disk/by-id

I have a 4 disk raidz1 pool "data" made up of 3TB disks. Each disk is so that that partition 1 is a 2GB swap partition, and partition 2 is the rest of the drive. The zpool was built out of /dev/disk/by-id(s) pointing to the second partition.
# lsblk -i
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 2.7T 0 disk
|-sda1 8:1 0 2G 0 part
`-sda2 8:2 0 2.7T 0 part
sdb 8:16 0 2.7T 0 disk
|-sdb1 8:17 0 2G 0 part
`-sdb2 8:18 0 2.7T 0 part
sdc 8:32 0 2.7T 0 disk
|-sdc1 8:33 0 2G 0 part
`-sdc2 8:34 0 2.7T 0 part
sdd 8:48 0 2.7T 0 disk
|-sdd1 8:49 0 2G 0 part
`-sdd2 8:50 0 2.7T 0 part
sde 8:64 1 14.9G 0 disk
|-sde1 8:65 1 100M 0 part /boot
`-sde2 8:66 1 3G 0 part /
I had a strange disk failure where the controller one one of the drives flaked out and caused my zpool not to come online after a reboot, and I had to zpool export data/zpool import data to get the zpool put back together. However, now it is fixed, but my drives are now identified by their device name:
[root@osiris disk]# zpool status
pool: data
state: ONLINE
scan: resilvered 36K in 0h0m with 0 errors on Wed Aug 13 22:37:19 2014
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
sdc2 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
errors: No known data errors
If I try to import by-id without a zpool name, I get this (its trying to import the disks, not the partitions):
cannot import 'data': one or more devices is currently unavailable
[root@osiris disk]# zpool import -d /dev/disk/by-id/
pool: data
id: 16401462993758165592
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
config:
data FAULTED corrupted data
raidz1-0 ONLINE
ata-ST3000DM001-1CH166_Z1F28ZJX UNAVAIL corrupted data
ata-ST3000DM001-1CH166_Z1F0XAXV UNAVAIL corrupted data
ata-ST3000DM001-1CH166_Z1F108YC UNAVAIL corrupted data
ata-ST3000DM001-1CH166_Z1F12FJZ UNAVAIL corrupted data
[root@osiris disk]# zpool status
no pools available
... and the import doesn't succeed.
If I put the pool name at the end, I get:
[root@osiris disk]# zpool import -d /dev/disk/by-id/ data
cannot import 'data': one or more devices is currently unavailable
Yet, if I do the same thing with the /dev/disk/by-partuuid paths, it seems to work fine (other than the fact that I don't want partuuids). Presumably because there are no entries here for entire disks.
[root@osiris disk]# zpool import -d /dev/disk/by-partuuid/ data
[root@osiris disk]# zpool status
pool: data
state: ONLINE
scan: resilvered 36K in 0h0m with 0 errors on Wed Aug 13 22:37:19 2014
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
d8bd1ef5-fab9-4d47-8d30-a031de9cd368 ONLINE 0 0 0
fbe63a02-0976-42ed-8ecb-10f1506625f6 ONLINE 0 0 0
3d1c9279-0708-475d-aa0c-545c98408117 ONLINE 0 0 0
a2d9067c-85b9-45ea-8a23-350123211140 ONLINE 0 0 0
errors: No known data errors
As another approach, I tried to offline and replace sda2 with /dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F28ZJX-part2, but that doesn't work either:
[root@osiris disk]# zpool offline data sda2
[root@osiris disk]# zpool status
pool: data
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: resilvered 36K in 0h0m with 0 errors on Wed Aug 13 22:37:19 2014
config:
NAME STATE READ WRITE CKSUM
data DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
sda2 OFFLINE 0 0 0
sdb2 ONLINE 0 0 0
sdc2 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
errors: No known data errors
[root@osiris disk]# zpool replace data sda2 /dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F28ZJX-part2
invalid vdev specification
use '-f' to override the following errors:
/dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F28ZJX-part2 is part of active pool 'data'
[root@osiris disk]# zpool replace -f data sda2 /dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F28ZJX-part2
invalid vdev specification
the following errors must be manually repaired:
/dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F28ZJX-part2 is part of active pool 'data'
I would appreciate if anyone else had any suggestions/workarounds on how to fix this
As I was typing this up, I stumbled upon a solution by deleting the symlinks that pointed to entire devices in /dev/disk/by-id (ata-* and wwn*). I then was able to do a zpool import -d /dev/disk/by-id data and it pulled in the partition 2's. It persisted after a reboot and my symlinks were automatically regenerated when the system came back up:
[root@osiris server]# zpool status
pool: data
state: ONLINE
scan: resilvered 36K in 0h0m with 0 errors on Wed Aug 13 23:06:46 2014
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F28ZJX-part2 ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F0XAXV-part2 ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F108YC-part2 ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F12FJZ-part2 ONLINE 0 0 0
It would appear to be an issue with specifically importing non-whole devices by-id. Although this was mainly rambling and no longer a question, hopefully this might help someone having issues re-importing a zpool by /dev/disk/by-id.
Matt

This just saved my morning Thank you!
I was using Ubuntu 14.04 and after an upgrade to 3.13.0-43-generic it somehow broke... Anyhow now the zpool survives restarts again and I don't have to import it every time using partuuids.

Can't get ZFS Pool to validate in HAStoragePlus

Hello.
We rebuilt our cluster with Solaris 10 U6 with Sun Cluster 3.2 U1.
When I was running U5, we never had this issue, but with U6, I can't get the system to validate properly the zpool resource to the resource group.
I am running the following commands:
zpool create -f tank raidz2 c2t0d0 c2t1d0 c2t2d0 c2t3d0 c3t0d0 c3t1d0 c3t2d0 c3t3d0 spare c2t4d0
zfs set mountpoint=/share tank
These commands build my zpool, zpool status comes back good.
I then run
clresource create -g tank_rg -t SUNW.HAStoragePlus -p Zpools=tank hastorage_rs
I get the following output:
clresource: mbfilestor1 - : no error
clresource: (C189917) VALIDATE on resource storage_rs, resource group tank_rg, exited with non-zero exit status.
clresource: (C720144) Validation of resource storage_rs in resource group tank_rg on node mbfilestor1 failed.
clresource: (C891200) Dec 2 10:27:00 mbfilestor1 SC[SUNW.HAStoragePlus:6,tank_rg,storage_rs,hastorageplus_validate]: : no error
Dec 2 10:27:00 mbfilestor1 Cluster.RGM.rgmd: VALIDATE failed on resource <storage_rs>, resource group <tank_rg>, time used: 0% of timeout <1800, seconds>
Failed to create resource "storage_rs".
My resource group and logical host all work no problems, and when I ran this command on the older version of Solaris, it worked no problem. Is this a problem with the newer version of Solaris only?
I though maybe downloading the most up to date patches would fix this, but it didn't.
I did notice this in my messages:
Dec 2 10:26:58 mbfilestor1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_validate> for resource <storage_rs>, resource group <tank_rg>, node <mbfilestor1>, timeout <1800> seconds
Dec 2 10:26:58 mbfilestor1 Cluster.RGM.rgmd: [ID 616562 daemon.notice] 9 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_validate>:tag=<tank_rg.storage_rs.2>: Calling security_clnt_connect(..., host=<mbfilestor1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Dec 2 10:27:00 mbfilestor1 SC[SUNW.HAStoragePlus:6,tank_rg,storage_rs,hastorageplus_validate]: [ID 471757 daemon.error] : no error
Dec 2 10:27:00 mbfilestor1 Cluster.RGM.rgmd: [ID 699104 daemon.error] VALIDATE failed on resource <storage_rs>, resource group <tank_rg>, time used: 0% of timeout <1800, seconds>
Any ideas, or should I put in a bug fix request with Sun?

Hi,
Thanks, I ended up just going back to Solaris 10 U5. It was too critical to get back up and running, and I got tired of messing with it, so I ended up going back. Everything is working like it should. I may try to do a LU on the server and see what happens. Maybe the pools and cluster resources will be fine.
Edited by: mbunixadm on Dec 15, 2008 9:09 AM

Install Solaris 11 on a RAID-1 mirror ZFS pool?

Hi all,
Sorry if this question has been asked here before.
I've searched docs.oracle.com about Solaris 11 and didn't see any related info.
I am installing the new Solaris 11 on my Dell Tower desktop workstation with two 3TB SATA hard drives.
I am planning to construct one RAID-1 mirror with ZFS just like previous Solaris 10 x86 installation.
After using lots of installation media of Solaris 11, I didn't find the options to create RAID-1 mirror on ZFS root partition / pool.
Could someone give me a hint or I must roll back to Solaris 10 again?
Thanks in advance.

Yes, it looks like this on a SPARC system:
ok boot net:dhcp
Boot device: /pci@780/pci@0/pci@1/network@0:dhcp File and args:
1000 Mbps full duplex Link up
Timed out waiting for BOOTP/DHCP reply
<time unavailable> wanboot info: WAN boot messages->console
<time unavailable> wanboot info: configuring /pci@780/pci@0/pci@1/network@0:dhcp
1000 Mbps full duplex Link up
<time unavailable> wanboot info: Starting DHCP configuration
<time unavailable> wanboot info: DHCP configuration succeeded
<time unavailable> wanboot progress: wanbootfs: Read 368 of 368 kB (100%)
<time unavailable> wanboot info: wanbootfs: Download complete
Mon Jul 1 14:28:03 wanboot progress: miniroot: Read 249370 of 249370 kB (100%)
Mon Jul 1 14:28:03 wanboot info: miniroot: Download complete
SunOS Release 5.11 Version 11.1 64-bit
Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved.
Remounting root read/write
Welcome to the Oracle Solaris installation menu
        1 Install Oracle Solaris
        2 Install Additional Drivers
        3 Shell
        4 Terminal type (currently xterm)
        5 Reboot

ZFS pool frequently going offline

I am setting up some servers with ZFS raids and and finding that all of them are suffering from I/O errors that cause the pool to go offline (and when that happens everything freezes and I have to power cycle... then everything boots up fine).
T1000, V245, and V240 systems all exhibit the same behavior.
Root is mirrored ZFS.
The raid is configure as one big LUN (3 to 8 TB depending on system) and that lun is the entire pool. In other words, there is no ZFS redundancy. My thinking was I would let the raid handle that.
Based on some searches I decided to try setting
set sd:sd_max_throttle=20
in /etc/system and rebooting, but that made no difference.
My sense is that the troubles start when there is a lot of activity. I ran these for many days with light activity and no problems. Once I started migrating the data over from the old systems did the problems start. Here is a typical error log:
Jun 6 16:13:15 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1 (mpt3):
Jun 6 16:13:15 newserver Connected command timeout for Target 0.
Jun 6 16:13:15 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1 (mpt3):
Jun 6 16:13:15 newserver Target 0 reducing sync. transfer rate
Jun 6 16:13:16 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:16 newserver SCSI transport failed: reason 'reset': retrying command
Jun 6 16:13:19 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:19 newserver Error for Command: read(10) Error Level: Retryable
Jun 6 16:13:19 newserver scsi: Requested Block: 182765312 Error Block: 182765312
Jun 6 16:13:19 newserver scsi: Vendor: IFT Serial Number: 086A557D-00
Jun 6 16:13:19 newserver scsi: Sense Key: Unit Attention
Jun 6 16:13:19 newserver scsi: ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0
Jun 6 16:13:19 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:19 newserver incomplete read- retrying
Jun 6 16:13:20 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:20 newserver incomplete write- retrying
Jun 6 16:13:20 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:20 newserver incomplete write- retrying
Jun 6 16:13:20 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:20 newserver incomplete write- retrying
<... ~80 similar lines deleted ...>
Jun 6 16:13:21 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:21 newserver incomplete read- retrying
Jun 6 16:13:21 newserver scsi: WARNING: /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0 (sd2):
Jun 6 16:13:21 newserver incomplete read- giving up
At this point everything is hung and I am forced to power cycle.
I'm very confused on how to proceed with this.... since this is happening on all three systems I an reluctant to blame the hardware.
I would be very grateful to any suggestions on how to get out from under this!
Thanks,
David C

which s10 are you running? You could try increasing the timeout value and see if that helps (see mpt(7d) - mpt-on-bus-time). It could be that when the raid controller is busy, it may take longer to service something that it is trying to correct. I've seen drives just go out to lunch for a while (presumably, the SMART firmware is doing something) and comes back fine (but the delay in response causes problems).

How to recover ZFS pool after re-installation of Solaris10

I have two disks(c1t0d0-disk1, c1t1d0-disk2) and I installed Solaris 10(fresh installation) in a slice of disk1. On the other hand, I have ZFS filesystem in a slice of disk2.
I would like to restore old ZFS filesystem in disk2.
How do I recover old ZFS filesytem?
Thank you.

I think you meant "zpool import" instead of "zfs import".
See the zpool man page.

ZFS pool, raidz

Hello,
I have a zfspool that has for disks in raidz. Is there a way to add to this pool? I understand that 'attach' is only for mirror sets. I have two more disks I want to add to the pool. If I just do an add, I get a mismatch as I would have a 4 way and a 2way raidz in the same pool.
What I have;
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
c4t8d0 ONLINE 0 0 0
c4t9d0 ONLINE 0 0 0
c4t10d0 ONLINE 0 0 0
What I tried;
# zpool add tank raidz c4t11d0 c4t12d0
invalid vdev specification
use '-f' to override the following errors:
mismatched replication level: pool uses 4-way raidz and new vdev uses 2-way raidz
Is there no way to increment this pool now that it is raidz other than adding devices four at a time?

I don't believe you can extend a raidz vdev.
But you can add new vdevs to the pool.
So you could add the 2 disks as a mirrored pair vdev

Poor performance on SSD-backed ZFS pools in mirror configuration only

We are currently debugging a system with 12x Toshiba 512GB SSDs on Solaris 11.1 (Oracle Solaris 11.1.10.5.0).
The system has each SSD directly attached to a dedicated port on an LSI 9206-16e card, so there should be full bandwidth to each device all the way through the SAS and PCIe bus.
We're running dd write-tests from /dev/zero to a test filesystem. We've tried most tunables with no success; at the moment a pool of 6 mirrors always tops out at a meagre 750 MB/s. CPU utilisation is actually quite low whilst this is happening.
By contrast, if we create a pool of 3 SSDs in a stripe, we get write speeds of around 1.5 GB/s.
A pool with a single SSD in it gives write speeds of around 480 MB/s, which is what we expect.
As soon as we put the SSDs into a mirror, the performance drops off, approximately in half.
Interestingly, we have also tried offlining half the drives in the pool using 'zpool offline testpool <dev>'. When we do this, the performance picks up to the expected levels.
Is there a bug in the mirror vdev code and/or a tunable for vdevs that we've missed?

Hi Tim,
Yes, very curious indeed. I would expect the mirrored configuration to perform better not slower.
Most of my resources who could look at this are tied up in meetings all week but I have a few
questions:
1. What kind of write workload is this? Can you provide your test dd syntax?
2. Do you see any SSD related errors in /var/adm/messages?
3. Any unusual SSD temp readings?
Thanks, Cindy

Zfs pool I/O failures

Similar Messages

Maybe you are looking for