7410 Cluster Recovery

Question

hi,
I'm working on a Try-and-Buy configuration of a Unified Storage 7410 Cluster.
After (intentionally) deleting the only storage pool via the GUI on the primary unit, the secondary cluster member wasn't responsive anymore. As such I tried a reboot. After the reboot, the serial console showed it trying to re-join the cluster with a spinning dash in ascii-art, but it would just carry on spinning and spinning..... As such, I removed the cluster configuration on the primary unit. Via the ssh cli, I told the second unit to factory-reset. Since that took a long time 20+ minutes, I suspected an error and rebooted.
Then, after grub, I was shown the following:
svc.configd: smf(5) database integrity check of:
+/etc/svc/repository.db+
failed.
which I was able to fix by following the instructions and running
/lib/svc/bin/restore_repository
Now, the unit does boot again, however, it hangs at the following screen, that refreshes periodically:
Sun Storage 7410 Configuration
Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
NET-0 <=>  NET-1 <X>  NET-2 <X>  NET-3 <X>
SUNW-MSG-ID: AK-8000-2U, TYPE: Defect, VER: 1, SEVERITY: Major
EVENT-TIME: Thu Apr 23 17:53:21 UTC 2009
PLATFORM: i86pc, CSN: 0912QAF048, HOSTNAME: head2
SOURCE: aktty, REV: 1.51
EVENT-ID: 2c8d4cd4-5e88-ca68-e871-e1ef0af6c040
DESC: The appliance experienced an unrecoverable protocol error while
attempting to join a cluster of appliances.  Refer to
http://sun.com/msg/AK-8000-2U for more information.
AUTO-RESPONSE: No automated response is possible for this defect.
IMPACT: The appliance cannot be configured into the cluster.
REC-ACTION: Reboot the system and attempt to configure the cluster again.
Contact your vendor for support.
ESC-3: Halt   ESC-4: Reboot   ESC-5: Info
For help, see http://www.sun.com/7410/
I've rebooted the box several times, with the primary unit powered on and powered off, but this message stays the same.
I would like to re-attempt the factory reset, however, this secondary unit, does not have an IP address and as such, I cannot log in to the webinterface and make it happen, nor can I access the CLI, as I'm locked inside this refreshing screen.
Is there any way that I can get to the CLI from this screen (or anywhere during the boot sequence) so that I can re-try the factory reset ?
Looking forward to your responses.
Thanks,
Frans

Answer 1

Ok, since there isn't very much documentation on the topic yet, I've played around a bit more.
I've rebooted and via grub I booted into single user mode.
Sniffing around the fileystem, I found a number of ZFS snapshots.
+# zfs list -t snapshot+
NAME                                     USED  AVAIL  REFER  MOUNTPOINT
system/home@install                       18K      -  21.5K  -
system/stash@install                      16K      -    18K  -
system/svc@install                      3.94M      -  4.78M  -
system/[email protected]_1-1.1  19.0M      -  23.8M  -
system/var@install                       844K      -  31.4M  -
So I rolled back to the original (@install) versions:
+# zfs rollback system/home@install+
+# zfs rollback system/stash@install+
+# zfs rollback system/svc@install+
cannot rollback to'system/svc@install': more recent snapshots exist
use '-r' to force deletion of the following snapshots:
system/[email protected]_1-1.1
+# zfs rollback -r system/svc@install+
svc.configd: Fatal error: /etc/svc/repository.db: db error: database is locked
+# zfs rollback -r system/var@install+
And after a reboot, I see:
Sun Storage 7410 Version ak/SUNW,[email protected],1-1.9
Copyright 2009 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Configuring devices.
Checking hardware configuration ... done.
Starting appliance configuration .......... done.
+Press any key to begin configuring appliance: [*]+
Jay !
Now, the version string after the reboot doesn't correspond to the 2008.11.20.0.1_1-1.1 that I was running upon installation, so it seems that I haven't completely restored the system to what it should be, but at least I have some form of control again. I'll try and upload the latest software again and see if we can get it all the way back up and running again.
Frans