Meta Data Lun Corrupted!!!!

Question

Hello All,
Our XSan went down today and upon rebooting the metadata controller it was discovered that the lun that contained the meta data had become corrupt. Whereas the two data luns showed up as CVFS format and had labels the metadata lun prompted an "unreadable, eject initialize ignore" message at startup and in the XSan admin was listed as unlabeled. It was listed as "unknown" format and unlabeled when listed with the other volumes in the command line.
How do I repair this lun? When I try to use CVFSCK command it complains that the lun "mylun2" which was the name of the corrupted lun is unavailable. Is there a disk utility of sorts that repairs a volume (lun) that is CVFS file system?
I see a few command line options but all come with serious warnings. So, I wish to seek guidance first.
A few more things.
1. the metadata lun is a Raid 1 two disk mirror.
2. the backup controller had not been updated from 1.1 to 1.2 and had taken over metadata controlling earlier this week before the first crash which was solved with a restart and the main metadata controller resumed responsibility. both controllers now have 1.2. I intend to have a conversation with the guy who only updated the one controller.
3. both meta data controllers are running 10.4.3 server
4. restarting all of the components of the san did not resolve the issue.
I really don't want to tell this client that all their data on the San is completely lost. They are a very good client and very good people. Any help that you could give would be THOROUGHLY appreciated.
Thank you for your time,
James L. McMahan Jr.
G5 1.8   Mac OS X (10.4.6)  

Answer 1

Uh... okay. If you have a better solution, Will, go for it. The fact of the matter, though, is if your metadata is lost and you don't have a backup (bad idea), you don't have any other option but to start over. There isn't some magic utility to run to make things all better again.
Xsan (or StorNext) is a cluster file system-based SAN, which is dependant on metadata stored on an Xserve RAID. Having redundant controllers isn't going to help with redundancy of his metadata -- it's going to help with redundancy of the controllers. It looks like he had an issue with the primary controller, which failed over to the secondary correctly. Why his primary controller failed, I do not know -- I haven't seen any logs (and even if I had, they might not be that much help to me at this juncture, anyway). The next problem he had was his secondary controller didn't have the same version of Xsan as the primary, which probably made matters worse.
At this point, I don't know what's up with his controllers, but he is stuck with a metadata LUN without a label on it, which is worthless to him. That's rare and unfortunate, but that's why you backup your data. If the metadata is lost, your data isn't accessible. A SAN without a backup isn't any better than a RAID set -- or even a single drive.
Will, while I appreciate your desire to help people, you can't always pull a rabbit out of a hat. Probably the reason why my response is what is a standard response from Apple is because it's the truth. I've put up and maintained many Xsan environments and serve as a Senior Storage Engineer for a large company. I know what I'm talking about, but I am not trying to claim that I know all. If there is someone out that who has a better solution, this is the forum to offer those suggestions, where James can be helped and the rest of us can maybe learn something we didn't know before.
But criticizing me for my response isn't going to help him, either. If you have a better solution, speak up. Otherwise, can it.
Sapridyne