Error in verify and rebuild

Question

I've got an XServe RAID with 14 drives. They are configured as two RAID 5 sets, merged into one in the OS.
The RAID is online, but when I do verify the parity data on one of the sets in raidadmin, it comes back with an error event. When I try to do rebuild parity in raidadmin, this also completes with an error event. Before the rebuild, I tried a repair in diskutil, which reported no errors. I've copied the event messages below. Any idea what's wrong or what I should do to fix it?
(Note: there seems to be a bug in raidadmin that always labels the raid sets as set #1 in the verify and rebuilt events, although the operations are taking place on set #2)
Timestamp: 12/13/05 03:33:51 PM
Priority: Info
Controller: Lower Controller
Type: 48
Event ID: 39
Event: RAID Set 1 Verify Started
Description: Verification of the RAID array started.
Timestamp: 12/13/05 04:25:35 PM
Priority: Error
Controller: Lower Controller
Type: 49
Event ID: 40
Event: RAID Set 1 Verify Complete (-200001)
Description: Verification of the RAID array completed.
Timestamp: 12/15/05 11:00:23 AM
Priority: Info
Controller: Lower Controller
Type: 46
Event ID: 41
Event: RAID Set 1 Rebuild Started
Description: The RAID array started rebuilding. This is the result of a spare drive being assigned to the array after a drive failure.
Timestamp: 12/15/05 12:11:08 PM
Priority: Error
Controller: Lower Controller
Type: 47
Event ID: 42
Event: RAID Set 1 Rebuild Complete (-200001)
Description: The RAID array finished rebuilding.
xserve G4 Mac OS X (10.2.x)

Answer 1

I have the exact same issue here:
over the course of several months there were a variety of drive errors of the kind that should not matter, i.e. amber messages, that were recoverable.
A couple of days ago, one drive failed, and the hotspare took over, was automatically rebuilt. The rebuilding reported that data was lost during the rebuild operation. I think "***!" This is a single drive failure, and the whole point of these being "intelligent" storage sub systems is that they should instantly let me know with a RED alert, if any error happens that requires a hardware component to be replaced, or that for some other reason puts data integrity in danger.
So then I run rebuild parity on both arrays (RAID-50 setup), and even the drive that had no non-recoverable drive failure can't finish the parity rebuilding!
So there I have a single drive die since I last completely reformatted and created the RAID-50 array (both at the Xserve and the Mac OS X level), and now I have to fret about our backups which are stored on this RAID?
This comes utterly out of the blue, and I have to wonder what I'm going to tell my clients about what we can use the Xserve RAID for in the future!
The unit is a bit over two years old, and we had one drive fail previously, and since Tiger redefined the RAID-0 software array, and things were a bit iffy, when that happened, we reformatted and set up everything from scratch.
Now we're back at the same place, again. This isn't what I consider fail-safe storage, and hot-swap components.
On the other hand, if we're supposed to replace drives with each non red alert error, then in addition to the failed drive, I'd have to replace four more, which would mean almost half the drives would have had to be replaced in a bit more than two years. At the cost of Apple's drive modules, that's quite a bit.
Further, most drive manufacturers offer two or three year warranties on the drives, but Apple's Xserve RAID has only a one year warranty.
At this point, I don't know if the Xserve RAID is still a trustworthy backup target. And if not that, then what is?