Disk I/O errors within every two weeks

Question

So here's my problem.
We've got 2 servers. First one is the Open Directory Master, iCal and Wiki Server.
The second one is the Replica of the first server. Also this is the server that has the homefolders.
Every night i make a backup of the homefolders to the first server.
Now for the homefolders we have a separate 500GB disk within our Xserves. Both servers have this. Straight out of the factory.
Now the problem is that randomly within every two weeks the homefolder disk gets Disk I/O errors.
The stupid thing is that the Finder thinks it's Okay until you try to do something with the disk.
Now this mostly happens around midnight.Right after the log files changed.
It's also easy to undo: restart the server or remount the disk. Now are both things a little hard to do with servers in production.
But when you do and check the disk using all the tools you know you can't find something wrong with it! They're fine!
But when this happens at midnight and I still haven't fixed this before our editors come in; we have a problem. Because they can't log in.
Now in order to solve this i had to write a script that tries to make and delete a folder on this disk.
If that's possible we make a backup. And if it's not.... reboot the server. Yeah you read it all right.
I have to reboot the server!! I mean come on Apple!!
Are we to take Leopard Server seriously this way?
There is no way that this is acceptable for a server! So here is my plea: "Apple please make a better system." 10.4 was so stable why not use that knowledge and use it in Leopard?
Both servers are running 10.5.1. Both servers have the very same problem.
I'm hoping that 10.5.2 will fix this problem. But i first have to get a service window.... see apple that's normally the time to reboot a server!
Message was edited by: Jasper Siegers

Answer 1

I'd be looking to swap in spare disks -- long before writing a script for this. This could be a thermal condition, or it could be a block-level error, or bus or who-knows-what upstream of the disks. But the first culprit in the line-up here is usually the spindle. And yes, this could be some sort of a software fault.
Modern disks do not last forever. Far from it. Disks fail. They go catatonic. They go off-line. They loose blocks.
And since this is a production environment (based on all of what you're saying here), do look at implementing RAID for your storage. (My usual preference is RAID-10. I tend to avoid RAID-5.)
For system monitoring, if Server Admin or some simple fast coding stuff can't deal with detecting and propagating an error notification, a management package such as [Lithium|http://lithiumcorp.com> might be in order.
But right now, swap the disk. Isolate the error. Protect your data. Then (if the lock-up is not cleared up) start looking at the software stack.
Do move to RAID, regardless. And do ensure you have good archives of your data, as RAID targets data accessibility and not data archival requirements.