XSERVE 10.4.7 failed start up errors best practice/recomnd Prev Maintenan?

Question

I have an Xserve, RAID 5 OS volume and data volumes stripped across the RAID 5 (with parity). The only other piece of software is Dantz Retrospect. Preventive maintenance was run 5/30 - which I do in this order. Reboot, see if server comes up clean, run disk warrior, repair permissions (both OS/Data volumes). All was clean - server has been up and running fine until 8/18. Rebooted server - was going to run disk warrior - server did hang on reboot and would not restart. Powered down hard, tried again - No good. Ran Disk Warrior produced a report but didn't make any changes - and it gave me a ton of errors stating: OS files chown/egripm tar grep etc - and dozens of other files under the /usr/bin sbin and usr/shared where gone - the errors were "the link file no longer points to the original file tigeros/usr/bin". It is almost as if I lost the entire File system - all data was intact on data volume. The server is on a UPS but did not have powerchute installed at the time. It has been perfectly stable. Nothing additional was installed . The only thing I can think of is power loss that exceeded the run time of the UPS and dropped the server like a rock - blowing out the file system. Any ideas/thoughts?
I reinstalled the OS - and run disk tools - which shows all is ok - is that proof enough that the OS remains intact and safe for a reboot?
What could have gone wrong?
What is recommended for preventive maintenance on 10.4.7 running on an Xserve? I can't afford to have this reoccur if I can at all help - that is why I am generally pro-active with Preventive Maintenance.
Alsoft - recommends that I run their product every 3-4 weeks to prevent such problems. Should I shy away from third party tools?
Thank you all!
SGM

Answer 1

I really want to thank you for your response and sharing with me some really very good advice and info! I believe I need to clarify something. I am a server admin have 5 certs in Novell, Windows and Cisco certified – and I do not just go around rebooting servers. I am however, trying (and your response is quite helpful), to find best practices for working with XServe and Tiger Server. I have a bit of experience with many other Oss – and don’t normally recommend rebooting for the heck of it. I use to have Novell boxes literally up for years.
My intention is to establish a standardized means of keeping OSX servers running with as few problems as possible. On this particular Xserve, I was to bring it down (because it was recommended to be a good practice), to do PM. By Preventive Maintenance, I did mean Disk Warrior/repair permissions. Now, before I would just run any disk utility (and yes – until now), I would reach for Disk Warrior for the reason that their tech support is usually quite good. I have spoken with them since the first release of their product – and perhaps (although not the best idea), trusted their recommendations.
So – I would reboot the server BEFORE doing any PM, only to make sure and prove that the server comes up clean so I wouldn’t be blamed for it, if it did not come up clean due to performing maintenance. That being said, I did run Disk Warrior back around 5/30/06 and I really had very few errors, the server did come up fine. As mentioned, it was on a UPS (less the PowerChute software at the time).
What I found was upon reboot – the server would not reboot. I proceeded to boot from the OS install CD, Disk Tools said it could not repair the problem. Secondly, I went to Disk Warrior – to run a report which did show that usr/bin and sbin contents were gone. The data volume was completely intact. So my first concern or question would be – what on earth could have caused this – a power fluctuation or the server possibly exceeding the run time – and dropping like a rock? Have you ever seen this? I just find I sleep better when I know the cause.
Now, if I didn’t reboot to do the Preventive Maintenance – I never would have known that this file system was affected, and at the most inopportune time, I could have been called with the server being down.
My goal is to keep this server up and running and as problem free and possible. It seems as if you have far more experience with this platform than I, and will absolutely implement your suggestions, and review your recommendations and suggested readings. I pride myself on the work I do, and I clearly need to bring my mastery of this OS up to a higher level. Books/other recommendations are welcome.
So to reiterate – watch my logs – any specific logs under var/log? I even thought perhaps a bad block could have been the cause – my apologies, I always like to know the cause of a problem. I did reinstall the OS to the TigerOS Volume which did erase that volume – but did not zero all data. As mentioned this machine is RAID 5 three drives both the OS and Data Partitions are across all three drives. I would have preferred if the hardware and budget supported it, to mirror the OS and use RAID 5 for data.
Is there a log that reports disk errors? You say repair permissions when you have a specific error – what specific OS Volume error would repairing permissions actually resolve?
As mentioned, I want to prevent what had happened – I am NOT in the habit of rebooting servers, not even Windows servers with any great frequency. I have several Windows servers that prior to Microsoft’s patch Tuesday – were up for 9-12 months – even longer! If it wasn’t for “Patch Tuesday” the Windows servers would be up far longer!
So then – in a nutshell – you are suggesting other than checking logs – there really isn’t a need to do Preventive Maintenance on an Xserve running 10.4.7? Would the log files have shown me that I had problems (still wondering what the cause was – would love your opinion), with files under usr/bin and sbin? What about running disk tools periodically from within the OS to Verify the integrity of the OS partition – no harm doing this – correct? If it shows no errors – is it safe to assume that I should be able to reboot?
I can’t thank you enough for your time, and sharing this info with me. Mt goal is to keep this server running and as stable and as trouble-free as possible, just seeking to fill the voids of experience I have with this particular OS. I will try your recommendation for creating an image (on a test box first), but what about Super Duper (I believe it is called)? I do have to bring the box down to create the images.
I can’t thank you enough
Sincerely
Be well
Mac OS X (10.3.8)