How do I achieve Full peace of mind on a tight Budget?

Question

I am looking to complete an Open Directory Project with managed Network Directories and Networked Home directories.
3TB of available storage should be sufficient (for now).
I will be moving users who are use to having their Macs set up as home computers on a network to a proper enterprise desktop system.
But here is the problem I need to be able to guarantee that users will have access to their home directories even in the event of a serious hardware failure, because not being able to use their computer because some other computer isn’t working will not be popular. If the situation last more then a few hours then... well I don't like the idea of a recovery plan that starts by ordering more hardware and ends in applying for a new job!
As I understand it if one half of a X server Raid Packs up the other half may still work, but can you take the disks out of the bad half and plug them in to the good half, (or even a spare half of a second X Server Raid) without having to rebuild the raid set, reformat it and restore the data from back up?
P.S.
I'm not looking for a clever dick answer about setting up desktops to function like laptops, but if there are any good ways that you know work to acheive total redundancy server side then I would love to know of them.
server Mac OS X (10.4.8)
server   Mac OS X (10.4.8)  

Answer 1

Disaster recovery and business continuity is a balance between acceptable risk and cost. What's acceptable to you may not be acceptable to someone else (and vice versa), and you're the one that has to decide what level of risk you're prepared to carry.
In this particular case you're focussing on the XServe RAID. The plan to run a half-populated array and switch drives and/or controllers is fine - provided you don't need more than half the capacity of an XServe RAID (currently 10TB), then the cost to implement this is nil - it's available 'out of the box'. However, it only mitigates a specific case, namely controller failure which William has stated is a rare event.
It doesn't address any of the other possibiities such as disk failure, data corruption, accidental file deletion, server hardware crashes, Open Directory corruption, etc., etc.
While none of these may be common events (hopefully), they are more likely to happen than a controller failure. so your no-cost plan isn't actually covering much of your risk.
What you need to do is determine a list of risks that could impact you and devise action plans around each of those. Some of those might be mitigated by additional hardware (e.g. a spare parts kit, second server, etc.), others might be process (backup, replication).
Each risk has a cost (e.g. time-to-recover, hardware replacement cost, etc.).
Each mitigator also has a cost (additional hardware, admin time, etc.)
Once you have those it's easy (or at least easier) to work out what you need to do. You just balance the risk cost combined with the chance of it occurring against the cost of implementing a solution.
Then you (or your boss) can draw a line in the sand - anything above the line is an acceptable cost to manage a certain set of potential problems, and anything below the line is either too costly to implement, or too unlikely to bother about. This is the only time that the budget comes into question - it sets the bar for the line.
At that point you know what you need to do, you know what level of risk you're prepared to carry and if one of the events below the line happens and you don't have a real-time solution for it you've at least covered your rear-end by knowing that in advance ("this is one of those events that we decided not to have an in-place solution for).
If the line in the sand isn't where you'd like to be, that's where you go back and argue for a bigger budget ("for the $x you're giving me, we can deal with a, b and c. For an additional $y we can also mitigate d, e, f, g, h and i").
Bear in mind that one solution might mitigate several risks - for example having a second server allows you to replicate your directory and covers all events that impact the primary server hardware (e.g. failed disk, failed power supply, directory corruption (provided you catch it before it's replicated). If the primary server crashes hard you simply swap out the server, reconnect the XServe RAID to the backup server and you're running again. Also consider that the backup server might not need to be the same configuration as the primary server - if it's running as a backup it might work well enough with less memory, might not need mirrored boot drives, etc.
Sorry if that's a longer response than you expected/wanted, but disaster recovery/business continuance is a complex task. Budget is always an issue, and you'll never cover all possibilities, so the only way you can sleep well at night is to know that you've thought through as many issues as you can, and have an acceptable, agreed plan for dealing with issues.