How important is a dedicated metadata controller?

Question

Hello,
I'm currently setting up a computational cluster consisting of 16 XServes (eventually 32) and 5 Mac Pros, each of which need to simultaneously write to a single volume on a shared 10TB Promise VTrak RAID. My motivation for using XSan is data integrity and redundancy (which is critical) and not speed (which is nice, but not necessary).
For this type of setup, how important is it to have a dedicated metadata controller? Ideally, I'd like to have one XServe as a dedicated controller for both XGrid and metadata, the Mac Pros as XGrid clients and agents, and the remaining XServes as XGrid agents. Everything I read says that there will be "decreased performance" if the metadata controller is used for functions other than XSan, but what exactly is this decreased performance? I can live with some latency in data access while the metadata controller is doing other things, provided all data gets written eventually. There will be no other traffic (mail, web, etc) on this network.
I know the first thought is that with so many machines, why not just use one as a dedicated metadata controller? Again, data access isn't the primary goal here; each machine performing computations takes about a day off my computation time so I want every available asset working towards that goal. Even if I have to wait five minutes to access the data, I'm still way ahead of the game using the other machine for computations.
In the event that using a dedicated metadata controller is vital to using XSan in this situation, are there other solutions which would perhaps better suit my needs?

Answer 1

The question really is 'how important is the reliability of your Xsan' and because its become a single shared resource everything depends on it, this is the same in post or data centre workflows.
The ideal situation is to have no other services pulling any clock cycles from the processor because it could coincide with xsan requiring resources. The tricky thing in xsan deployment is latency reduction, the controllers and switches don't look busy and they aren't but the smooth running of the system depends on that reduced latency and when you lose it your going to run into some strange and probably intermittent problems.
Ive cut corners in the past by running open directory and DNS on the backup controller but never on the primary. Really it depends how busy the whole system is going to get but dont expect Xsan to behave simply when the latency starts to go. Its a very robust file system but it wont slow down in a linear way like a busy web server it will slow then hiccup stop, start or fall over entirely, which is not good in a critical situation.
One solution Ive used in a similar situation is a smalltree 6 port ethernet card in an xserve with all ports bound together, make sure your switch can handle link aggregation but it creates a nice big pipe to a fileshare in a situation where xsan wont work. You can't edit video like this but you can remove the server bottleneck when you have several client computers requiring big file transfers.
Another route is iSCSI but its still early days in the Apple world, ATTO now make an iSCSI initiator for Macs but I can say Ive deployed it.