Big Big Clusters / Data Centers

Question

Hi folks. I have a bit of a theoretical question for the group, seeing as some of you should know how to handle this.
I am facing a job that could get quite big due to the sheer size of the nature of what it does. I would like to use Apple technology and MySQL. Naturally this comes down to XServes and XServe RAID.
I know MySQL has a table size limit of 2 TB. If I can juggle multiple raids and XServes, how does one go about designing something like this?
I will have a middleware taking incoming data (web services) as well as serving reports. So archives of data would need to be stored in MySQL. Nothing fancy. Pretty dry, relatively speaking.
So do I just create a whole bucket of xserves, put fibre channel between them, hook them up to these RAIDS, stripe them, and measure if I need to add more?
How does one go about designing one conceptually? What if my growth rates are doubling once a month? What limitations should I look for, assuming cash isn't a problem?
I didn't find much online, as I'm sure this is a rare application. I'm kind of blind with this, outside of my past experience.
Cheers

Answer 1

Hi there.
Please understand, that I have done some homework. I've looked at VT as well as the information on Apple's site. VT's storage seems rather light compared to what a data-intensive application might need, as opposed to a processing-intensive application.
As far as I can see, there is a 2 channel fibrechannel limit to an XServe RAID. Putting a loop together means that an increasing amount of data needs to be accessed through the LAN widths, so that places restrictions as well.
In the meantime, I've found a caching system that seems to alleviate the issue of massive amounts of data. It's a caching system called memcache.
I've done some calculations on the data I'll be dealing with, and similar large computers have the luxury of having time on their side with respect to serving and crunching. As you said hospitals, it is relatively small compared to my application.
As well, finding people to ask isn't a challenge. The reason I am posting here is to get insight on the direct application, not a suggestion to 'ask someone'. I am hoping within these forums, there is a way to start discussions on the topics themselves. I do have VT on the list of people to contact, but from the information they have online, it seems it could lead me towards something they might not know about. So, that's why I posted here, to see if anybody within these forums, would know the difference in the various cluster types that can be designed, and how to optimize them. It's not that VT isn't a solution, I just see it misleading and could be a very costly consulting job. Regardless, I do want to optimize forum questions in order to see what information I can get my head around before I just avoid getting non-stakeholder input by paying people to tell me about their solutions.
I also find that Apple cluster solutions are not that common. The largest data center here in Canada, doesn't have a single XServe in it, from what they tell me. It is the first reason I am asking in these forums.
I do have an MBA and know how to contact people to ask questions. It won't discard me asking here.
Cheers