Mac OS X Render Farm

I need to setup a render farm that supports the following applications:
Adobe After Effects, Final Cut Pro, Motion, Shake, Houdini, Modo, Maya, Cinema4D and Compressor.
What would I need?
Does Xgrid support these apps or is the Sun Grid Engine a better bet?
Sun Engine Grid
http://gridengine.sunsource.net/
XSan is a must unless ZFS is a better option not too sure if it would be compatible though. What RAID arrays would work best? Can I use the Sun StorageTek 6140 array with the XServe being that XRaid is no defunct (Promise RAID looks iffy, not to mention ugly).
How would I manage the Queues (Can I run multiple Queue Applications to support different apps i.e. Apple Qmaster and Sun Grid Engine)?
RUSH http://www.seriss.com/rush/
Dr. Queue http://drqueue.org/cwebsite/
Qube http://www.pipelinefx.com/
Anyone with experience.
Thanks,
Juan

Well I mentioned ZFS mainly because the Sun StorageTek Array supports ZFS and XSAN looks somewhat similar to ZFS in that the storage is pooled. I imagine Apple is planning on replacing XSAN with ZFS in the not too distant future, if so Sun Storage looks great!
Regarding Qmaster, I'm going to investigate further. Its supposed to work according to Apple's advertising:
Clusters for encoding and rendering
Q Master
Qmaster uses the processing capacity of your network computers for a wide range of tasks, including encoding and rendering for motion graphics produced by applications *such as Apple Shake, Adobe After Effects, and Autodesk Maya.* Configure Qmaster clusters to run jobs for encoding, rendering, or both, and change the configuration in a few simple steps. You can output files faster than ever by configuring clusters to take advantage of multiple processors, or “services,” as well as multiple computers. For even faster throughput, configure Qmaster to manage individual processors in each computer, turning any Mac Pro into a virtual render farm. To further optimize your workflow, create clusters to be used for specific jobs, users, or applications.
http://appleclub.com.hk/finalcutstudio/compressor/encoding.html
Modo seems to have network rendering built in as well. Still waiting for the Mac version of Houdini to test it out. I've also have been playing with Real Flow.
I'll keep updating the post if I find anything.
Thanks,
Juan

Similar Messages

Mac OSX 10.4 Server on a MacMini + Mac Mini Network render farm

Hi Guys,
I am working at a small vfx post house in Bangkok. We are looking to install some sort of server and network renderfarm.
There was talk of going Windoze based when I first arrived, so I am doing my best to steer them to Mac & OSX. (Most of the current workstations are MacPros)
For a budget solution I said it may be possible to run a OSX Server on a new Mac Mini, and also create a 5-10 machine Mac Mini renderfarm (for rendering 3d scenes from Maya, encoding with Compressor, and 2d scripts from Shake).
So my questions are:
1. WIll OSX Server work on a mac Mini?
2. Will the Mac Mini be able to handle it? It won't be a render box itself, more a machine to keep control of permissions on our central storage (stop people moving and deleting files and saving things in the incorrect place), and control the QMaster Adminstration.
3. Has anyone had any experience creating a renderfarm from a stack on Mac Minis?
4. If no one has any idea can they recommend a person/forum to ask?
Thankyou very much!
Adam Teale

I found answer to my first question - does OSX server work on a Mac Mini - yes
http://www.macgeekery.com/tips/customization/macmini_and_os_xserver

New Mac Owner - Render Farm Question

I am new to the mac world. I have purchased the macbook pro, 17" with 2 gig ram and 7200 rpm HDD. I am using the machine for video editing, effects, and DVD authoring. I have read through several files about compressor2 and Qmaster and their ability to improve render time when producing a video file. Being new to the mac world, I do not have multiple machines to link together via network to utilize these amazing applications. I do, however, have several PC's that are high speed, dual processor, with loads of ram (previous video editing systems). Can a PC run the Qmaster and/or compressor software and allow for the render farming? Any input and advice is greatly appreciated.

By default you CANNOT run anything called "QMaster" or "Compressor" on a Windows PC.
BUT!
There is something buried deep in the Apple QMaster documentation that describes how to use Terminal on your Mac and the command line to set up any Unix or Linux machine as an external [what Apple calls,] "EXTENDED NODE."
But to do this, you have to have at least ONE extra Mac to use as the "INTERMEDIATE NODE."
How it works is that you use Terminal and command line codes to make the EXTENDED (non-mac) nodes talk to (and offer their processors to) the INTERMEDIATE node, which then says to the Cluster Controller, "Hey! I've got more SERVICES NODES to offer you."
So, in a manner of speaking, this process tricks the Cluster Controller into thinking that the Intermediate node has more services to offer than it really does, because the Intermediate node is kinda "subleting" the processors of the Extended Node(s).
So, it's up to you to decide if it would be worth it to install Unix or Linux on all those PC boxes you have, but it would be possible, provided you purchased another mac to use as an Intermediate Node. Perhaps an Intel Mac Mini would do the trick.
Good Luck.

After effects and premier render farm

Hey
Im working on a Mac Pro mid2010 and Mac pro Late 2013.
I want to know if and how can I work on one of them but when I render or creating ram previews make them "work together" ?
like a render farm ...
Thanks

Yes let's say its a really small farm.!
There are no differences in the operating systems (10.9)
and im using cs6 in both of the machines (but I am planing to upgrade to cc).
so how can I create this "farm" thing ? And if I got you right, it will work only
when exporting a file. not in a ram previews. Right?
thanks

Render Farm PowerPC

Hello!
I need to convert a ton of files from .dv format to .m4v .mov or similar. Since it will take years to do it (almost 3TB of videos) with my current Mac Mini (first Intel version) I was thinking to set up a render farm with all my desktop PowerPC Macs. I've never tried this before, so I don't know how to do it. I've heard that Apple uses Qmaster to do that. I've also read something about Xgrid, but Xgrid seems to be for different tasks (I would like to know wich ones, because I want to do something useful with my old PowerPC Macs and a distributed computing solution may be the answer).
My macs:
- PowerMac G3 B&W (400Mhz)
- PowerMac G3 B&W (450Mhz)
- PowerMac G4 PCI Graphics (450Mhz)
- PowerMac G4 Digital Audio (733Mhz)
- PowerMac G4 Quicksilver (933Mhz)
- PowerMac G4 Mirrored Drive Doors (Dual 867Mhz)
All of them with at least a bootable partition of Tiger 10.4.11. The PowerMac G4 MDD also has Mac OS X Tiger Server installed.
Thanks!

I think your best bet is simply to schedule it yourself. Set up a simple automater script that reads from a folder and performs the conversion. Simply drop a bunch of files into the watch folder and let it run. You can experiment with how many concurrent processes you can run on each machine before it bogs down.
Don't expect much out of the G3's.

Cluster Mini Render Farm

I'm terrible at server things, I wanted to start with that and be rather upfront about it. But that's why I'm asking out to you in the community. I'm a video editor and in Final Cut Pro it takes literally forever sometimes for things to render when adding effects. Unfortuantly rather than buying a Mac Pro, I instead bought a MacBook Pro. Which don't get me wrong I love wholeheartedly, except the graphics card inside just is embarrassing, plus with how quickly the processors heat up it's hard for me to work constantly and quickly. So I was curious, I asked out in the general communities last week about putting some minis together and using them as a small "render farm" and what I got back was some more or less maybes... so before I continue on I thought that I should pose the question to those that are basically lord in the mac tech services for help or rather "how to". Because finding out just how much work that would have to go in it would really set the tone of whether or not I should do it.
I was thinking just purchasing two of the Mac Mini Servers (Quad Cores each) and using them.
So finally, is it indeed possible to even use two minis to process the graphical outputs of Final Cut Pro X that I'm running? And if so, just how do I go about doing such a project?
Thanks so much!

I think your best bet is simply to schedule it yourself. Set up a simple automater script that reads from a folder and performs the conversion. Simply drop a bunch of files into the watch folder and let it run. You can experiment with how many concurrent processes you can run on each machine before it bogs down.
Don't expect much out of the G3's.

After Effects CS6 Render Farm

Hi All,
Another Friday, Another test of new AE waters....
I am busy reading up and trying to figure out how to use the three MAC's in the office to work together as a render farm. I have gone through the documentation and created a very simple scenario:
1) Windows XP "server" (it has the shared disk that i use for storage - 2TB spinning disk)
2) Created "Watch Folder" on XP Machine
3) all three MAC's can see and access the shared folder
4) Created a simple project - One 192MB video file (XMP) - applied keylight and matte choker
5) I did a render on the MAC, using only itself and standard setttings to a TGA sequence, to another shared folder on the XP server - this process took 3 minutes and 7 seconds to complete
6) I re-rendered the same project, with no changes on the mac, with the following settings:
and send them all to the watch folder (project saved there, files collected there etc.
7) once the project starts, it takes 20 minutes.
Is there a specific reason why the time would increase so drastically rather than decrease?
The macs all have 1 quad core CPU - 16GB memory and they are connected to a GB switch (all connecting at 1GBps) - If i check the tas manager on the XP pc, network utilization never goes above 25%.
Thanks guys,
Pierre

Hi Pierre,
I believe what you maybe seeing is a limitation on XP's file sharing. For the sake of simplicity network rendering treats every core as a user. So depending on which version of XP you are running it will either allow 5 or 10 users to connect to the computer at a given time. If you have other sharing services enabled (such as printer sharing) any connections to these devices count against this number of users. Given your other thread on where you discuss the rendering nodes, if possible you may want to try using windows 7 (this allows 20 users) or one of the windows server products (max # of users is based on CAL's)
Edited:
A couple of other things came to mind if you are using watch folders.
- If you are running anti-virus software on the server, ensure the watch folder directory and subfolders are excluded from scanning.
- Turn off render multiple frame simultaneously' in the preferences on the render nodes.
Hope it helps
Message was edited by: StevRo

Render Farm - How do nodes Work?

Hello,
My name is Anatoly and I've recently set up a rendering farm connected to an I-Mac as a file-server.
The Problem: Not all nodes come to work together on a project, some arbitrarily choosing to render the assets while others just watching the watch folder idly.
The Computer Setup: There are six headless nodes (all Mac-Minis). The hardware and OS specifications are below.
MAC-MINI Farm:
2.66Ghz Core 2 Duo, 4Gb 1067 Mhz DDR3 RAM, NVIDIA Geforce 320M 256Mb, OSX 10.8
2.66Ghz Core 2 Duo, 4Gb 1067 Mhz DDR3 RAM, NVIDIA Geforce 320M 256Mb, OSX 10.8
2.3Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, Intel HD Graphics 3000 384Mb, OSX 10.7.5
2.3Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, Intel HD Graphics 3000 384Mb, OSX 10.8.5
2.3Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, Intel HD Graphics 3000 384Mb, OSX 10.7.5
2.3Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, Intel HD Graphics 3000 384Mb, OSX 10.7.5
I-MAC File Server:
2.5Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, AMD Radeon HD 6750M 512Mb, OSX 10.7.4
The Software Setup:
Each MAC-MINI node has a copy of After Effect CC Render Engine (version 12.0.0.404) running at startup and set to run in Watch Folder mode.
On the machine that is used to manipulate assets for rendering the following procedure is used:
Open a project
Add assets to render queue
Set Render Settings to Multi-Machine Settings
Check box in settings to “Skip existing files (allows multi-machine rendering)
Set Output Module to Multi-Machine Sequence
Select my desired output settings
Then I click File – Dependencies – Collect Files
In Collect Files Setup the following is chosen:
Collect Source Files: All
Change Render Output to: “custom name” folder
Enable Watch Folder render
Reveal collected project in Finder when finished
Maximum Number of Machines: 10
I then click Collect and save my project to a Watch Folder
From there-onwards, presumably, the Render Farm is now supposed to render out my project, but not all nodes of my 6-node farm work together. More often than not, a single (or two) node(s) render the projects while the others wait idly, watching the Watch Folder.
Can anyone please advise me of why this happens, and perhaps what step I’m missing if any to get all my nodes to render until the project is done?
Thank you,
-Anatoly

Here's a picture of render items in my queue, follwed by a picture of how many nodes are working on it:

Render farm between multiple os

Hi All,
I have a question about setting up a render farm for After Effects CC. Basically my main computer runs on OSX Maveriks, and I have two more Windows computers that I would like to help out rendering After Effects compositions. I have set up one of the machines with the After Effects CC render engine, and the Mac and other Windows machine have After Effects CC installed as part of the 2 machine license.
When I setup a watch folder render from the first windows machine to the second one, everything works fine, but what I want to do is setup the render from my Mac to both the windows machines who are monitoring a network watch folder.
Permissions are seemingly all in order, but if I set it up from the Mac both PCs give me the message: "Render control file not valid".
I have searched all over the Internet for a solution, but there seems to be nothing there. Also, I suspected it might have something to do with the way a Mac does line endings in files compared to a Windows machine, so I rewrote the RCF file on the Windows machine itself, making sure it was in the right format, but this did not work either.
Any help?
With warm regards,
Daniel

hello fastlane, for firefox you can use firefox sync to share bookmarks, passwords & history between your three OSs - [[How do I set up Firefox Sync?]]
& for keeping your mails synced it's possibly the easiest way to use a mail-provider that offers IMAP, which keeps the mails on the server so that your mail program on any OS has the same data when it's connecting...
(but we're not specialised on thunderbird here, you might head over to https://getsatisfaction.com/mozilla_messaging/ for more detailed questions).

Render farm question

If the company I am working for is doing a lot of videos on a per month basis should we setup a render farm?
is so should we use window or mac mechines?
can someone provide information for setting up a render farm? or a link to a good how to?

It depends on how render-intensive your projects are.
An octo-core machine with enough RAM will often render in minutes things that used to take hours, or than can still take hours on lesser machines.
If your Comps turned out to be something that such a multi-core machine would render in (say) an hour, would that be fine? That's the question you need to ask yourself. And to answer that question, you first would have to create a typical Comp for this project and render it.
Setting up the project and the machines to perform rendering over a network is very easy. See Network rendering with watch folders and render engines in AE Help.
is so should we use window or mac mechines?
Use whatever you like better. AE is designed to perform perfectly on either platform. In any case, make sure you take a look at AE's system requirements/recommendations.

Building a "render farm"

Okay, so after reading the threads here I have qmaster distributing the tasks between 2 G5 desktop machines and it's going much faster than with one. (it seems like its about 3x faster)
Now I want to build a mini render farm to speed things up even more.
My question is this. Do I NEED to use OSX servers and cluster nodes? The engineer at Apple said I would be better off but I can buy more dual 2ghz G5 desktops with my budget than servers and it would seem they do the same jobs.
I could string 4-5 new G5's together as a start or 3-4 servers. Not to mention I would have to learn the server OS and everything before getting started.
Any advice is appreciated

PowerMacs are faster than Xserves. Just not as convienent space wise. You don't need xserves. Any mac will work, just don't have a mix of really slow machines and really fast, cause they slow machines will really slow things down. Let me qualify that: With compressor slow machines will really slow things down. Frame renderers like Shake and Maya would be fine when you adjust the segment size.
Server OS is not needed. In fact, there is no difference between server and regular OS X, just the admin tools that come with server. You can run all the "server" services on the consumer version.

Cs6 Render farm 10.2.1

I am currently running Cs6 on my mac pro, i have several G4 machines sat doing nothing so i thought i may use them in a render farm connected to my mac pro.
How do i install the render engine on the old machines? My Cs6 installer doesnt seem to work
G4's are running Osx 10.2.1

You can't. AE hasn't supported any PPC processors ever since it went 64bit and that is now already6 or 7 years ago. Your G4s realyl are nothing more than decorative screap. Sell them on eBay. Some people build aquariums/ terrariums from such stuff or some case modder may want to use it for building a fancy new PC in disguise...
Mylenium

Is there any point in a G4 render farm?

Hi all.
I've been getting quite intrested in the idea of setting up a very small render farm laterly as I have the option of getting some old G4s for free.
I currently have a 2.0Ghz DP G5 as my main mac running FCP2 and Shake and was already grabbing a G4 466 digital audio to set up as a file/print and back up server running tiger server, a postscript RIP and retrospect. I have the possibility of getting atleast one more G4 466 DA and maybe a couple of G4 400 AGPs with Gigbit ethernet cards. As the main use would be farming out DV to mpg2 compression via compressor and shake renders (though not large shake stuff to begin with as very new to this) is there any point in looking into this or am I going to need to lots of G4s to make a real improvement in render times.
Also is RAM a major consideration. For example if I got 4 G4s with say 512Mb ram would it be better runing all four or using three but nicking the fours ram to give more ram in the three machine set up?
Any advice would be appreciated
Cheers
Steve

I vote no. G4 isn't the problem so much as 400mhz is. The ram shouldn't be a problem.
But you could test if you set up Compressor on one G4. Compress a job. Compress the same job with the same settings on the G5.
If the G5 is 4 times (or even 3, since time is spent sending the render data over the network between the G4's slow system busses) as fast the single G4, a render farm will not help you.
Using the G4 and G5 together in the same cluster does not work because Compressor looks at all processors and divides the job into twice as many segments. In this case, the G5 finishes it's two segments years before the G4's and then sits around waiting. You could try splitting the G5 into 4 instances, but then you have 8 slow processors attacking your job. Not very efficient.
Good luck though. You should test the one G4 just so you know. Report back if you do.

AE render farm cross platform

hi all,
just wondering i have set up a AE render farm on pc (2014 v13.1.1), i know it can render PC AE renders if a pc sends a job to it but can it render mac render jobs?
rob

After Effects project files are cross-platform compatible. Things to watch out for are that file paths are valid, and that fonts and plug-ins are installed and working. You should open the project on a Windows machine to check for any errors, especially with fonts. Some fonts may have different names between platforms.
When using Collect Files, the source file paths are set relative to the project so this is usually not a problem. But you must make sure that the output file paths are also valid; using the Change Render Output To option in the Collect Files dialog can help with this.
More detail here:
After Effects Help | Automated rendering and network rendering

Hardware Encoder for H.264? Encoder for Flash? Render Farm?

I use Compressor to create H.264 QuickTime files, and have setup Qmaster to use a 8 core cluster to expedite the process, but it is not reliable, and fails quite often. I have a total of 3 Mac Pros in my facility which can all share their resources to help this process be quicker. I was wondering if anyone knew of a hardware encoder that will encode H.264 files very quickly? I have a bid on a project that if granted, I will need to encode a VAST amount of H.264 QuickTime files. Is the best solution a dedicated Mac Pro render farm with a Qmaster setup?
Next, on to the Flash. I currently use Sorenson Squeeze 5 to encode 2 pass VBR flash files, and it takes forever even on my 2.8ghz Octo with 10gb of RAM. Does anyone know of a piece of software or hardware that can expedite this process? Maybe a Flash render farm? Maybe a gnome with a wand?
Thanks in advance.

Keep your eyes open for when the Matrox CompressHD goes on sale.
While it probably won't be available to use on the job you've just bid on (tentative sale date is May 2009 but who knows if that can be trusted), it otherwise seems like the answer your prayers - it speeds up h.264 and Flash encoding. It will, supposedly, list for $495.
Otherwise, as mentioned, the Turbo264HD is the way to go for now.

Mac OS X Render Farm

Similar Messages

Maybe you are looking for