Building render farm

Hi!
I'm new to all this video editing thing, and have been trying to edit and render with my laptop. Now i had the chance to help a company to get rid of 10 desktop computers with intel core duo prosessor from 1.8 to 3.0GHz. Is it possible to build a rendering farm from these unit? only problem is that none of them has a hardrive, do i need to buy hardrives to make somethins out of them?
Highly appreciating Your time and help!
Ilkka

For an AE render farm, you need a hard drive with a Windows or Mac OS in order to install the render engine software.

Similar Messages

Building a "render farm"

Okay, so after reading the threads here I have qmaster distributing the tasks between 2 G5 desktop machines and it's going much faster than with one. (it seems like its about 3x faster)
Now I want to build a mini render farm to speed things up even more.
My question is this. Do I NEED to use OSX servers and cluster nodes? The engineer at Apple said I would be better off but I can buy more dual 2ghz G5 desktops with my budget than servers and it would seem they do the same jobs.
I could string 4-5 new G5's together as a start or 3-4 servers. Not to mention I would have to learn the server OS and everything before getting started.
Any advice is appreciated

PowerMacs are faster than Xserves. Just not as convienent space wise. You don't need xserves. Any mac will work, just don't have a mix of really slow machines and really fast, cause they slow machines will really slow things down. Let me qualify that: With compressor slow machines will really slow things down. Frame renderers like Shake and Maya would be fine when you adjust the segment size.
Server OS is not needed. In fact, there is no difference between server and regular OS X, just the admin tools that come with server. You can run all the "server" services on the consumer version.

Question on potential configurations for resource cluster as virtual machine server / beowulf / render farm

I am looking for a starting point on where to focus my efforts in researching a desired configuration. Of course anyone who spends anytime researching in forums knows that one thing that people hate is for people
to ask questions that can be easily answered with a simple Google search, so let me start by saying I have done many hours of research on this already. The problem I am having is that although there is a huge amount of information out there on what I
am looking to do, I don't believe many people have attempted this on as small of a scale as a single user system. Also, the amount of routes to go down seems endless, which for me so far has led to many hours of wasted trial and error. So keep
in mind that I am not so much looking for a handout on answers but rather a more narrowed path of searching and researching.
What I am looking to do is take the current equipment that I have and consolidate it into a 9-12U server rack using mounted 2 or 3U atx cases so that I can have a more organized, space saving setup. That is the
easy part, what I have been looking into though is the combining of cores and processing power to make a "super computer" yet I use this term very loosely. From what I have read, the closest thing I can compare the idea in my head to is referred
to as a Beowulf Cluster, or a Render farm but I believe as the name suggests that this is more dedicated to a single task of rendering. I like the idea of a virtual machine server where I would never actually locally use this machine but rather set up
many virtual machines that I can remote into. I started to compile a ProxMox Server, and this was the right idea, however when running a VM, that VM is constrained to the hardware resources of the single node that it is assigned to. If there is
a way to be able to have a machine that would utilize the 20 some odd cores that I would have in the cluster, along with the 40-50 gigs of ram, well that would be ideal but sounds too good to be possible. Also, ProxMox isnt quite there in compatibility for
a home/home office user. I understand that the programs that are used have to be written in a way where the architecture allows for such, but if it could even be used for speeding up my render time with Adobe and Autodesk (which I know do have such functionality)
that would be wonderful.
So without writing a whole book I am looking for suggestions on how to proceed. Can I literally build a "super computer" that is not really that super... more mediocre super... or am I better off just consolidating the hardware but not the
OS's and programs. Also, I do currently use three or four different distros of Linux, mostly for ethical hacking and security type work like pen testing. One thing I have never been a fan of though is dual booting. Perhaps investing
in a hot-swap HDD chassis that I could load all the different OS's on where I could just pop in whatever OS I am looking to use. As you can see I am far from knowing where to spend my efforts in research.... THANK YOu for ANY help.
I was recently suggested Microsoft HPC Pack as a platform? Anything?
Here is what I currently have:
these are all fully built)
Intel i5 3.7 quad core
Nvidia (ASUS) GTX-760
16 Gb DDR3 @ 1600 Corsair
ASUS Sabertooth Z87 MoBo
Autodesk - Inventor, Revit, 3Ds Max, Maya, Fusion 360, Flow, etc.
Adobe - PhotoShop, After Effects, Dreamweaver, Illustrator, etc.
Basically as you can see this is used for my work in 3d modeling, rendering, and drafting along with 3D printing.
Sys. #2
AMD 4.7-5.0 8 Core
Nvidia GTX 760 (Same as sys #1)
16 Gb DDR3 @ 2400 Mhz
ASUS Crosshair V Formula Z
Various programs for programming, app development, gaming, etc. (This is my daily driver)
Sys #3
AMD 4.5-4.7 8 core
16 Gb DDR3 @ 1600 Mhz
Asus Crosshair V Formula Z MoBo
Not currently using this one but is fully functional.
Sys #4
HP built i5 3.4 quad core
12 Gb DDR3 @ 1600
Nothing Special - this is the one I was using for school work but being that school and programming became one and the same I have all but abandoned it.
THANK YOU AGAIN TO ANYONE WHO HAS MADE IT THIS FAR in reading this.

The cluster works only for standalone webcache and httpserver installations types.
If you want to do this you have to use dcmctl command to change from one farm to another.
try checking with dcmctl command with out options an dyou will get a full list of options.
then run dcmctl usage(orhelp) option to see what you need.
As far as I remember
you have to check the farm id of the server you want to be the master
then in the second, you have to run dcmctl joinfarm or something like that.
Regards.

Render farm for DCP creation

We are starting to get more and more requests to make DCPs for clients and we need to speed up the process. Currently doing a DCP through the Wraptor AME plugin takes around 20 hours for a feature length film. It's a bit faster if we do tiffs through AME and use OpenDCP for the rest but even then it's still taking a while. When we are doing work for festivals etc we are simply running out of time where we need to do dozens of films.
Is there a way to use render farms to speed up the AME processing? And if so, any advice about setting it up?

Rather than a farm, if you can find out from Adobe or the Plug-in maker how well the code threads you might consider a single computer with multiple CPU's llike the Xeon E7 v2 chips that can have up to 15 cores and up to 8 sockets. Here is Harm's picture on his Tweakers Page. To run a beast like this you have to us Windows Server OS which not officially supported by Adobe but apparently should work. Of course the pricing is astonomical.each CPU chip (15 Cores 8-way) is $6841.
I would suggest contacting Eric ADK and see if they can build you a Super Workstaion for your processing. If it can be done they are the one's to do it

Cs6 Render farm 10.2.1

I am currently running Cs6 on my mac pro, i have several G4 machines sat doing nothing so i thought i may use them in a render farm connected to my mac pro.
How do i install the render engine on the old machines? My Cs6 installer doesnt seem to work
G4's are running Osx 10.2.1

You can't. AE hasn't supported any PPC processors ever since it went 64bit and that is now already6 or 7 years ago. Your G4s realyl are nothing more than decorative screap. Sell them on eBay. Some people build aquariums/ terrariums from such stuff or some case modder may want to use it for building a fancy new PC in disguise...
Mylenium

Render Farm Possible in Premiere?

I multi-machine render in in AE all the time. (For those who don't know, that's using several different computers to render the same project faster.)
Can you do this in Premiere? The closest solution I've come up with is rendering out and assiging frames to TIFF sequences on 3 computers like this:
Computer 1: Frames 1-2000
Computer 2: Frames 2001-4000
Computer 3: Frames 4001-6000
But if computer 1 finishes first, it doesn't go on and help out its buddies, it just sits there like the last intern I had waiting to get fired and not looking around for anything to do, like move the 30 c-stands and sandbags to the grip truck instead of drinking pop and watching all of us work...
Oh yeah, render farming.
In AE you can click it to "skip existing files" (or somethign to that effect) so that if AE sees that frame 1001 is done already, it automatically skips to frame 1002, and on and on until every last frame is processed.
Anything like this in Premiere that I've HOPEFULLY just been overlooking ?

Here's the straight poop,
I don't know if they'll ever get render farming up and running for anything outside of 3d effects, but the reason it hasn't been built yet is the same reason render farming took so long to develop in the first place: the workflow.
Render farming doesn't output a single file. Why?
It uses the FRAMES and outputs a number of them from each machine in the farm, all to the same location (on any chosen machine). However, these FRAMES are simply picture files. That's right guys, just single picture files. The same is true for video files really, that they are made up of nothing but picture files being rendered to the screen at super fast speeds, but they are all bundled together. In compressed video, you lose some of the quality of the images in favor of saving speed and space, but with lossless codecs you are actually processing every frame (or every single picture). This means that you'd have to add that processing step to the workflow for render farming.
Steps originally:
Comp A process these files as images from frame x to frame y and store in folder z
Comp B process these files as images from frame v to frame w and store in folder z
and so on for every machine
Added steps for video rendering:
comp A frames x to y scan and compress to file a
comp B frames v to w scan and compress to file b
and so on through each machine until all are finished
finally: selected comp combine files a to (whatever)
However, with this function you may get extra I-frames or B-frames, and you'll end up with jitter in the final product. For this reason, the logic holds that instead of processing the video compression on multiple machines and combining files at the end, full video render farming should use a DATA PROCESSING BALANCE algorithm, allowing one machine to pass data between the processors of other machines while handling the file build on it's own. Separating the file on the fly (as it is being processed) is a more useful tactic. Start with I-frame A and I-Frame B and send the processing to Machine a, take I-Frame B and I-Frame C and send to machine B and so on and so forth. This allows there to be I-frame overlap, and all the writing is still done with a single machine, while the processing is done across several.
Of course, if you use the first set and change it so that the i, b and m frames over lap, then match them when putting the files together into one, you could, in theory, process it much the same as standard render farming. The second method is only faster due to the fact that all the processing would happen on the fly, and the file would be all one file right away, removing the step of putting several files together, and it wouldn't matter which sections were finished first, how many machines there were, or how many frames overlapped, as the overlapping frames would simply overlap and all the others would fall into place. The only bottleneck would be drive speed and fragmentation of the file, which could be a last step in finalization or an intermediate step along the way. If you use a dedicated and allocated space for the storage, on the fly is the way to go; it also works if you simply run a defrag pass over the file when a section overlaps, thereby moving other sections forward or backward in storage so they line up properly; however, this could just as easily work at the end of the whole process.
From a logical standpoint, it works. From a programming standpoint, it's a lot of work, and it's hard on a system. Personally, farming out a few effects and then wrapping them in a lossless file for use while editing is easy enough. And the quality of the preview is amazing. Rendering out to a single full video file is saved for last or for the end of the day when go home to my family; that way I don't have to ***** about how slow everything was. I can just relax and let the PMS'ing wife yell about how nothing is going right, let the kids yell "he did this" and "she did that", and enjoy the fact that my computers are more obedient. They do what I say without slamming doors and pouting.

Is a windows network render farm/slave compatible with an iMac?

I have an iMac user in my office who would like to set-up a render farm or slave for his iMac. Since Mac products are expensive I am hoping we can build a windows machine to accomplish this. We currently use Adobe CC 2014, so we would need the render farm/slave to work with After Effects and Premier Pro.
Looking for good affordable solutions that won't be a cross platform nightmare to set-up and configure. Any advise would be much appreciated. Thanks.

Hi, Indeed. I am selecting the Bootcamp partition, and If I do Format, It informs me that all we be deleted, ok fair enough. then I can click the NEXT button, but I got then a msg stating :
Setup was u nable to create a new system partion or locate an existing system partition. See the Setup Log Files for more informations.
I have not more option here, I can delete, then create a new one with same size then format again, but same message at the end. this is my issue.
A guy done a Youtube video http://www.youtube.com/watch?v=vD7dbSl9-b8
I am all ok up to 5min 50 on the video. Then My issue is what he has....
However, when he format the partition he does not have my error msg stating Setup was unable to create. etc...
This is my issue.

What is a render farm?

I read a post that said build a render farm and it will be faster....
What is a render farm?
How do you build one?
JKai

I read a post that said build a render farm and it will be faster....
It will. But only for certain things. QMASTER allows you to spread the render across several processors and several machines, but only COMPRESSOR in the Final Cut Studio packages can utilize it. FCP cannot, Motion and Color cannot.
What is a render farm?
Several machines connected together, say via Ethernet for example, that all put their processing power towards completing a task of crunching data.
How do you build one?
Well, in your case, and in mine...hire someone to do it. This is a seriously technical thing. But as I said, it really doesn't help to have one with FCS. Only Compressor can utilize it. Get your network IT guy to connect all the computers via Ethernet, and install QMASTER on all of them...then you can get one going. But there's more to it than that.
Shane

AE render farm Help needed

Hello All,
James here, I'm new to the community, glad to be here.
I am in the process of building a render farm and need some advice on the particulars of going about it.
I'm not a newbie with computers, but I am a newbie regarding setting up a rendering farm.
Here is the equipment I have:
Main computer Tower:
4 core processor that unlocks to 6 core.
Asus board.
ATI Radeon 5570 graphics card
16GB ram
Server Hardware:
3 x Proliant DL380 G4 servers
I have a couple questions on all this, first, will these servers work for a rendering farm?
Second, if so, how would I go about configuring the software on all of them to communicate?
Third, what could I expect from the increase in rendering with this rig?
Thanks in advance for any help you can provide.
Jim

83 :: xx are all resource errors, many times related to fonts as listed on my site. The logical conclusion therefore is that you have changed something outside AE, that is added a rubbish font or updated an external component like Quicktime. Still, as the others said, it's miraculous AE runs/ ran at al lon your system and you were actually able to use it.
Mylenium

After effects and premier render farm

Hey
Im working on a Mac Pro mid2010 and Mac pro Late 2013.
I want to know if and how can I work on one of them but when I render or creating ram previews make them "work together" ?
like a render farm ...
Thanks

Yes let's say its a really small farm.!
There are no differences in the operating systems (10.9)
and im using cs6 in both of the machines (but I am planing to upgrade to cc).
so how can I create this "farm" thing ? And if I got you right, it will work only
when exporting a file. not in a ram previews. Right?
thanks

Render Farm PowerPC

Hello!
I need to convert a ton of files from .dv format to .m4v .mov or similar. Since it will take years to do it (almost 3TB of videos) with my current Mac Mini (first Intel version) I was thinking to set up a render farm with all my desktop PowerPC Macs. I've never tried this before, so I don't know how to do it. I've heard that Apple uses Qmaster to do that. I've also read something about Xgrid, but Xgrid seems to be for different tasks (I would like to know wich ones, because I want to do something useful with my old PowerPC Macs and a distributed computing solution may be the answer).
My macs:
- PowerMac G3 B&W (400Mhz)
- PowerMac G3 B&W (450Mhz)
- PowerMac G4 PCI Graphics (450Mhz)
- PowerMac G4 Digital Audio (733Mhz)
- PowerMac G4 Quicksilver (933Mhz)
- PowerMac G4 Mirrored Drive Doors (Dual 867Mhz)
All of them with at least a bootable partition of Tiger 10.4.11. The PowerMac G4 MDD also has Mac OS X Tiger Server installed.
Thanks!

I think your best bet is simply to schedule it yourself. Set up a simple automater script that reads from a folder and performs the conversion. Simply drop a bunch of files into the watch folder and let it run. You can experiment with how many concurrent processes you can run on each machine before it bogs down.
Don't expect much out of the G3's.

Converting PowerMacs to a Render Farm?

Perhaps like many of us, upgrading over the years, I've accumulated a collection of G4's and now a couple of G5s, and I'm reluctant to part with them...their resale value being minor, at best. I would be more interested in finding uses for them. (well, one may become an iTunes jukebox - to replace my CDs and player eventually)
I remember once learning about something called a 'render farm' and how several computers can be linked together to take advantage of their combined processing power. And then there was that experiment some years ago - was it at MIT or some university? - where 1000 G5s were used to create a 'supercomputer'.
With 7 PowerMacs, is there a (fairly easy) way to take advantage of this possibility? I use PhotoShop, a CG program and an NLE, and would certainly benefit from faster rendering times. I'm not an early adaptor by any means, and until a MacPro lands in my lap, I'm content enough with what I've got.
So, even better if I could hitch these up like a team of horses!
thanks,
k

Yes, it was Xgrid...
http://en.wikipedia.org/wiki/Xgrid
http://www.youtube.com/watch?v=jpqfDC_4Xos

Cluster Mini Render Farm

I'm terrible at server things, I wanted to start with that and be rather upfront about it. But that's why I'm asking out to you in the community. I'm a video editor and in Final Cut Pro it takes literally forever sometimes for things to render when adding effects. Unfortuantly rather than buying a Mac Pro, I instead bought a MacBook Pro. Which don't get me wrong I love wholeheartedly, except the graphics card inside just is embarrassing, plus with how quickly the processors heat up it's hard for me to work constantly and quickly. So I was curious, I asked out in the general communities last week about putting some minis together and using them as a small "render farm" and what I got back was some more or less maybes... so before I continue on I thought that I should pose the question to those that are basically lord in the mac tech services for help or rather "how to". Because finding out just how much work that would have to go in it would really set the tone of whether or not I should do it.
I was thinking just purchasing two of the Mac Mini Servers (Quad Cores each) and using them.
So finally, is it indeed possible to even use two minis to process the graphical outputs of Final Cut Pro X that I'm running? And if so, just how do I go about doing such a project?
Thanks so much!

I think your best bet is simply to schedule it yourself. Set up a simple automater script that reads from a folder and performs the conversion. Simply drop a bunch of files into the watch folder and let it run. You can experiment with how many concurrent processes you can run on each machine before it bogs down.
Don't expect much out of the G3's.

After Effects CS6 Render Farm

Hi All,
Another Friday, Another test of new AE waters....
I am busy reading up and trying to figure out how to use the three MAC's in the office to work together as a render farm. I have gone through the documentation and created a very simple scenario:
1) Windows XP "server" (it has the shared disk that i use for storage - 2TB spinning disk)
2) Created "Watch Folder" on XP Machine
3) all three MAC's can see and access the shared folder
4) Created a simple project - One 192MB video file (XMP) - applied keylight and matte choker
5) I did a render on the MAC, using only itself and standard setttings to a TGA sequence, to another shared folder on the XP server - this process took 3 minutes and 7 seconds to complete
6) I re-rendered the same project, with no changes on the mac, with the following settings:
and send them all to the watch folder (project saved there, files collected there etc.
7) once the project starts, it takes 20 minutes.
Is there a specific reason why the time would increase so drastically rather than decrease?
The macs all have 1 quad core CPU - 16GB memory and they are connected to a GB switch (all connecting at 1GBps) - If i check the tas manager on the XP pc, network utilization never goes above 25%.
Thanks guys,
Pierre

Hi Pierre,
I believe what you maybe seeing is a limitation on XP's file sharing. For the sake of simplicity network rendering treats every core as a user. So depending on which version of XP you are running it will either allow 5 or 10 users to connect to the computer at a given time. If you have other sharing services enabled (such as printer sharing) any connections to these devices count against this number of users. Given your other thread on where you discuss the rendering nodes, if possible you may want to try using windows 7 (this allows 20 users) or one of the windows server products (max # of users is based on CAL's)
Edited:
A couple of other things came to mind if you are using watch folders.
- If you are running anti-virus software on the server, ensure the watch folder directory and subfolders are excluded from scanning.
- Turn off render multiple frame simultaneously' in the preferences on the render nodes.
Hope it helps
Message was edited by: StevRo

Render Farm - How do nodes Work?

Hello,
My name is Anatoly and I've recently set up a rendering farm connected to an I-Mac as a file-server.
The Problem: Not all nodes come to work together on a project, some arbitrarily choosing to render the assets while others just watching the watch folder idly.
The Computer Setup: There are six headless nodes (all Mac-Minis). The hardware and OS specifications are below.
MAC-MINI Farm:
2.66Ghz Core 2 Duo, 4Gb 1067 Mhz DDR3 RAM, NVIDIA Geforce 320M 256Mb, OSX 10.8
2.66Ghz Core 2 Duo, 4Gb 1067 Mhz DDR3 RAM, NVIDIA Geforce 320M 256Mb, OSX 10.8
2.3Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, Intel HD Graphics 3000 384Mb, OSX 10.7.5
2.3Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, Intel HD Graphics 3000 384Mb, OSX 10.8.5
2.3Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, Intel HD Graphics 3000 384Mb, OSX 10.7.5
2.3Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, Intel HD Graphics 3000 384Mb, OSX 10.7.5
I-MAC File Server:
2.5Ghz Intel i5, 4Gb 1333Mhz DD3 RAM, AMD Radeon HD 6750M 512Mb, OSX 10.7.4
The Software Setup:
Each MAC-MINI node has a copy of After Effect CC Render Engine (version 12.0.0.404) running at startup and set to run in Watch Folder mode.
On the machine that is used to manipulate assets for rendering the following procedure is used:
Open a project
Add assets to render queue
Set Render Settings to Multi-Machine Settings
Check box in settings to “Skip existing files (allows multi-machine rendering)
Set Output Module to Multi-Machine Sequence
Select my desired output settings
Then I click File – Dependencies – Collect Files
In Collect Files Setup the following is chosen:
Collect Source Files: All
Change Render Output to: “custom name” folder
Enable Watch Folder render
Reveal collected project in Finder when finished
Maximum Number of Machines: 10
I then click Collect and save my project to a Watch Folder
From there-onwards, presumably, the Render Farm is now supposed to render out my project, but not all nodes of my 6-node farm work together. More often than not, a single (or two) node(s) render the projects while the others wait idly, watching the Watch Folder.
Can anyone please advise me of why this happens, and perhaps what step I’m missing if any to get all my nodes to render until the project is done?
Thank you,
-Anatoly

Here's a picture of render items in my queue, follwed by a picture of how many nodes are working on it:

Building render farm

Similar Messages

Maybe you are looking for