Render Farm Possible in Premiere?

I multi-machine render in in AE all the time.  (For those who don't know, that's using several different computers to render the same project faster.)
Can you do this in Premiere?  The closest solution I've come up with is rendering out and assiging frames to TIFF sequences on 3 computers like this:
Computer 1:  Frames 1-2000
Computer 2:  Frames 2001-4000
Computer 3:  Frames 4001-6000
But if computer 1 finishes first, it doesn't go on and help out its buddies, it just sits there like the last intern I had waiting to get fired and not looking around for anything to do, like move the 30 c-stands and sandbags to the grip truck instead of drinking pop and watching all of us work...
Oh yeah, render farming.
In AE you can click it to "skip existing files" (or somethign to that effect) so that if AE sees that frame 1001 is done already, it automatically skips to frame 1002, and on and on until every last frame is processed.
Anything like this in Premiere that I've HOPEFULLY just been overlooking ?

Here's the straight poop,
I don't know if they'll ever get render farming up and running for anything outside of 3d effects, but the reason it hasn't been built yet is the same reason render farming took so long to develop in the first place: the workflow.
Render farming doesn't output a single file.  Why?
It uses the FRAMES and outputs a number of them from each machine in the farm, all to the same location (on any chosen machine).  However, these FRAMES are simply picture files.  That's right guys, just single picture files.  The same is true for video files really, that they are made up of nothing but picture files being rendered to the screen at super fast speeds, but they are all bundled together.  In compressed video, you lose some of the quality of the images in favor of saving speed and space, but with lossless codecs you are actually processing every frame (or every single picture).  This means that you'd have to add that processing step to the workflow for render farming.
Steps originally:
Comp A process these files as images from frame x to frame y and store in folder z
Comp B process these files as images from frame v to frame w and store in folder z
and so on for every machine
Added steps for video rendering:
comp A frames x to y scan and compress to file a
comp B frames v to w scan and compress to file b
and so on through each machine until all are finished
finally: selected comp combine files a to (whatever)
However, with this function you may get extra I-frames or B-frames, and you'll end up with jitter in the final product.  For this reason, the logic holds that instead of processing the video compression on multiple machines and combining files at the end, full video render farming should use a DATA PROCESSING BALANCE algorithm, allowing one machine to pass data between the processors of other machines while handling the file build on it's own.  Separating the file on the fly (as it is being processed) is a more useful tactic.  Start with I-frame A and I-Frame B and send the processing to Machine a, take I-Frame B and I-Frame C and send to machine B and so on and so forth.  This allows there to be I-frame overlap, and all the writing is still done with a single machine, while the processing is done across several.
Of course, if you use the first set and change it so that the i, b and m frames over lap, then match them when putting the files together into one, you could, in theory, process it much the same as standard render farming.  The second method is only faster due to the fact that all the processing would happen on the fly, and the file would be all one file right away, removing the step of putting several files together, and it wouldn't matter which sections were finished first, how many machines there were, or how many frames overlapped, as the overlapping frames would simply overlap and all the others would fall into place.  The only bottleneck would be drive speed and fragmentation of the file, which could be a last step in finalization or an intermediate step along the way.  If you use a dedicated and allocated space for the storage, on the fly is the way to go; it also works if you simply run a defrag pass over the file when a section overlaps, thereby moving other sections forward or backward in storage so they line up properly; however, this could just as easily work at the end of the whole process.
From a logical standpoint, it works.  From a programming standpoint, it's a lot of work, and it's hard on a system.  Personally, farming out a few effects and then wrapping them in a lossless file for use while editing is easy enough.  And the quality of the preview is amazing.  Rendering out to a single full video file is saved for last or for the end of the day when go home to my family; that way I don't have to ***** about how slow everything was.  I can just relax and let the PMS'ing wife yell about how nothing is going right, let the kids yell "he did this" and "she did that", and enjoy the fact that my computers are more obedient.  They do what I say without slamming doors and pouting.

Similar Messages

  • Problems with Dynamic Link and Render Farm

    We have a 10-node render farm that we use for all of our AE renders, and it works great, so we know the farm isn't the issue. But if we place a Premiere sequence into an AE file and attempt to send the After Effects file to the render farm, we get this error from every node:
    "aerender Error: After Effects error: Failed to connect to Adobe Premiere Pro Dynamic Link."
    Does anyone know of a possible workaround for getting an AE file using Dynamic Link to render to a farm? This is a very important workflow for us, and right now we're having to do local renders for any AE file that uses Dynamic Link.

    Why not ask such questions in the AE forum? Aside from that, using AE and PPRo DL at the same time can easily exhaust all your system resources and may cause any kind of issues, as effectively only one connection is meant to exist at any given time. So it's definitely in the way you use it, but not necessarily "wrong", just an unfortunate combination that makes things unworkable.
    Mylenium

  • Question on potential configurations for resource cluster as virtual machine server / beowulf / render farm

    I am looking for a starting point on where to focus my efforts in researching a desired configuration.  Of course anyone who spends anytime researching in forums knows that one thing that people hate is for people
    to ask questions that can be easily answered with a simple Google search, so let me start by saying I have done many hours of research on this already.  The problem I am having is that although there is a huge amount of information out there on what I
    am looking to do, I don't believe many people have attempted this on as small of a scale as a single user system.  Also, the amount of routes to go down seems endless, which for me so far has led to many hours of wasted trial and error.  So keep
    in mind that I am not so much looking for a handout on answers but rather a more narrowed path of searching and researching.
    What I am looking to do is take the current equipment that I have and consolidate it into a 9-12U server rack using mounted 2 or 3U atx cases so that I can have a more organized, space saving setup.  That is the
    easy part, what I have been looking into though is the combining of cores and processing power to make a "super computer" yet I use this term very loosely.  From what I have read, the closest thing I can compare the idea in my head to is referred
    to as a Beowulf Cluster, or a Render farm but I believe as the name suggests that this is more dedicated to a single task of rendering.  I like the idea of a virtual machine server where I would never actually locally use this machine but rather set up
    many virtual machines that I can remote into.  I started to compile a ProxMox Server, and this was the right idea, however when running a VM, that VM is constrained to the hardware resources of the single node that it is assigned to.  If there is
    a way to be able to have a machine that would utilize the 20 some odd cores that I would have in the cluster, along with the 40-50 gigs of ram, well that would be ideal but sounds too good to be possible. Also, ProxMox isnt quite there in compatibility for
    a home/home office user. I understand that the programs that are used have to be written in a way where the architecture allows for such, but if it could even be used for speeding up my render time with Adobe and Autodesk (which I know do have such functionality)
    that would be wonderful. 
    So without writing a whole book I am looking for suggestions on how to proceed.  Can I literally build a "super computer" that is not really that super... more mediocre super... or am I better off just consolidating the hardware but not the
    OS's and programs.   Also, I do currently use three or four different distros of Linux, mostly for ethical hacking and security type work like pen testing.  One thing I have never been a fan of though is dual booting.   Perhaps investing
    in a hot-swap HDD chassis that I could load all the different OS's on where I could just pop in whatever OS I am looking to use.  As you can see I am far from knowing where to spend my efforts in research.... THANK YOu for ANY help.
    I was recently suggested Microsoft HPC Pack as a platform?  Anything?
    Here is what I currently have:
    these are all fully built)
    Intel i5 3.7 quad core
    Nvidia (ASUS) GTX-760
    16 Gb DDR3 @ 1600 Corsair
    ASUS Sabertooth Z87 MoBo
    Autodesk - Inventor, Revit, 3Ds Max, Maya, Fusion 360, Flow, etc.
    Adobe - PhotoShop, After Effects, Dreamweaver, Illustrator, etc.
    Basically as you can see this is used for my work in 3d modeling, rendering, and drafting along with 3D printing.
    Sys. #2
    AMD 4.7-5.0 8 Core
    Nvidia GTX 760 (Same as sys #1)
    16 Gb DDR3 @ 2400 Mhz
    ASUS Crosshair V Formula Z
    Various programs for programming, app development, gaming, etc.  (This is my daily driver)
    Sys #3
    AMD 4.5-4.7 8 core
    16 Gb DDR3 @ 1600 Mhz
    Asus Crosshair V Formula Z MoBo
    Not currently using this one but is fully functional.
    Sys #4
    HP built i5 3.4 quad core
    12 Gb DDR3 @ 1600
    Nothing Special - this is the one I was using for school work but being that school and programming became one and the same I have all but abandoned it.
    THANK YOU AGAIN TO ANYONE WHO HAS MADE IT THIS FAR in reading this.

    The cluster works only for standalone webcache and httpserver installations types.
    If you want to do this you have to use dcmctl command to change from one farm to another.
    try checking with dcmctl command with out options an dyou will get a full list of options.
    then run dcmctl usage(orhelp) option to see what you need.
    As far as I remember
    you have to check the farm id of the server you want to be the master
    then in the second, you have to run dcmctl joinfarm or something like that.
    Regards.

  • Converting PowerMacs to a Render Farm?

    Perhaps like many of us, upgrading over the years, I've accumulated a collection of G4's and now a couple of G5s, and I'm reluctant to part with them...their resale value being minor, at best.  I would be more interested in finding uses for them. (well, one may become an iTunes jukebox - to replace my CDs and player eventually)
    I remember once learning about something called a 'render farm' and how several computers can be linked together to take advantage of their combined processing power.  And then there was that experiment some years ago - was it at MIT or some university? - where 1000 G5s were used to create a 'supercomputer'. 
    With 7 PowerMacs, is there a (fairly easy) way to take advantage of this possibility?  I use PhotoShop, a CG program and an NLE, and would certainly benefit from faster rendering times.  I'm not an early adaptor by any means, and until a MacPro lands in my lap, I'm content enough with what I've got.  
    So, even better if I could hitch these up like a team of horses!
    thanks,
    k

    Yes, it was Xgrid...
    http://en.wikipedia.org/wiki/Xgrid
    http://www.youtube.com/watch?v=jpqfDC_4Xos

  • Cluster Mini Render Farm

    I'm terrible at server things, I wanted to start with that and be rather upfront about it.  But that's why I'm asking out to you in the community.  I'm a video editor and in Final Cut Pro it takes literally forever sometimes for things to render when adding effects.  Unfortuantly rather than buying a Mac Pro, I instead bought a MacBook Pro.  Which don't get me wrong I love wholeheartedly, except the graphics card inside just is embarrassing, plus with how quickly the processors heat up it's hard for me to work constantly and quickly.  So I was curious, I asked out in the general communities last week about putting some minis together and using them as a small "render farm" and what I got back was some more or less maybes... so before I continue on I thought that I should pose the question to those that are basically lord in the mac tech services for help or rather "how to".  Because finding out just how much work that would have to go in it would really set the tone of whether or not I should do it. 
    I was thinking just purchasing two of the Mac Mini Servers (Quad Cores each) and using them. 
    So finally, is it indeed possible to even use two minis to process the graphical outputs of Final Cut Pro X that I'm running?  And if so, just how do I go about doing such a project?
    Thanks so much!

    I think your best bet is simply to schedule it yourself. Set up a simple automater script that reads from a folder and performs the conversion. Simply drop a bunch of files into the watch folder and let it run. You can experiment with how many concurrent processes you can run on each machine before it bogs down.
    Don't expect much out of the G3's.

  • After Effects CS6 Render Farm

    Hi All,
    Another Friday, Another test of new AE waters....
    I am busy reading up and trying to figure out how to use the three MAC's in the office to work together as a render farm. I have gone through the documentation and created a very simple scenario:
    1) Windows XP "server" (it has the shared disk that i use for storage - 2TB spinning disk)
    2) Created "Watch Folder" on XP Machine
    3) all three MAC's can see and access the shared folder
    4) Created a simple project - One 192MB video file (XMP) - applied keylight and matte choker
    5) I did a render on the MAC, using only itself and standard setttings to a TGA sequence, to another shared folder on the XP server - this process took 3 minutes and 7 seconds to complete
    6) I re-rendered the same project, with no changes on the mac, with the following settings:
    and send them all to the watch folder (project saved there, files collected there etc.
    7) once the project starts, it takes 20 minutes.
    Is there a specific reason why the time would increase so drastically rather than decrease?
    The macs all have 1 quad core CPU - 16GB memory and they are connected to a GB switch (all connecting at 1GBps) - If i check the tas manager on the XP pc, network utilization never goes above 25%.
    Thanks guys,
    Pierre

    Hi Pierre,
    I believe what you maybe seeing is a limitation on XP's file sharing.   For the sake of simplicity network rendering treats every core as a user.  So depending on which version of XP you are running it will either allow 5 or 10 users to connect to the computer at a given time.  If you have other sharing services enabled (such as printer sharing) any connections to these devices count against this number of users.  Given your other thread on where you discuss the rendering nodes,  if possible you may want to try using windows 7 (this allows 20 users) or one of the windows server products (max # of users is based on CAL's)
    Edited:
    A couple of other things came to mind if you are using watch folders.
    - If you are running anti-virus software on the server, ensure the watch folder directory and subfolders are excluded from scanning.
    - Turn off render multiple frame simultaneously' in the preferences on the render nodes.
    Hope it helps
    Message was edited by: StevRo

  • Render farm between multiple os

    Hi All,
    I have a question about setting up a render farm for After Effects CC. Basically my main computer runs on OSX Maveriks, and I have two more Windows computers that I would like to help out rendering After Effects compositions. I have set up one of the machines with the After Effects CC render engine, and the Mac and other Windows machine have After Effects CC installed as part of the 2 machine license.
    When I setup a watch folder render from the first windows machine to the second one, everything works fine, but what I want to do is setup the render from my Mac to both the windows machines who are monitoring a network watch folder.
    Permissions are seemingly all in order, but if I set it up from the Mac both PCs give me the message: "Render control file not valid".
    I have searched all over the Internet for a solution, but there seems to be nothing there. Also, I suspected it might have something to do with the way a Mac does line endings in files compared to a Windows machine, so I rewrote the RCF file on the Windows machine itself, making sure it was in the right format, but this did not work either.
    Any help?
    With warm regards,
    Daniel

    hello fastlane, for firefox you can use firefox sync to share bookmarks, passwords & history between your three OSs - [[How do I set up Firefox Sync?]]
    & for keeping your mails synced it's possibly the easiest way to use a mail-provider that offers IMAP, which keeps the mails on the server so that your mail program on any OS has the same data when it's connecting...
    (but we're not specialised on thunderbird here, you might head over to https://getsatisfaction.com/mozilla_messaging/ for more detailed questions).

  • How to make a render farm with FCP X / Compressor??

    Wondering how to setup a min render farm with FCP X on three machines. Played around with setting in Compressor but never see any machine available when I go to render on each machine?

    Ok, so I'm using the wrong terminology.
    I want to have all my machines transcode for me, is this possible, yes or no and how do I set it up. I've buggered around with all the settings in the link above but I'd get anything to show up?
    Also I believe that compressor needs to copy the entire file to be rendered over the network for the other machine to work on it - is this correct ? I'm on wifi guess I could connect machines together via ethernet.
    I'm looking at 4x time to do a simple render on prores proxy material and when I have to do it on a 2 hour wedding video I'll be looking for the nearest bridge to jump off!

  • Is there any point in a G4 render farm?

    Hi all.
    I've been getting quite intrested in the idea of setting up a very small render farm laterly as I have the option of getting some old G4s for free.
    I currently have a 2.0Ghz DP G5 as my main mac running FCP2 and Shake and was already grabbing a G4 466 digital audio to set up as a file/print and back up server running tiger server, a postscript RIP and retrospect. I have the possibility of getting atleast one more G4 466 DA and maybe a couple of G4 400 AGPs with Gigbit ethernet cards. As the main use would be farming out DV to mpg2 compression via compressor and shake renders (though not large shake stuff to begin with as very new to this) is there any point in looking into this or am I going to need to lots of G4s to make a real improvement in render times.
    Also is RAM a major consideration. For example if I got 4 G4s with say 512Mb ram would it be better runing all four or using three but nicking the fours ram to give more ram in the three machine set up?
    Any advice would be appreciated
    Cheers
    Steve

    I vote no. G4 isn't the problem so much as 400mhz is. The ram shouldn't be a problem.
    But you could test if you set up Compressor on one G4. Compress a job. Compress the same job with the same settings on the G5.
    If the G5 is 4 times (or even 3, since time is spent sending the render data over the network between the G4's slow system busses) as fast the single G4, a render farm will not help you.
    Using the G4 and G5 together in the same cluster does not work because Compressor looks at all processors and divides the job into twice as many segments. In this case, the G5 finishes it's two segments years before the G4's and then sits around waiting. You could try splitting the G5 into 4 instances, but then you have 8 slow processors attacking your job. Not very efficient.
    Good luck though. You should test the one G4 just so you know. Report back if you do.

  • FCP 2 / Maya Render Farm

    I'm setting up a render farm for one of the labs I manage at my University. I currently have 10 2GHz G5's on a gig switch behind an 8-core MacPro as the controller. I want to know if its possible to set up the 250GB drives that is in those 10 machines as a single share drive mounted to the controller? If not by native software, what about 3rd party software?
    Is this there anything I should know about the newest version of Maya with Tiger and would it need to be installed into each of the nodes if they are in a dedicated render farm?

    I dont know anything off the top of my head to answer your first paragraph. You should use a file server and mount the volume on all the nodes, which is easy.
    And yeah, every node needs to have Maya installed. The good thing is they dont need to be licensed to do command line rendering, unless you use mental ray renderer.

  • Building render farm

    Hi!
    I'm new to all this video editing thing, and have been trying to edit and render with my laptop. Now i had the chance to help a company to get rid of 10 desktop computers with intel core duo prosessor from 1.8 to 3.0GHz. Is it possible to build a rendering farm from these unit? only problem is that none of them has a hardrive, do i need to buy hardrives to make somethins out of them?
    Highly appreciating Your time and help!
    Ilkka

    For an AE render farm, you need a hard drive with a Windows or Mac OS in order to install the render engine software.

  • Render Farm for FCP 7 PLEASE HELP ASAP!

    I have been searching all day for a way to get a render farm going on FCP. Problem is, any of the places that have any type of remote answer are from at least a year ago, and apply to FCP 6.
    I have no experience with setting up a render farm, but I need to render out a lot of footage with effects ASAP to meet a deadline.
    If anyone out there could A. tell me if it's even possible to set up a render farm on FCP 7, and B. how to do it, in dummy's guide terms, because I am setting this up for the first time in desperation.
    PLEASE HELP!!!
    ~ Sean

    Just as well; deadlines are not where you want to try to attempt unknown hardware hacks.
    When you get time, after your deadline has expired, tell us about your project and how you allowed your effects to go unrendered for so long. We can help you not do that again.
    bogiesan

  • Hardware recommendations for render farm?

    Hello,
    what would you recommend for a small render farm for mixed AE projects (mostly HD content, short projects with only a few seconds and layers and bigger ones with multiple HD layers and up to 15min or longer.)
    is it better to have more cores (AMD opteron servers with 48 cores for example) or better more performance per core but less cores (Intel Xeon servers)?
    And how much memory would be optimal?
    What can you recommend for network connection to the shared storaged, how fast shall this be?

    I like Harm's analogy better then mine. One prize cow vs. 20 regular cows. The imagery is very clear. In a production environment, where volume counts, more is better then a single top notch one. A single cow can only produce so much mik per day, no matter how extraordinary it is.  But 20 can theoretically produce 20 times more (practically, maybe  17 to 18). Quantity, not quality. I'm not saying that you should get crappy hardware. Just good hardware, not high performance ones.
    As you know, it's not because you buy a computer that costs twice as much, that you'll get twice the processing power. First of all, forget dual CPU computers for a render farm, it's not worth it. Get a regular quad core. As you compare prices vs. performance (do a chart), get the CPU that gives you the best bang for the buch (just before the price curve goes up).
    I personally think 32GB is overkill in a render farm environment. 16GB is way more then enough. Remember that you're not doing a RAM preview farm (if such a thing exists), but rather, fractioning the render process to various computers. While 16GB is good, 2GB isn't. But 8GB can be an nice compromise. Remember, you're trying to drive the cost down of a computer, so you can buy many computers. But RAM is cheap, so, best bang for the buck, 16GB can be possible. Then again, it depends on other overall factors to keep the price for the computer down.
    Network speed. Yes you could get 10GB NICs and a 10GB switch. But that can drive the price up substantially. If you can get a good price on them, great. If not, stick to 1GB. Get a motherboard that has 2x 1GB NICs on board, so you can aggregate the network connections together to get 2GB. You could eventually add a quad 1GB PCIe card down the road. The computer that has the project assets should have an aggregated quad (or more) NIC inside. You want to be able to push the media as fast as possible to the render farm.
    Other things to consider... or rather, not consider. Graphic card is a non issue on a server farm. AE is all about CPU. Some mobos come with on-board graphics. Usually, it's cheaper, and consumes less power. You also want to drive the energy costs down when you multiply by X number of computers. Just get a KVM solution that will be able handle X computers. You don't need a RAID on render farm computers. A regular hard drive will do. I'd even go with a green drive that consumes less power.
    And depending on how many computers you have, to reduce the footprint, you might want to get low profile rackmount casings and put them in a rack.  Plus think of putting it in it's own room, because it'll be very noisy!

  • New Mac Owner - Render Farm Question

    I am new to the mac world. I have purchased the macbook pro, 17" with 2 gig ram and 7200 rpm HDD. I am using the machine for video editing, effects, and DVD authoring. I have read through several files about compressor2 and Qmaster and their ability to improve render time when producing a video file. Being new to the mac world, I do not have multiple machines to link together via network to utilize these amazing applications. I do, however, have several PC's that are high speed, dual processor, with loads of ram (previous video editing systems). Can a PC run the Qmaster and/or compressor software and allow for the render farming? Any input and advice is greatly appreciated.

    By default you CANNOT run anything called "QMaster" or "Compressor" on a Windows PC.
    BUT!
    There is something buried deep in the Apple QMaster documentation that describes how to use Terminal on your Mac and the command line to set up any Unix or Linux machine as an external [what Apple calls,] "EXTENDED NODE."
    But to do this, you have to have at least ONE extra Mac to use as the "INTERMEDIATE NODE."
    How it works is that you use Terminal and command line codes to make the EXTENDED (non-mac) nodes talk to (and offer their processors to) the INTERMEDIATE node, which then says to the Cluster Controller, "Hey! I've got more SERVICES NODES to offer you."
    So, in a manner of speaking, this process tricks the Cluster Controller into thinking that the Intermediate node has more services to offer than it really does, because the Intermediate node is kinda "subleting" the processors of the Extended Node(s).
    So, it's up to you to decide if it would be worth it to install Unix or Linux on all those PC boxes you have, but it would be possible, provided you purchased another mac to use as an Intermediate Node. Perhaps an Intel Mac Mini would do the trick.
    Good Luck.

  • After effects and premier render farm

    Hey
    Im working on a Mac Pro mid2010 and Mac pro Late 2013.
    I want to know if and how can I work on one of them but when I render or creating ram previews make them "work together" ?
    like a render farm ...
    Thanks

    Yes  let's say its a really small farm.!
    There are no differences in the operating systems (10.9)
    and im using cs6 in both of the machines (but I am planing to upgrade to cc).
    so how can I create this "farm" thing ? And if I got you right, it will work only
    when  exporting a file. not in a ram previews. Right?
    thanks

Maybe you are looking for

  • Same interface name in alert for the synchronous messages

    Dear Friends,    I have configured the alert for my interfaces. In the container i have added the message id, sender interface and receiver interface variables. While the error occurs, the alert is getting triggered. But in the alert long text in bot

  • Music gone?

    I just got one of those Product Red nanos, and when I plugged it into my computer to update it, all the songs went on without a problem. Now when I open iTunes, all the songs are gone from my library. I do have an iPod mini that doesn't work anymore

  • Koyote Soft HD Converter V1.6 Problems?

    The free Koyote Soft HD Converter is a popular choice for converting AVCHD to MPEG2 HD for use in Premiere Elements. My experiences with that software have been with the 1.4 version of it. One of the first persons who brought this software to my atte

  • JMS C API library for AIX

    Does anyone knows where I can locate the library for AIX without having to install Websphere or MQ?

  • Design a possible set up to simulate the vibrations of a suspension

    Hello everybody, I have no clue how can I solve this lab practice (Pinctures attach) . I guess I have to set up the rotary potentiometer to act like a suspension, Do you have any idea where should I start? If you guys have any solved examples in labv