How to share a job with Compressor 4.1?

Question

Can anyone explain how to set up Compressor on two or more computers to share a coding job? I was never successful, neither with the old versions nor the new one. I have connected two computers running Mavericks via Ethernet. They appear in the preferences list of Compressor as inactive and can be selected (with a tick). Starting a job produces no error. Only the little network monitor window shows some activity: "inactive" (in white or yellow), sometimes: "not found" (in red). The computer which sends the job waits endlessly.
I deactivated the firewall, connected the computer with DHCP or fixed IP but no success. What else do I have to do?

Answer 1

Hi Steffen, hats off to you for gathering this valuable information!  I'm going to title this post:
Setup Distributed Node Processing for Distributed Segmented MULTIPASS Transcoding in Compressor.app V4.1 (2013 version)
Summary:
A quick look at those logs of yours.., Qmaster is having trouble accessing its cluster storage and probably your transcode source and target elements.
This is a bit of a giveaway - looks like the part time helpers at Apple didn’t look at it hard enough
msg="CSwampService::startupServer: servicecontroller:com.apple.stomp.transcoderx couldn't advertise server, error = error: DNSServiceRegister failed: CRendezvousPublisher::_publish"/>
<mrk tms="412112937.953" tmt="01/22/2014 20:48:57.953" pid="1975" kind="begin" what="log-session"/>
<mrk tms="412113195.964" tmt="01/22/2014 20:53:15.964" pid="1975" kind="begin" what="service-request" req-id="D6BAF26C-DD43-4F29-BD72-81BC9CF25753:1" msg="Processing."></mrk>
<log tms="412113209.037" tmt="01/22/2014 20:53:29.037" pid="1975" msg="Shared storage mount failed: exception = CNFSSharedStorage::_subscribe: command [/sbin/mount 127.0.0.53:/Users/steffen/Library/Application Support/Compressor/Storage/21D262F0-BF7EC314/shared] failed, error = 61"/>
Let’s look at this and then propose a clean method of establishing and consolidating your cluster.
Simply the Bonjour service is having a hard time trying to find you and also qmaster been running ragged trying to mount your Cluster 21D262F0-BF7EC314 storage.
Let's fix it.
Basics for the above with Compressor v4.1 and Qmaster.
much has been abstracted from the user to help easy implentation and use. This is good me thinks!
avoid ticking every option that is available on each host , such facilities aggravate and add to the complexity of your workflow environment
isolate the network subnets to use for all host access, file system paths, communication, diagnosis, monitoring  and finally data transfer (see later)
use source objects that will develop segments that will be distributed for processing. A 3 minute clip generally won't segment to cause distribution.
review any workflow gains by distributed transcoding: slow node hols up process and additional time to assemble qt segments. A cluster dedicated to an iMac or macpro can often be faster.  (Have several clusters defined and submit accordingly (long , most and short )!!)
All elements/objects used in the source and any target folders SHOULD (not must) be mounted and accessible by each Qmaster node.  You can use sym links I recall. For reasons of efficiently and easy of diagnosis.   
So.. I'd propose you try and test your setup as follows .
Try this and start from beginning.  Do your best to perform these work instructions. Try also not to deviate if u can
Simple Architecture Approach:
Your main Macbookpro or main work mac (refered to by you as "master") shall be designated the qmasterd controller that services batch submissions AND that also provides transcode services.
The other macs ("service or slave nodes) will ONLY perform transcoding services and will NOT accept batch submissions.  The slaves / service nodes  will not be able to send their jobs to your master controller for transcoding for example.
Keep it simple! and please follow these steps.
Step 1: Quiesce your clusters and Qmaster
in Compressor.app v4.1 / Preferences / Shared Computers, stop / disable all hosts (both your macs) from automatic file sharing - tick it OFF (it causes issue u have).. More later
In Compressor.app v4.1 / Preferences / My Computer, stop / disable all hosts (both your macs) stop allowing others to add batches to your host. Slide to OFF
On all hosts, quit or force out compressor.app v4.1
On all hosts (macs) use activity monitor.app or unix ps command to Force Quit ( or kill) any currently running qmasterd task and any compressord tasks if you can.
On all hosts, purge | clean out | delete  the Qmaster and compressor structures. This is documented by several of us on this forum but fundamentally you want to preserve your settings and destination templates and delete the rest.  Do these sub-steps on all hosts where u intend to deploy compressor/Qmaster for your distributed transcode processing
a. Navigate to /Users/shared/library/Application Support/ and delete the Compressor folder if it exists. By all means use the osx Finder to put it in the trash or just use the faithfully unix rm command to delete it immediately without serialisation : rm -rf /Users/Shared/Library/Application Support/Compressor
b. Navigate to your home directory ~/Library/Application Support/Compressor and move or copy any personalised values to your desktop so we can reinstate them later. Copy these two folders if they exist.
Settings
Layouts
And also copy /move any customised destination templates that u used. These are files ending in ".template"
Now using the Finder or unix command delete your ~/Library/Application Support/Compressor folder and all it's objects such: rm -rf  ~/Library/Application Support/Compressor
c. Dismount (+E or drag into trash) any shared file systems you have manually or automatically shared between your hosts. (Your two macs) . Turn off any auto mounts you may have setup in login items for Qmaster and your source and target libraries.
d. After you have done Steps 1a - 1c on all your hosts ,
then RESTART
and log back into your hosts
attempt to delete the trash on each.
     6. Check the activity monitor and confirm there are no compressord sub tasks running. Qmasterd might be running. That's ok
Step 2: set up your dedicated network for your transcoding cluster .. Important!
In this step you will isolate your cluster network to a dedicated subnet. BTW ,o DNS is needed unless you get exotic with many nodes!
You will:
use the macs Wifi network as your primary network for NON TRANSCODING work such as email, internet , iChat, iCloud and bonjour (.local) I'm assuming u have this in place
use the Ethernet on your macs as your dedicated Qmaster cluster subnet.
For this procedure will make an IP address range 1.1.1.x subnet and manually assign the IP addresses to each host. Ofcourse you can use a smart DHCP router if u have one or a simple switch and use OSX SERVER.app erver 10.9 ($HK190 , €19) on your MAcbookpro... The later for another time
a). using system preferences/network on your controller mac("master"), configure the built in Ethernet to manual IP address of 1.1.1.1 Yes yes, dhcp would be nice if we also had a dns for this subnet to identify each host (machine name) however we don't. Leave the subnet default to 255.255.255.0, and router to 1.1.1.1 .. It's not escaping anywhere! ( very private LAN for your cluster! )
b) repeat step 2a to set the other "slaves" service node only macs whose built in Ethernet to 1.1.1.2 , 1.1.1.3 and so on
c) connect these hosts (macs) ethernets together in a dedicated hub / zoned switch or if only two macs, just use a cat5/cat6 Ethernet cable
d) on each host (mac) using system preferences/network check for a Green light against the built in Ethernet
e) on each host (mac)system preferences/network to make a new network configuration (so that you can fall back incase of error) :  using system preferences/network make a new network location on each mac
- edit the Location listbox, edit and DUPLICATE the current location
- select and over type the name and change it to "qmaster work" or some name u like save and close the dialogue
- back in sys prefs / network select your new location "qmaster work" (from the location list box and "apply"
- now click the gear wheel icon on lower left and Reorder the network interfaces so that the WIfi is top (first) followed by Builtin Ethernet .
- click "apply"
- confirm Ethernet is still green status
Do this on each host (mac) .. The slave/service nodes
f) on each host (mac) verify that you can address each mac over you new subnet. There's many ways to do it however do it simply via the /applications/utilities/Terminal.app.
From mac #1 whose IP address is 1.1.1.1,
Enter:
traceroute 1.1.1.2 press return and one line should come back.
ping 1.1.1.2 and a continuous lines appear with packets and time is ms. Watch 3-4 then use control+C to stop
Do the same to the other nodes you may have such as 1.1.1.3 etc
Repeat the above from the other hosts. For example from one of the service (slave) macs say 1.1.1.2, t
Test the network path back to your main mac 1.1.1.1: using terminal.app from that slave,
Enter:
traceroute 1.1.1.1 press return and one line should come back.
ping 1.1.1.1 and a continuous lines appear with packets and time is ms. Watch 3-4 lines then use control+c  to stop
At this point you should have a solid network path between your hosts over Ethernet on the subnet 1.1.1.x
Step 3: mount all filesystems over the Ethernet 1.1.1.x subnet that are to be used for transcoding source (input | read )  and target (output | to be written )
Simplicity is important at this stage to make sure you know what being accessed.  This is one reason for disabling all the automatic compressor settings
You will use the Finder's "Connect to Server" (+k) from each slave (server) node to access the source and target filesystems on your master mac for the transcoding.
These can be saved as favourites in the "Connect to Server" dialogues
Do this:
A) locate the volumes / filesystems and folders on your mac master where your source objects is contained. Do the same for where the final distribution transcode is to be written with you user access.. "Steffen"
B) On each slave mac, use the Finder's "connect to server" dialogue to MOUNT those folders as network volumes on your slave macs
mount the Source folder. Finder / Go / Connect to Server  or +K
enter "[email protected]//Users/steffen/movies/my-fcpx-masters. ( choose you source directory path) .
Click connect & password and use the "+" sign to save as favourite
- mount the target folder. Finder / Go / Connect to Server  +k
- enter "[email protected]/users/movies/my-fcpx-transcodes. ( choose your target directory path) . Click connect n password and use the "+" sign to save as favourite
Do these for all your slave macs.  Remember you are specifying file paths over the 1.1.1.x subnet
Obviously make sure your slaves have read and write access. Yes and you could also mark these folders and Shared in Finder info then everyone can see them ... your choice
C) verify your access: on each slave use the finder to create a ftest older in those recently mounted volumes. Delete the test folder.
So now all your networks and workflow folders are all mounted and accessible by your slave hosts and it's user
step 4: Set up Compressor v4.1 and Qmaster
Care is initially required here NOT to click needlessly on options.
Recall that you purged most of the compressor.app v4.1 state information on step 1. ?
Compressor.app v4.1 will appear new when u start it. 
on the master mac 1.1.1.1 , launch compressor.app v4.1
open compressor v4.1 preferences (command+,)
using compressor.app V4.1 preferences:
My Computers tab: turn OFF . Don't allow others to process on this computer
Share Computers tab: UNTICK (disable) automatically share files.
Advanced tab: WHen Sharing My Computer listbox: select Ethernet as the preferred network interface 1.1.1.x). Don't use all interfaces : - trouble
Do not specify additional instances yet! Less to troubleshoot
On each slave mac 1.1.1.2 -1.1.1.x
launch compressor.app v4.1
open compressor v4.1 preferences (command+,)
using compressor.app preferences:
My Computers tab: turn ON (yes ON) to allow others to process their stuff on this slave computer
Share Computers tab: UNTICK (disable) automatically share file
Advanced tab: WHen Sharing My Computer listbox select Ethernet as the preferred network interface 1.1.1.x). Don't use all interfaces : - trouble
Do not specify additional instances yet! Less to troubleshoot !
On the master mac, 1.1.1.1
using Compressor.app v4.1, select destinations and add a new CUSTOM destination and navigate the dialogue to your target folder you specified in step 3b ~/movies/my-fcpx-transcodes as an example.
Use this custom destination on the batch later
in Compressor.app V4.1 preferences/Shared Computers, click the plus "+" sign in bottom left cornet to create a new cluster called "unnamed ".
- click in the name and change the name to "Steffenscluster" (I'm not connected to my network as I do this..)
- tick on the slaves 1.1.1.2 to 1.1.1.x to add the to your new cluster .. assumethee are on the right in a list.
Your cluster "Steffenscluster" is now active!
Care care! One more thing to do. You SHOULD make the cluster storage for master 1.1.1.1 available to all your slaves. This is important for this test !!
On the master 1.1.1.1, Use the finder to verify that you have these directories built from compressor.app v4.1
/Users/Shared/Library/Application Support/Compressor
and your own home directory: ~/Library/Application Support/Compressor
Dig deeper for the /storage folder in each to see the hexadecimal named folder that represents this cluster "Steffencluster"!
These should be manully mounted on each slave. Recall we DISABLED automatic file sharing.
on each slave mac 1.1.1.2 - 1.1.1.x, mount the masters cluster storage file systems. Do this for verifying access of each cluster Slave
on each slave mac, use the Finder's "connect to server" dialogue to MOUNT those folders as network volumes on your slave macs
mount Qmaster cluster storage folder 
Use Finder / Go / Connect to Server  or +k
enter "[email protected]/Users/Shared/Library/Application Support/Compressor/Storage
Click connect & password and use the "+" sign to save as favourite
- mount users Qmaster cluster storage folder 
Use Finder / Go / Connect to Server  or command+k
enter "[email protected]/Users/steffen/Library/Application Support/Compressor/Storage
Click connect & password and use the "+" sign to save as favourite
This you may have 4 new network volumes (file systems) mounted on each mac slave over your dedicated 1.1.1.x subnet!
Step5: submit a job
On master mac 1.1.1.1 launch the new compressor.app v4.1 "Network Encoder Monitor " .. Use +E . new in COmpressor.app V4.1
You should see all your nodes all cluster service points for your master and slaves.  Heres you see just one (this macbookair!)
On each host (mac) Launch the activity monitor.app and filter compressord .. There they are on each host! 
Nearly there.  
On the mac that's the master 1.1.1.1 (controller )
Submit a job:
Use finder to move some footage etc that more than 15 mins for example into your source directory folder from step 3b (eg ~/movies/my-fcpx-masters.)
In compressor.app v4.1 Add this +I to the batch
Drag your custom destination on to the batch
Set your encoding setting (apple devices best)
Open the inspector in compressor.app and select "video" and make sure MULTIPASS is checked. Then change to frame controls at your leisure.  Better means slower
Select the General tab and make sure JOB SEGMENTING is ticked!
Now cross fingers and submit it (or +B)
Step 6: Monitoring the Workflow Progress
In compressor.app v4.1 Use the active tab to watch the progress
Open the disclosure triangle and see the segments
Unfortunately u can really see which node is processing. (No more share monitor .. btw for those who care.. thats buried now in /Applications/Compressor.app/Contents/PlugIns/Compressor/CompressorKit.bundle/Contents/Embedde dApps/Share Monitor.app/Contents/MacOS/Share Monitor
Look at the network encoder monitor (cmd + E) to see the instances processing your work
Lots of small and overdetailed steps here Steffen and its worth working through.
Simply all these things need to be availble to get your cluster to work EVERYTIME.
I might update this and post a more detailed disalogue/transcript on my blog and post it here.
Epilogue:
I for one rather like the new compressor.app V4.1. Qmaster is buried and works well when not teased or unintentionally fooled.
I would like the ability to :
specify the location of the qmaster cluster storage rather than have it in the root systems file system - I used to have it on my disk array
compressor to be applescriptable
Post your results for others to see.
Warwick
Hong Kong