Compressor merging distributed quicktime components VERY SLOW. WHY?

I've got an 8 core mac pro, quickcluster with services, 4 instances.
The quickcluster works GREAT. it's super fast, good CPU Utilization. awesome.
the problem is when all the segments are done and compressor merges the bits together, it goes VERY VERY slowly, in fact it takes much much longer to re-assemble the parts of a 43gb prores file than it does to actually compress it. Activity monitor shows disk transfer speeds of 20mb/s max, and CPU utilization drops to zero. batch monitor says that is has no time remaining.. yet this process takes an hour or more.
I have compressor set up to NOT copy source to cluster since the cluster is in fact my local machine.
rendering to "this computer" only uses maybe 30% of my CPU.
***? why is is so slow? Is there anything I can do to speed it up? for one thing, It seems like the QT segments could be rendered to another drive, and there might be a bit less thrashing that way. but this slowness really is remakably silly for such a powerful system. I mean REALLY silly.

Somoflange,
Yes, the "overhead" to which you are referring is something I have experienced as well. One of the Apple Pro Apps guys said to be aware of this problem; that at a certain point, depending on input file size and number of instances you have set the rejoining of the segmented AV clips may start to take longer than it takes to render / compress them.
We experience a major problem with this "overhead" while encoding lots of 1-hour blocks of programming into MPEG-2 transport streams, across 4 Xserves and 1 Mac Pro (the Mac Pro set as cluster controller). Eventhough your setup is all local being a QuickCluster, it sounds like the same problem. We were crunching 1 hour files in under 10 minutes, but the rejoining was taking much longer.
The fix, for us, was dialing down the number of instances in your Qmaster prefs. We did some benchmark tests with the same input file for all 8 dif't instance numbers. For longer files, it actually was faster to only use 2 instances.

Similar Messages

Super slow 'merging distributed quicktime components' problem

I've got an 8 core mac pro, quickcluster with services, 4 instances.
The quickcluster works GREAT. it's super fast, good CPU Utilization. awesome.
the problem is when all the segments are done and compressor merges the bits together, it goes VERY VERY slowly, in fact it takes much much longer to re-assemble the parts of a 43gb prores file than it does to actually compress it. Activity monitor shows disk transfer speeds of 20mb/s max, and CPU utilization drops to zero. batch monitor says that is has no time remaining.. yet this process takes an hour or more. It's all going to an separate, non system internal SATA drive of course. Its sort of like compressor is saying "whew, all done with the hard work. now I'll just ease the movie segments back together while I take a nap"
I have compressor set up to NOT copy source to cluster since the cluster is in fact my local machine.
rendering to "this computer" only uses maybe 30% of my CPU.
So why is this process so slow? Is there anything I can do to speed it up? for one thing, It seems like the QT segments could be rendered to another drive, and there might be a bit less thrashing that way. but this slowness really is remakably silly for such a powerful system. I mean REALLY silly.

How long is the video you are encoding? What is the average disk activity during encoding? What version of compressor are you running?

Speed up Qmaster's "Merging distributed QuickTime segments" process?

I'm finding that Qmaster's "Merging distributed QuickTime segments" process takes as long or LONGER than the time it takes to generate the encode itself.
I found this related discussion but no resolution: http://discussions.apple.com/thread.jspa?threadID=1107862&tstart=56
There must be some un-optimized or rate-limiting bit of code in qmaster... i can't explain why the "Status: Processing: Merging distributed QuickTime segments" takes so long. We have a Compressor v3 setup with 24-36 virtual cores depending on config. The thing blazes through the encode process, doing a SD movie in about 4-6 minutes. It's beautiful how fast it runs, but when it gets back to merging segments, the disk activity does not exceed 15-20MB/sec read/write.
I tested my disks on concurrent read/write (this is a simple RAID0 array) and was seeing 80-90MB/sec concurrent read and write on the same volume. There's no knob, switch, or settings file parameter from what I can tell. It's incredibly frustrating for the disk operation to take LONGER than the encode.
Has anyone been able to overcome this problem? I am contemplating SSDs for the qmaster temp, but I don't think that would speed anything up. Does anyone know if this also exists in Qmaster 3.5?

Hi all, thanks for the replies thus far.
Jon Chappell > Compressor Repair is my best friend! Appreciate the links to the other forum with the benchmark results. I'm going to re-create those tests and see how my results compare.
All nodes are connected via 2Gb ethernet (bonded), so network speeds/transfers between cluster storage, source, and encoders are excellent. During encoding, each node is pulling data in over the network from the centralized storage server at well over 100MB/sec and placing the resulting segment on the qmaster temp area on the controller node.
The bottleneck is what happens when the controller (an 8 core 2.66Ghz Xserve with 2TB raid0'd drives) stitches the segments back together again. It's painfully slow, around 20MB/sec when I know the disks are capable of about 3-4x that. During this process, no data is being pulled over the network from the encoder nodes, as they've dumped their segments in he cluster storage node's temp area. It's all local disk i/o during the stitch operation.
Perhaps there's some single-threadedness in that process that prevents qmaster from harnessing all the resources available. All machines in the cluster are identical, so I'm convinced it's something in qmaster's design that may not be user serviceable.

I using 4s and upgrade new version IOS8 but my unit is very slow why?

I using 4s and upgrade new version IOS8 but my unit is very slow why?

And how I can get lower version. I need to be lowerversion because IOS7 is very smooth on my unit.

XI pre-prd (distributed system) is very slow

Hi,
Our XI pre-prd which is on distributed configuration is very very slow.
CI host is having 8 GB RAM, db host is having 6 GB RAM.
When we look at SM50, all the dialog proceses gets occupied by PIAPPLUSER user even though we run a single interface.
All the queues are getting stuck.
There is no problem with the interface then why all the dialog proceses gets occupied by PIAPPLUSER user and queue is getting stuck?
Any suggestion??
Regards,
Akki..............

Hi,
> gets occupied by PIAPPLUSER user even though we run a single interface.
Can you try with other User and check out whether you face the same issue.
Regards
Sangeetha

Converting .docx merges to PDF is very slow.

Hi all first post, so hello!
As the thread question states, I am trying to figure out creating PDFs from docx merges are very very slow to complete. We are using Adobe Acrobat 8, and Microsoft Word 2007 on Windows XP.
Before we upgraded to MS Office 2007 and the docx format, Acrobat 8 created PDF merges in the the 2003 .doc format just fine and was very quick. We can even created PDFs from .doc merges through Office 2007 and it is still works great and is super fast. But once we try a creating a PDF from a .docx merge, it crawls and is very slow, and is starting to be time consuming for the users. I checked out the 3-4 people that perform this function, and they all seem to have the same issue. So i would guess this would have something to do with the .docx format and the way Adobe 8 interprets the format to create a PDF from it. The documents normally fall within the 1200-1500 page range.
Acrobat 8 has been updated with the latest program updates, but I am a loss as to why .doc's create so fast, but docx's are very very slow.
Any input would be greatly appreciated. I tried to include all pertinent info you would need, if I left something out, please let me know.
Thanks!

I solved my problem. I updated the Acrobat pro 9 version several times, then I coppiend the job options according to the last step in this thread: http://forums.adobe.com/thread/780318
Now Acrobat gently merges word files to one pdf, independent if it is Doc or Docx.
My system
Windows 7 64bit
Microsoft Office 2010 32bit
Acrobat Pro 9.4.1
I solved my problem. I updated the Acrobat pro 9 version several times, then I copied the job options according to the last step in this thread: http://forums.adobe.com/thread/780318
Now Acrobat gently merges word files to one pdf, independent if it is Doc or Docx.
My system
Windows 7 64bit
Microsoft Office 2010 32bit
Acrobat Pro 9.4.1
Note: I read in another thread that if you have Micosoft Office 64bit, you may have serious problems with Acrobat 9.

TS4002 icloud mail appears to be very slow.why?

icloud mail appears to be very slow today,why?

I spoke to Apple also. They confirmed that the problem is on their end. They put my account into "trouleshooting mode", issued me with a temp. password and told me it should be fixed by Wednesday of next week. I did not hold back on expressing my disastisfaction with how long it would take them to fix - email being critical to my business.

On opening up PSE9 it runs very slow, Why?

When I open up my recently installed PSE9 program it runs very slow. all photos etc load up and are visable but it almost feels like there is a program running in the background. The cursor moves very slow and in a delayed way. I mostly have photos and very few videos, projects etc but just in case I go to view and uncheck vidoes, PDF, projects etc just in case this is what is slowing everything down but it doesn"t change anything. Nothing else is running while I am in PSE. Previuosly I was working with version 8 and I did not have this problem. Can someone help me with this this annoying problem that is keeping me from using PSE 9. Thanks Rey

You really don't provide enough information for anyone to help
How to ask a question http://forums.adobe.com/thread/416679
Some specific information that is needed...
Brand/Model Computer (or Brand/Model Motherboard if self-built)
How much system memory you have installed, such as 2Gig or ???
Operating System version, such as Win7 64bit Pro... or whatevevr
-including your security settings, such as are YOU the Administrator
-and have you tried to RIGHT click the program Icon and then select
-the Run as Administrator option (for Windows, not sure about Mac)
Your Firewall settings and brand of anti-virus are you running
Brand/Model graphics card, sush as ATI "xxxx" or nVidia "xxxx"
-or the brand/model graphics chip if on the motherboard
-and the exact driver version for the above graphics card/chip
-and how much video memory you have on your graphics card
Size(s) and configuration of your hard drive(s)... example below
-and how much FREE space is available on each drive (in Windows
-you RIGHT click the drive letter while using Windows Explorer
-and then select the Properties option to see used/free space)
While in Properties, be sure you have drive indexing set OFF
-for the drive, and for all directories, to improve performance
My 3 hard drives are configured as... (WD = Western Digital)
1 - 320G WD Win7 64bit Pro and all programs
2 - 320G WD Win7 swap file and video projects
3 - 1T WD all video files... read and write
Some/Much of the above are available by going to the Windows
Control Panel and then the Hardware option (Win7 option name)
OR Control Panel--System--Hardware Tab--Device Manager for WinXP
And, finally, the EXACT type and size of file that is causing you problems
-for pictures, that includes the camera used and the pixel dimensions

Contacts app is very very slow; why?

With my Z1, searching for contacts was very fast. It was real-time as I type the characters.
With my Z2, it is unacceptably slow. It can be a few minutes to search for contacts. Even when I go to Favouites where I have only eight contacts, the page takes a while to load. When I go to Contacts and search for a contact it can take a few minutes for the contact to be found. Then, when I select that contact, it can take a few minutes again to display the content.
Why is that?
Yes, I have many contacts (maybe 4,000+) and many are duplicates linked from Exchange and LinkedIn. But even when I turn off Auto-Sync, it can still be unacceptably slow.
And I don't keep many apps open at the same time.
What are the possible things I can try?
Thanks,
CWL88

I'm afraid it's not possible to disable the online search. Even if you disable synchronization of contacts it will still search against the server. I will forward this as feedback internally but currently you would need to remove the ActiveSync account to stop online search. A workaround may be to use another client instead like Moxier, RoadSync or Nitrodesk.
The more contacts you have the longer it will take to load all contacts if it's no longer stored in RAM. You can force contacts to stay in RAM but to avoid it being closed, close other apps when you're not using them by pressing the lower right icon and swipe them away. But Z2 has 3GB RAM so usually this is not needed but it depends on what other apps you have any how much RAM they use.
- Official Sony Xperia Support Staff
If you're new to our forums make sure that you have read our Discussion guidelines.
If you want to get in touch with the local support team for your country please visit our contact page.

When is Apple going to fix the problems with IOS8 Safari either won't work or is very slow.why can't you contact Apple themselves rather than this method?

ssince updating my IPad to IOS8 Safari does not work properly ie Marriott website it tells me no server is available yet Google works and other web search have the same problem when is Apple going to get fixes out soon I hope.Also why can't we contact Apple direct on this matter ?

Try some basic troubleshooting steps. Clear Safari, force close the app and reset the iPad.
Go to Settings>Safari>Clear History, Cookies and Data.
In order to close apps in iOS 8, you have to drag the app up from the multitasking display. Double tap the home button and you will see apps lined up going left to right across the screen. Swipe to get to the app that you want to close and then swipe "up" on the app preview thumbnail to close it.
Reset the iPad by holding down on the sleep and home buttons at the same time for about 10-15 seconds until the Apple Logo appears - ignore the red slider if it appears on the screen - let go of the buttons. Let the iPad start up.
There re are ways to contact Apple directly. Look here for help.
Contact Apple for support and service

SSAS ProcessAdd: ProcessAdd only one row to a big dimension(100 minion records) is very slow, Why?

I'm using the xmla to add a custom dsv to doing processadd, in my testing, only process add 1 row to a big dimension(already has more than 100minion rows) will cost more than 10 mins. bellow is the test enviorement:
DimensionTable: TestDimension with only one column PKey, which is the identity key, type is int.
Test machine: Vitual Machine, Windows Server 2012+SSAS 2012. the machine has 64G Memory and 8 cores CPU.
Does anyone can tell me why processadd only 1 row cost so much time? what does the SSAS do when processadd a dimension. i want internal explanations. Using profiler tool, i found many read data, is this a bug with SSAS? does it need read all the dimension
data to rebuild the dimension structure or level or index?

Hi Hong,
According to your description, you want to know why processadd only 1 row cost 10 minutes, right? ProcessAdd sound like a useful option (faster than ProcessUpdate, less impact than ProcessFull etc) but when you try and use it you will find it is not as straightforward
as the others. Here is a blog which explain the ProcessAdd option deeply, please refer to the link below.
SSAS - ProcessAdd explained
Regards,
Charlie Liao
TechNet Community Support

Exporting Quicktime movie very slow

I imported an .avi file into Final Cut and did some basic editing. When I export it as a self-contained Quicktime movie it tells me it will take something like 20 hours to export/save it?
File must be self-contained because I need to encode it as an flv on another machine.
Is there a faster way to export?
Thanks.
Pedro

AVI means absolutely nothing. Without knowing the complete specifications of your media and what you're doing in FCP there's no way to tell. FCP is a QuickTime based application, not an AVI based application. Most likely the media need to be converted into a format that's properly supported in FCP.

Java virtual machine very slow: why?

Since I have upgraded Tiger to Leopard, the startup of the Java virtual machine is unbelievably slow: for example, just executing java -version for the first time requires almost one minute. This happens only the first time the JVM is started: after that, everything seems to work properly. Any idea of what is the problem? Thank you.

I don't have this problem at all.
Which version of Unix are you using? And which version of Java are you using on the Unix computer? How old is your Computer?

First Request to BlazeDS is VERY slow. why?

Hi guys,
When I executing first RPC via AMF on my blazeDS from flex it's so slow. It's take about 10s (!)
Each next request is executing much faster, I didn't benchmark it but I'm sure it's less than 0.5s
I'm beggining in flex/blazeDS/java development so please be patient.
My web server is 6.0.14
JVM 1.6.0_17-b04
BlazeDS 3.2.0.3978
it's my first post
Regards,
Beynar

Hello Beynar,
Which Servlet container are you using? Most likely the problem you are experiencing comes from caching and other configuration derived from container's settings.
Sadly, I'm also experiencing similar problem in performance. Currently, using Glassfish and switching to Tomcat.

Intercompany is very slow, why and how to improve?

Hi All,
I have installed intercompany 1.1 PL3 in B1882 PL12 with SQL2008R2.
there are two questions : one is performance issue while another is performance logic
Firstly, I placed an PO to vendor which is a branch (global BP) on Monday morning then I can't have SO draft being created until one day later.
Does anything got stuck? how can I tune up the process?
Secondly, after I place an PO to vendor (branch B) and SO has been created in branch B, will the corresponding SO be updated if any change to original PO? Will the original PO be updated if any change made in corresponding SO?
Are there any references or document I can read?
Thanks for any help.
Hellen

Hi LC,
In the eventsender setup,
you copy the EventFilterList from c:\.....\sap\Business One Integration\EventSender\eventsenderconfig into Event Filter, and restart the eventsender.
Also refer to the Agneesh' comments.
regards,
Hellen

Compressor merging distributed quicktime components VERY SLOW. WHY?

Similar Messages

Maybe you are looking for