What is Best Practice for Adding Thousands of Tasks to CloudJob?

I am looking for some guidance on adding lots of tasks to a CloudJob.
I have thousands of tasks to add to a job, and several parent tasks running across multiple TVMs are participating in adding the new tasks to the job. However, I can reproduce my problem with a single task that tries to add new tasks to the job with
a max degree of parallelism of 15.
In each thread, I call IBatchClient.OpenWorkItemManager, then IWorkItemManager.GetJobAsync, then ICloudJob.AddTaskAsync, followed by ICloudJob.CommitAsync in order to add a new task. I will occasionally get a "Server encountered an internal error. Please
try again after some time." exception. After I delay, and then retry the entire series of operations again (from OpenWorkItemManager all the way to ICloudJob.CommitAsync), I receive the "A task instance can only be added to a single job." exception.
Since I am receiving these exceptions frequently, I am hoping that there is a better way to add a lot of tasks to the job. I see a way to add multiple tasks to a work item, but I don't know what the tasks will be ahead of time. I have to wait until I am
well into processing the job before I know what new tasks need to be added to the job.

Am I correct in assuming you're doing something like this?
Parallel.For(0, 100, new ParallelOptions() { MaxDegreeOfParallelism = 15}, async (idx) =>
IWorkItemManager manager = batchClient.OpenWorkItemManager();
ICloudJob job = await manager.GetJobAsync("wiName", "jobName");
await job.AddTaskAsync(new CloudTask("foo", "cmdline"));
You are doing some extra exception handling and stuff too as well I assume.
There are a few improvements you can/should make that ought to make your life easier.
First, the GetJobAsync() call actually does a round trip to the server -- you can avoid doing this for every task you add and reduce your round trips by half:
IWorkItemManager manager = batchClient.OpenWorkItemManager();
ICloudJob job = manager.GetJob("wiName", "jobName");
//Foo("Test", "Abc", "test");
Parallel.For(0, 100, new ParallelOptions() { MaxDegreeOfParallelism = 15}, async (idx) =>
await job.AddTaskAsync(new CloudTask("foo", "cmdline"));
Even more than that, we provide a helper method which has this functionality and performs bulk adds behind the scenes, which will reduce your round trips by about a factor of 50-100.
IWorkItemManager manager = batchClient.OpenWorkItemManager();
List<ICloudTask> tasksToAdd = new List<ICloudTask>(); //Popualte this with your tasks
await manager.AddTaskAsync("wiName", "jobName", tasksToAdd, new BatchClientParallelOptions() { MaxDegreeOfParallelism = 15 });
Additionally, we have a way where you can configure a "retry policy" without having to manually do it yourself on every call.  It's pretty handy, you can set it on the BatchClient itself, or on an individual call, below you can see I set it
on both (just to show you - in a real application usually you would just set it on the batchClient and be done with it -- it would then apply to all operations done by the batch client):
IBatchClient batchClient = BatchClient.Connect("", new BatchCredentials());
IRetryPolicy retryPolicy = new LinearRetry(TimeSpan.FromSeconds(5), 6);
batchClient.CustomBehaviors.Add(new SetRetryPolicy(retryPolicy));
IWorkItemManager manager = batchClient.OpenWorkItemManager();
List<ICloudTask> tasksToAdd = new List<ICloudTask>(); //Popualte this with your tasks
await manager.AddTaskAsync(
"wiName",
"jobName",
tasksToAdd,
new BatchClientParallelOptions() { MaxDegreeOfParallelism = 15 },
additionalBehaviors: new List<BatchClientBehavior>() { new SetRetryPolicy(retryPolicy) });
Also a clarification -- you cannot ever add tasks to a work item.  The method for add tasks happens to live on the "WorkItemManager" which a bit confusingly also manages job related stuff as well (since job is a child of work item). 
Whenever you are adding tasks, you are always adding tasks to a job, so the "workItemManager" methods related to adding tasks all take job name as a parameter.
Another issue you may be hitting has to do with .NETs service point manager... see:
https://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.defaultconnectionlimit%28v=vs.110%29.aspx
You may want to set this property to be something larger than the default of 2.
Now, it's possible after making the changes I suggested you still experience some issues -- give these changes a try and if you're still having issues come back and let us know what they are and we can help you improve the submission code further.
You can also see the sample code for TextSearch which demonstates this in the JobManager task.
https://code.msdn.microsoft.com/windowsazure/Azure-Batch-Sample-Text-87d08017/sourcecode?fileId=129811&pathId=1120079659
Hope that helps,
-Matt

Similar Messages

  • What is best practice for adding copyright page, etc?

    Hi Folks,
    I'm using iBooks Author for the first time. When I added a copyright page using Insert > Pages > Section > Copyright, I get a loose page that I was able to drag in front of the table of contents. However, the copyright page now has no formatting. That is, it spans a full two-page spread without the page break in the middle, like the rest of the book. How can I get the the same format for pages before the TOC?
    Thanks,
    Rob

    iBooks Author will definitely need a code view feature to users can hack the XML manually and fix these things.
    While I don't see that happening , (a) you can show package contents and go nuts now, if you like and (b). feel free to use iBA's menu to feedback [ http://www.apple.com/feedback/ibooks-author.html ] for any features, changes, etc. you may have in mind for the future and (c) remember, the best defense is a good offense...best way to stay out of jams w/IBA is to not get into them in the first place...practice with different templates to see which may support specific needs.
    For use of the appllcation, if you haven't seen it already, try:
    Publishing With iBooks Author
    http://shop.oreilly.com/product/0636920025597.do

  • What is best practice for...

    What is best practice for deploying applications through the IPCU to 10 ipads?
    I'm looking for a complete step-by-step of the best way to do this.
    Thanks in advance!

    Just place a modem into any console port. Ideally you use a terminal server, but is not always really needed.

  • What's best practice for logging messages in pageflow?

    What's best practice for logging messages in pageflow?
    Workshop complains when I try to use a Log4J logger by saying it's not serializable. Is there a context similar to JWSContext that you can get a logger from?
    There seems to be a big hole in the documentation on debug logging in workflows and JSP pages.
    thanks,
    Rodger...

    Make the configuration change in setDomainEnv.cmd. Find where the following variable is set:
    LOG4J_CONFIG_FILE
    and change it to your desired path.
    In your Global.app class, instantiate a static Logger like this:
    transient static Logger logger = Logger.getLogger(Global.class);
    You should be logging now as long as you have the categories and appenders configured properly in your log4j.xml file.

  • What is best practice for installing Yosemite

    I am currently on OS X Mavericks version 10.9.5 Macbook pro 13.  2.6 ghz intel for i5, 8gb 1600 mhz ddr3.
    I am now downloading yosemite 10.10.1 but since i've been reading all these negative feedback so far, i am having second thoughts if i should continue to install the upgrade or not.
    Any suggestion What is best practice for installing Yosemite?  Or is it not yet time to upgrade since the platform is premature yet?
    Thanks in advance.

    Check your apps are compatible with 10.10 - roaringapps.com
    http://www.etresoft.com/etrecheck can show what is running & installed - look for updates on the developer own sites.
    If you have many kernel extensions or startup items look for updates to them too
    Take a full bootable backup to another disk via Carbon Copy Cloner, Super Duper! or Disk Utility
    Disconnect the backup before you begin any install (ideally set it aside & leave it untouched incase you need to go back to 10.9)
    Personally I prefer a clean install when there are signs of multiple migrations (if you have upgraded several OS for a period of years). Setup Assistant/ Migration Assistant can import user data from a backup, but consider that Apps & 'other data' should be manually reinstalled from the latest versions.
    If you clean install (erase the HD before installation) then make sure you deauthorise iTunes & any other apps that are associated online (like find my Mac).
    Basically the steps you would take before selling a Mac…
    What to do before selling or giving away your Mac - Apple Support

  • What are best practices for managing my iphone from both work and home computers?

    What are best practices for managing my iphone from both work and home computers?

    Sync iPod/iPad/iPhone with two computers
    Although it isn't possible to sync an Apple device with two different libraries it is possible to sync with the same logical library from multiple computers. Each library has an internal ID and when iTunes connects to your iPod/iPad/iPhone it compares the local ID with the one the device normally syncs with. If they are the same you can go ahead and sync...
    I have my library cloned to a small 1Tb USB drive which I can take between home & work. At either location I use SyncToy 2.1 to update the local copy with the external drive. Mac users should be able to find similar tools. I can open either of the local libraries or the one on the external drive and update the media content of my iPhone. The slight exception is Photos which normally connects to a specific folder on a specific machine, although that can easily be remapped to the current library if you create a "Photos" folder inside the iTunes Media folder so that syncing the iTunes folders keeps this up to date as well. I periodically sweep my library for new files & orphans withiTunes Folder Watch just in case I make changes at one location but then overwrite the library with a newer copy from the other. Again Mac users should be able to find similar tools.
    As long as your media is organised within an iTunes Music or Tunes Media folder, in turn held inside the main iTunes folder that has your library files (whether or not you let iTunes keep the media folder organised) each library can access items at the same relative path from the library folder so the library can be at different drives/paths on different machines. This solution ensures I always have adequate backups of my library and I can update my devices whenever I can connect to the same build of iTunes.
    When working with an iPhone earlier builds of iTunes would remove any file not physically present in the local library, even if there was an entry for it, making manual management practically redundant on the iPhone. This behaviour has been changed but it will still only permit manual management with a library that has the correct internal ID. If you don't want to sync your library between machines on a regular basis just copy the iTunes Library.itl file from the current "home" machine to any other you want to use, then clean out the library entires and import the local content you have on that box.
    tt2

  • What is best practice for conditional rendering?

    i have a set of radio buttons that conditionally render another set, which in turn conditionally render a 3rd set.
    i am doing this by using a valueChangeListener on an h:selectOneRadio
    when i click yes on the 1st set it renders the 2nd set.
    when i click yes on the 2nd set it renders the 3rd set.
    when i click no on the 1st set now it correctly removes the 2nd & 3rd sets, the method also nulls the values.
    when i click yes on the 1st set, it renders the 2nd set with the old values even though they were nulled. and i can see in the debugger they are still null.
    so there seems to be an issue updating the model, but i dont know what that issue is. im sure it just has something to do with the way i am trying this conditional render. please see below for my first field and my valueChangeListener method.
    any help with what is a best practice for this scenario would be greatly appreciated.
    <h:panelGrid id="grid20a" columns="1">
                             <h:outputText value="Does this consult pertain to a specific "/>
                             <h:outputText value="planned or ongoing research project?: *"/>
                        </h:panelGrid>
                        <h:selectOneRadio id="specificResearch1" value="#{ethicsConsultBacking.bean.specificResearch}"
                             layout="lineDirection" required="true"
                             valueChangeListener="#{ethicsConsultBacking.processValueChangeSpecificResearch}">
                                  <f:selectItems value="#{ethicsConsultBacking.yesNoMap}" />
                                  <a4j:support event="onclick" reRender="form"/>
                        </h:selectOneRadio>
                        <h:message for="specificResearch1" styleClass="redText" />
    public void processValueChangeSpecificResearch(ValueChangeEvent e) throws AbortProcessingException
              String newValue = e.getNewValue().toString();
              //reinitialize
              this.setRenderHumanSubjectResearch(false);
              this.setRenderIrbSection(false);
              this.setRenderIrbProtocolNumber(false);
              this.getBean().setHumanSubjectResearch(null);
              this.getBean().setPrimaryIrb(null);
              this.getBean().setIrbStatus(null);
              this.getBean().setIrbProtocolNumber(null);
              //check condition
              if (newValue.equalsIgnoreCase(this.YES))
                   this.setRenderHumanSubjectResearch(true);
               * Clearing validation messages.
               * This will get around the issue with having a field
               * on the form that is both required & immediate.
              Iterator it = this.getFacesContext().getMessages();
              while (it.hasNext())
                   it.next();
                   it.remove();
              this.getFacesContext().renderResponse();          
         }i also tried doing this another way by just using an action method attached to the a4j:support tag, but that introduced a different set of issue. so ill leave that out for now unless that is the direction someone would like to direct my issue in.
    Thanks in advance

    Similar issue is covered and explained here: [http://balusc.blogspot.com/2007/10/populate-child-menus.html]. Not sure if it solves your problem as you're using ajax4jsf whereas I don't, but it might give new insights. To the point you might need to bind the component(s) and use setValue(null) or setSubmittedValue(null).

  • What are best practice for packaging and deploying j2EE apps to iAS?

    We've been running a set of J2EE applications on a pair of iAS SP1b for about a year and it has been quite stable.
    Recently however we have had a number of LDAP issues, particularly when registering and unregistering applications (registering ear files sometimes fails 1st time but may work 2nd time). Also We've noticed very occasionally that old versions of classes sometimes find their way onto our machines.
    What is considered to be best practice in terms of packaging and deployment, specifically:
    1) Packaging - using the deployTool that comes with iAS6 SP1b to package is a big manual task, especially when you have 200+ jsp files. Are people out there using this or are they scripting it with a build tool such as Ant?
    2) Deploying an existing application to multiple iAS's. Are you guys unregistering old application then reregistering new application? Are you shutting down iAS whilst doing the deployment?
    3) Deploying ear files can take 5 to 10 mins, is this normal?
    4) In a clustered scenario where HTTPSession is shared what are the consequences of doing deployments to data stored in session?
    thanks in asvance for your replies
    Owen

    You may want to consider upgrading your application server environment to a newer service pack. There are numerous enhancements involving the deployment tool and run time layout of your application that make clear where you're application is loading its files from.
    If you've at a long running application server environment, with lots of deployments under your belt, you might start to notice slow downs in deployment and kjs start time. Generally this is due to garbage collecting in your iAS registry.
    You can do several things to resolve this. The most complete solution is to reinstall the application server. This will guarantee a clean ldap registry. Of course you've got to restablish your configurations and redeploy your applications. When done, backup your application server install space with the application server and directory server off. You can use this backup to return to a known configuation at some future time.
    For the second method: <B>BE CAREFUL - BACKUP FIRST</B>
    There is a more exhaustive solution that involves examining your deployed components to determine the active GUIDS. You then search the NameTrans section of the registry searching for Applogic Servlet *, and Bean * entries that represent your previously deployed components but are represented in the set of deployed GUIDs. Record these older GUIDs, remove them from ClassImp and ClassDef. Finally remove the older entries from NameTrans.
    Best practices for deployment depend on your particular environmental needs. Many people utilize ANT as a build tool. In later versions of the application server, complete ANT scripts are included that address compiling, assembly and deployment. Ant 1.4 includes iAS specific targets and general J2EE targets. There are iAS specific targets that can be utilized with the 1.3 version. Specialized build targets are not required however to deploy to iAS.
    Newer versions of the deployment tool allow you to specify that JSPs are not to be registered automatically. This can be significant if deployment times lag. Registered JSP's however benefit more fully from the services that iAS offers.
    2) In general it is better to undeploy then redeploy. However, if you know that you're not changing GUIDs, recreating an existing application with new GUIDs, or removing registered components, you may avoid the undeploy phase.
    If you shut down the KJS processes during deployment you can eliminate some addition workload on the LDAP server which really gets pounded during deployment. This is because the KJS processes detect changes and do registry loads to repopulate their caches. This can happen many times during a deployment and does not provide any benefit.
    3) Deploying can be a lengthy process. There have been improvements in that performance from service pack to service pack but unfortunately you wont see dramatic drops in deployment times.
    One thing you can do to reduce deployment times is to understand the type of deployment. If you have not manipulated your deployment descriptors in any way, then there is no need to deploy. Simply drop your newer bits in to the run time space of the application server. In later service packs this means exploding the package (ear,war, or jar) in to the appropriate subdirectory of the APPS directory.
    4) If you've changed the classes of objects that have been placed in HTTPSession, you may find that you can no longer utilize those objects. For that reason, it is suggested that objects placed in session be kept as simple as possible in order to minimize this effect. In general however, is not a good idea to change a web application during the life span of a session.

  • What is best practice for dealing with Engineering Spare Parts?

    Hello All,
    I am after some advice regarding the process for handling engineering spare parts in PM. (We run ECC 5)
    Our current process is as follows:
    All materials are set up as HIBE's
    Each material is batch managed
    The Batch field is used for the Bin location
    We are now looking to role out PM to a site that has in excess of 50,000 spare parts and want to make sure we use best practice for handling the spare parts. We are now considering using a basic WM setup to handle the movement of parts.
    Please can you provide me with some feedback on what you feel the best practice is for dealing with these parts?
    We are looking to set up a solution that will us to generate pick lists etc and implment a scanning solution to move parts in and out of stores.
    Regards
    Chris

    Hi,
    I hope all the 50000 spare parts are maintained as stock items.
    1. Based on the usage of those spare parts, try to define safety stock & define MRP as "Reorder Point Planning". By this, you can avoid petty cash purchase.
    2. By keeping the spare parts (atleast critical components) in stock, Planned Maintenance as well as unplanned maintenance will not get delayed.
    3. By doing GI based on reservation, qty can be tracked against the order & equipment.
    As this question is MM & WM related, they can give better clarity on this.
    Regards,
    Maheswaran.

  • What is best practice for deploying agent(10204) on RAC 9i

    Hello,
    What would be best practice for deploying agent(10204) on RAC 9i? Should the agent be deployed on each node or should the agent be deployed on the cluster file system? What are the advantages/disavantages deploy on individual nodes vs. on cluster file system? Please advice. Thank you in advance.

    Please use agent push application to deploy agent on all the nodes at one shot
    Please refer the obe
    http://www.oracle.com/technology/obe/obe10gemgc_10203/agentpush/agentpush.htm

  • New mac - what is best practice for accounts?

    I am about to get a new mac (imac g5), and would like it to work well (ie file transfer and backup to from) my existing powerbook.
    Is there a best practice for account set up? should I use the same accounts between the two or can I set up a new account on the new mac?
    Related to this: what will key change sync give me? does that only work with the same accounts on two macs?
    thanks
    John

    With Tiger there is a migration assistant that will move everything over from your Powerbook to your new iMac G5. All you need is a firewire cable and when prompted in your first start-up select migration assistant and connect the two computers. You will need to boot up your Powerbook holding down the 't' key before you connect the two together. Good luck, Jack.

  • What is best practice for remotely managing bank of switches over POTS

    I need to be able to have a back door into several catalyst switches and ASA.
    What is the best practice for accessing them remotely. ?

    Just place a modem into any console port. Ideally you use a terminal server, but is not always really needed.

  • Best practice for adding text to Flex container?

    Hi,
    I'm having some troubles to lay a TextFlow class out properly
    inside a Flex container. What's the best practice to achieving
    this, for example adding a lot of text to a small Panel?
    Is it possible to pass anything other than a static width and
    height to DisplayObjectContainerController constructor, or is this
    not the place to implement this? I guess what I am looking for is
    the layout logic I'd normally pack into a custom Flex component and
    implement inside measure() and so on.
    My use case: a chat application which adds multiple TextFlow
    elements to a Flex container such as Panel. Or use TextFlow as a
    substitute for UITextField.
    Some example code would help me greatly.
    I'm using Flex 3.2.
    Regards,
    Stefan

    Thanks Brian, the example helps. However problems quickly
    arise if I modify it slightly to this (please compile it to see):
    <?xml version="1.0" encoding="utf-8"?>
    <mx:Application xmlns:mx="
    http://www.adobe.com/2006/mxml"
    layout="absolute" initialize="init()">
    <mx:Script>
    <![CDATA[
    import flashx.textLayout.compose.StandardFlowComposer;
    import
    flashx.textLayout.container.DisplayObjectContainerController;
    import flashx.textLayout.container.IContainerController;
    import flashx.textLayout.elements.TextFlow;
    import flashx.textLayout.conversion.TextFilter;
    private var _container:Sprite;
    private var _textFlow:TextFlow;
    private function init():void
    _container = new Sprite();
    textArea.rawChildren.addChild(_container);
    var markup:String = "<TextFlow xmlns='
    http://ns.adobe.com/textLayout/2008'><p><span>Hello
    World! Hello World! Hello World! Hello World! Hello World! Hello
    World! Hello World! Hello World! Hello World! Hello World! Hello
    World! Hello World! </span></p></TextFlow>";
    _textFlow = TextFilter.importToFlow(markup,
    TextFilter.TEXT_LAYOUT_FORMAT);
    _textFlow.flowComposer.addController(new
    DisplayObjectContainerController(_container, 200, 50));
    _textFlow.flowComposer.updateAllContainers();
    ]]>
    </mx:Script>
    <mx:Canvas width="100" height="100" id="textArea" x="44"
    y="46" backgroundColor="#F5EAEA"/>
    </mx:Application>
    What is the best way to make my textflow behave like a
    'normal' UIComponent in Flex? Should I use UIComponent instead of
    Sprite as a Container? Will that take care of resize behaviour?
    I have never before needed to use rawChildren.addChild for
    example, maybe you can explain why that's needed here?
    I realise that the new Textframework works on an AS basis and
    is not Flex or Flash specific, but this also poses some challenges
    for those of us using the Flex framework primarily.
    I think it would help to have some more basic examples such
    as using the new text features in a 'traditional' context. Say for
    example a TextArea that is just that, a TextArea but with the
    addition of inline images. I personally feel that the provided
    examples largely try to teach me to run before I can walk.
    Many thanks,
    Stefan

  • Best practice for adding and removing eventListeners?

    Hi,
    What is the best practice in regards to CPU usage and performance for dealing with eventListeners when adding and removing movieclips to the stage?
    1. Add the eventListeners when the mc is instantiated and leave them be until exiting the app
    or
    2. Add and remove the eventListeners as the mc is added or removed from the stage (via an addedToStage and removedFromStage listener method)
    I would appreciate any thoughts you could share with me. Thanks!
    JP

    Thanks neh and Applauz78.
    As I understand it, the main concern with removing listeners is to conserve memory. However, I've tested memory use and this is not really an issue for my app, so I'm more concerned if there will be any effect on CPU (app response) if I'm constantly adding and removing a list of listeners every time a user loads an mc to the stage, as compared to just leaving them active and "ready to go" when needed.
    Is there any way to measure CPU use for an AIR app on iOS?
    (It may help to know my app is small - I'm talking well under 100 active listeners total for all movieclips combined.)

  • What is best practice for integration with freight forwarders?

    Hello,
    We are looking into the possibilities for automatically exchanging data with one of our freight forwarders. We will send them our shipment information and they will send back shipment status and date information including some additional information like the house bill. Sending the shipment data from our SAP (ECC 6) system is no issue, we have done that before. What is new to us is receiving back the status updates from the forwarder. Is there a kind of best practice of where to store this information on the shipment (or in a separate tabel) and what standard function module or BADI to use for this?
    We are using ECC 6.0 sales and distribution, but no transportation management or SCM modules.
    Would, like to hear the experiences of people who have done this type of intergration with their forwarders.
    Regards,
    Ed

    SAP have added SAP TM 8.10 as a separate package which is also integrated with R/3 which means, a separate server is required if SAP TM needs to be implemented which will take care of your expectations.  For more information on this, search in Google so that you will get couple of documentations on this topic.
    G. Lakshmipathi

Maybe you are looking for

  • While syncing iPad with my computer, i get an error message: cannot back-up to this computer

    while syncing iPad with my computer, i get an error message: cannot back-up to this computer. I'd like to know why?  Is it safe to download IOS6 if the ipad is not backed up?

  • Lost recipes in Recipe Gallery app.

    I just bought new iPad mini 3.  Recipe Gallery app no longer displayed any of my recipes.  I made sure all data was in iCloud, ,I deleted app, reinstalled it, Still no recipes! Help!  Sure don't want to lose all my recipes!

  • Scenario Testing

    I am new in XI and created file to file scenario. While testing the configuration in Directory .. i found the following error. Interface Mapping              

  • Oracle Alert "NT ACTION"

    My alert is working fine but I see "NT ACTION" in my alert email.How do I get rid of that. Thanks, --John.                                                                                                                                                

  • Is it possible to save other documents like photoshop and illustrator files on icloud

    Is it possible to save other documents like photoshop and illustrator files on icloud