Parallel processing on multiple IdM instances -- real enterprise class

Question

Hi all. We run what we earnestly hoped would be a true enterprise class IdM v6 SP1 installation: 2x v440 Sol9 app servers each with 2 IdM instances (AS virtual servers), each host dual CPU and 4GB RAM available; connected to a 4 node dual CPU Oracle RAC cluster.
But our performance, to use a technical term, sucks bollocks, and has done for >12 months. The main area where this hurts is when we run a DBTable ActiveSync process to update the user base and any associated resources.
We suspect a few things, but ultimately the most frustrating and intractable problem is this: IdM processes all the Update User tasks stemming from the ActiveSync poll one by one, sequentially, in series, and all on the originating IdM instance. So if we have, say, 5000 updates to process, we watch 5000 sequential User Update tasks, one after the other. Even if each task takes only a couple seconds, we often notice inexplicable gaps of many seconds between one sequential task completing (start time + execution time) and the next beginning. The end result is a throughput rate of usually less than 300/hr -- more than 10hrs to process those 5000 updates!
Despite setting the [custom] Update User wf to execMode='async', IdM seems to refuse to run these tasks in parallel. In an installation of this size and resource capacity, this is excruciating. Plus there's the fact that as I write, and as it crawls along, the IdM instance running the tasks is showing up in prstat like this:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
28946 sentinel 2583M 2379M cpu3 28 0 87:27:53 25% appservd/177
That's quite a lot of resource-use for not a lot of outcome...
So, does anyone know: how can we get these tasks to run multiple in parallel rather than sequentially; how can we get the task load to be spread across all 4 available IdM instances rather than just the one that executed the ActiveSync poll?
In long suffering desperation, any help greatly appreciated!
PS - 2 things regarding parallel:
1. on the TaskDefinition for the Update User wf, there is also an attribute named 'syncControlAllowed' -- anyone know what this means/does? Would setting it to 'false' perhaps give us true async/parallel mode?
2. I suspect forcing the task into background could potentially give parallel behaviour, but the problem is that Update User is of course called by any user update task, and when done interactively the administrator may not wish to have the task run in the background.

Answer 1

Hi,
i'm afraid i don't know an easy answer to your question. But there are several things that should be considered:
1) 12s average update time seems a lot. I'm sure that can be optimized. Have you thought about a dedicated sync form (assigned to the proxy admin used for active sync)? If you have that you might also consider setting viewOptions.Process there to use a simplified workflow optimized for your sync process. Take a look at the dynamic tabbed user form example as well - some of the resources you have may not be subject to synchronization and those should be ruled out by setting the sync forms "TargetResources" property to an aproptiet value.
2) i recently found an interessting way of parallelizing tasks. This is potentially dangerous as you have to build something in that prevents this from being a "fork bomb". Still some example code for a simple scenario where you just want to sync global.email, populated by your AS form. If the workflow consisted out of "start", "nonblocking task launch" and "end" the "nonblocking task launch" had an action like
<Action id='0'>
    <expression>
      <block>
        <defvar name='session'>
          <invoke name='getLighthouseContext'>
            <ref>WF_CONTEXT</ref>
          </invoke>
        </defvar>
        <defvar name='tt'>
          <new class='com.waveset.object.TaskTemplate'>
            <invoke name='getObject'>
              <ref>session</ref>
              <s>TaskDefinition</s>
              <s>Nonblocking Workflow</s>
              <map/>
            </invoke>
          </new>
        </defvar>
        <invoke name='setSubject'>
          <ref>tt</ref>
          <invoke name='getSubject'>
            <ref>session</ref>
          </invoke>
        </invoke>
        <invoke name='setVariable'>
          <ref>tt</ref>
          <s>accountId</s>
          <ref>user.waveset.accountId</ref>
        </invoke>
        <invoke name='setVariable'>
          <ref>tt</ref>
          <s>email</s>
          <ref>user.global.email</ref>
        </invoke>
        <invoke name='runTask'>
          <ref>session</ref>
          <ref>tt</ref>
        </invoke>
      </block>
    </expression>
  </Action>The workflow "Nonblocking Workflow" then would have accountId and email available in its variables and if it is defined as "async" it will really be launched that way - build in something that prevents your system from exploding...
3) probably safer than trying what i implied in 2) (i only used this for a totally different task that can not explode) you could consider to have several instance of the database table resource adapter. Lets say your primary key in the db is "employeeId". If you define for seperate resources, each handling employeeIds modulo [1,2,3,4] only you could distribute the load among your cluster. I did a similar thing with flatfileactivesync before.
4) back to the average 12s again. If you don't have some slow resources this could mean that the parallel resource limit kicks in. Take a look at waveset.properties about limits like this.
Synchronization is not the ootb strength of IDM - but with some optimization you should be able to get to reasonable results.
Regards,
Patrick