Strange behaviour in parallel processing (aRFC)

Question

Hi,
I have programmed a data extraction program in SAP IS-U. Due to the sheer size of the tables (900 million+) if had to use parallel processing to keep the runtime acceptable.
When running in the QA-environment I see a funny behaviour. There are about 49 dialog processes available, but the program never uses more than around 15. In transaction SARFC the settings are, that the appserver may get a load of 100% processes.
Furthermore I see that in the first few minutes a lot of jobs are being created. Then for a few minutes almost nothing happens (maybe 2 or 3 are running) and then a few minutes later it's back to normal. This cycle repeats on and on and takes around 20 minutes. The Basis-people say the system load is not very high, neither is the DB-load.
My questions are:
1) Why does the job counter never exceed the 15 jobs, even though there are plenty available (65 in total, 49 on my app server)?
2) Why is the performance so wobbly? I would expect that the slots in SM51 should always be filled with fresh jobs.
With kind regards,
Crispian Stones
P.S.
The mechanism I use is similar to the following:
loop at tb_todo into wa_todo.
  call function 'SPBT_GET_CURR_RESOURCE_INFO'
   importing FREE_PBT_WPS = available.
  check available gt 1.
  call function 'Z_MY_EXTRACTOR'
    starting new task my_taskname
    in destination my_destination
    performing my_callback on end of task
    exporting
      i_data = wa_todo.
  if sy-subrc eq 0.
    add 1 to created.
  endif.
endloop.
wait until returned ge created.
form my_callback.
  add 1 to returned.
  receive results from function 'Z_MY_EXTRACTOR'.
endform.

Answer 1

Hello,
I am facing a similar issue in one of my || processing program as well. The program, when executed with a data-set of 10,000 records takes 65 minutes to complete. One would expect it to take 650 minutes (or even lesser) to process a data-set of app. 100,000 records.
However, when I run the program for a file with app. 100,000 records the program runs OK initially (i.e; I could see multiple dialog processes getting invoked in SM50) but, after a while it starts running on ONLY ONE dialog process. I am not quite sure where, when and why this PARALLEL to SEQUENTIAL switch is happening. Due to this, the program drags on and on and on. I would highly appreciate your suggestions/tips to put this bug to sleep.
Here is a summary of the logic used...
  w_group = 'BATCH_PARALLEL'.
  w_task  = w_task + 1.
  CALL FUNCTION 'SPBT_INITIALIZE'
   EXPORTING
     group_name                           = w_group
   IMPORTING
     max_pbt_wps                          = w_pr_total   "Total processes
     free_pbt_wps                         = w_pr_avl     "Avail processes
   EXCEPTIONS
     invalid_group_name                   = 1
     internal_error                       = 2
     pbt_env_already_initialized          = 3
     currently_no_resources_avail         = 4
     no_pbt_resources_found               = 5
     cant_init_different_pbt_groups       = 6
     OTHERS                               = 7.
  IF sy-subrc <> 0.
  Raise error mesage and quit
    w_wait = c_x.
If everything went well, continue processing
  ELSE.
    CLEAR: w_wait.
The subroutine that receives results from the parallel FMs will reduce
this counter and set the flag W_WAIT once the value is equal to ZERO
    w_count = LINES( data ).
Refresh the temporary table that will be populated for every partner
    REFRESH: t_data.
    LOOP AT data.
Keep appending data to the temporary table
      APPEND data TO t_data.
      AT END OF partner.
        CLEAR: w_subrc.
        CALL FUNCTION 'Z_PARALLEL_FUNCTION'
          STARTING NEW TASK w_task
          DESTINATION IN GROUP w_group
          PERFORMING process_return ON END OF TASK
          TABLES
            data                  = t_data
          EXCEPTIONS
            communication_failure = 1      "Mandatory for || processing
            system_failure        = 2      "Mandatory for || processing
            RESOURCE_FAILURE      = 3      "Mandatory for || processing
            OTHERS                = 4.
        w_subrc = sy-subrc.
Check if everything went well...
        CLEAR: w_rfcdest.
        CASE w_subrc.
          WHEN 0.
This variable keeps track of the number of threads initiated. In case
all the processes are busy, we should compare this with the variable
w_recd (set later in the subroutine 'PROCESS_RETURN'), and wait till
w_sent >= w_recd.
            w_sent = w_sent + 1.
Track all the tasks initiated.
            CLEAR: wa_tasklist.
            wa_tasklist-taskname = w_task.
            APPEND wa_tasklist TO t_tasklist.
          WHEN 1 OR 2.
Populate the error log table and continue to process the rest.
          WHEN OTHERS.
There might be a lack of resources. Wait till some processes
are freed again. Populate the records back to the main table
            CLEAR: wa_data.
            LOOP AT t_data INTO wa_data.
              APPEND wa_data TO data.
            ENDLOOP.
            WAIT UNTIL w_recd >= w_sent. "IS THIS THE CULPRIT?
        ENDCASE.
Increment the task number
        w_task = w_task + 1.
Refresh the temporary table
        REFRESH t_data.
      ENDAT.
    ENDLOOP.
  ENDIF.
Wait till all the records are returned.
  WAIT UNTIL w_wait = c_x UP TO '120' SECONDS.
FORM process_return USING p_taskname.                       "#EC CALLED
  REFRESH: t_data_tmp.
  CLEAR  : w_subrc.
Check the task for which this subroutine is processed!!!
  CLEAR: wa_tasklist.
  READ TABLE t_tasklist INTO wa_tasklist WITH KEY taskname = p_taskname.
If the task wasn't already processed...
  IF sy-subrc eq 0.
Delete the task from the table T_TASKLIST
    DELETE TABLE t_tasklist FROM wa_tasklist.
Receive the results back from the function module
    RECEIVE RESULTS FROM FUNCTION 'Z_PARALLEL_FUNCTION'
      TABLES
        address_data          = t_data_tmp
      EXCEPTIONS
        communication_failure = 1      "Mandatory for || processing
        system_failure        = 2      "Mandatory for || processing
        RESOURCE_FAILURE      = 3      "Mandatory for || processing
        OTHERS                = 4.
Store sy-subrc in a temporary variable.
    w_subrc = sy-subrc.
Update the counter (Number of tasks/jobs/threads received)
    w_recd = w_recd + 1.
Check the returned values
    IF w_subrc EQ 0.
Do necessary processing!!!
    ENDIF.
Subtract the number of records that were returned back from the
total number of records to be processed
    w_count = w_count - LINES( t_data_tmp ).
If the counter is ZERO, set W_WAIT.
    IF w_count = 0.
      w_wait = c_x.
    ENDIF.
  ENDIF.
ENDFORM.                    " process_return
Thanks,
Muthu