Siebel EDQ batch job performance

Hi there,
We are using Siebel Public Sector 8.2.2.4 with EDQ 11.1.1.7 version to clean up existing contact records and also set up real-time deduplication for ongoing contacts. We didn't tune any parameters both from Siebel and EDQ side till now.
We observed that the process is taking 45-50 minutes to process 6000 records. We noticed that the process is taking more time while inserting the records from Siebel to staging database. The EDQ contact deduplication is just fine and it it taking around 8-14 minutes based on number of records to process.
Is there a way we can tune anything from Siebel side to improve the performance while loading the data into staging database. May be Batch max number of records or number of DB connections or sessions etc to improve the performance?
Also, can you suggest if we could run the jobs from Siebel ( by splitting into multiple batches) in parallel? I am not sure if this is an option as the parallel tasks uses the same sharing database and it might have an conflict.
Please suggest.
Regards
Siva Sonti

Hi,
So to understand correctly, these are true batch jobs with the data being handed over using the staging tables.
It seems like the problem here - which we have seen before - is the performance of the candidate selection query in Siebel for incremental batch jobs - the piece where the candidate records for the selected batch of records to match are selected. How long does a Full Batch job take to run (as this does not need to make this query)?
Also, is this Siebel CRM or UCM? If UCM, you can use the 'UCM Batch' workflow which although it calls EDQ in real-time can be scaled using as many tasks in Siebel as the system can handle; customers have been able to load up to 9000 records/minute with this method.
The nature of the shared staging database means it only supports a single job for data from a single business component (e.g. Contact) at any one time, as you suspected.
If you are really 'loading records' into Siebel, you may be better of loading them using the EIM interface - perhaps with EDQ pre-matching the data and directly writing to the EIM tables.
Regards,
Mike

Similar Messages

Batch Job Performance Issue in BW

Hi All
I would like to if there are any performance tuning methods for batch jobs in BW. Few jobs are taking much longer time and i need to figure out a method to tune them.
Thanks in advance for your help
Regards
JP

Hi JP,
Dinesh is right, you need to consider the tuning for specific scenarios. If some of the jobs are taking longtime just access at what point they are taking time by going to the JobLog in source sytem/BW, and try to take up the optimization of the resources used inthe source system. For Eg: It can be even in the Extractor. I had come across such issue with long loading times for few ledgers in FISL. But could encounter that by spitting load with small number of records to BW with an ABAP program incorporated inthe InfoPack with the selections for the Reconcilation key.
In this way, you need to first identify the delay for the jobs. Then you can plan the things accordingly..
Hope this will help you..
Regards,
Madhu

Batch Job Performance Issue.

Hi Experts,
I am facing a performance issue in a background job, on a selection screen it contains month start date, end date and also effective date as 01/01/2008.
Client executing this report once in every month. Since the effective date is constant, the database contains large volume of data i.e. more than 3 lakh records Presently. Finally the job gets cancelled.
Could you please suggest what should be done in this scenario? Is there any concept like splitting the records as batches like multiples of 20 thousands.
Thanks and Regards,
Stanly

hi
As you said you can split the records as packages. ( as teh issue is related to low memory of internal table )
select ....package size...
*-- program logic
endslect.
regards,
Naveen

Batch job performance

Hi guys,
Our requirement is such that we have to process 3000 to 5000 back orders on daily basis.
If we use the standard transaction V_RA, it takes min 45 min to 1 hour to process all the back orders.So standard program is not meeting our requirement
Is there any way we can process 5000 orders in 3 to 5 min via batch program.
We will create Z program by copying V_RA and can do required modifications
Please advice
Regards
Mano

Hi ,
Discuss with your abaper to schedule the job more than work process ( distribute the work on multiple work process)
then its possible to get finished whatever time you want.
Thanks
siva

Can a long running batch job causing deadlock bring server performance down

Hi
I have a customer having a long running batch job (approx 6 hrs), recently we experienced performance issue where the job now taking >12 hrs. The database server is crawling. Looking at the alert.log showing some deadlock,
The batch job are in fact many parallel child batch job that running at the same time, that would have explain the deadlock.
Thus, i just wondering any possibility that due to deadlock, can cause the whole server to be crawling, even connect to the database using toad is also getting slow or doing ls -lrt..
Thanks
Rgds
Ung

Kok Aik wrote:
According to documentation, complex deadlock can make the job appeared hang & affect throughput, but it didn't mentioned how it will make the whole server to slow down. My initial thought would be the rolling back and reconstruct of CR copy that would have use up the cpu.
I think your ideas on rolling back, CR construction etc. are good guesses. If you have deadlocks, then you have multiple processes working in the same place in the database at the same time, so there may be other "near-deadlocks" that cause all sorts of interference problems.
Obviously you could have processes queueing for the same resource for some time without getting into a deadlock.
You can have a long running update hit a row which was changed by another user after the update started - which woudl cause the long-running update to rollback and start again (Tom Kyte refers to this as 'write consistency' if you want to search his website for a discussion on the topic).
Once concurrent processes start sliding out of their correct sequences because of a few delays, it's possible for reports that used to run when nothing else was going on suddenly finding themselves running while updates are going on - and doing lots more reads (physical I/O) of the undo tablespace to take blocks a long way back into the past.
And so on...
Anyway, according to the customer, the problem seems to be related to the lgpr_size as the problem disappeared after they revert it back to its orignial default value,0. I couldn't figure out what the lgpr_size is - can you explain.
Thanks
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
"Science is more than a body of knowledge; it is a way of thinking" Carl Sagan

Creating a simple batch job to run every month and perform a deletion

dear all;
This is just for learning purposes
I have created a simple table below
create table t1
vid varchar2(30),
quantity number(6,2),
primary key (vid)
insert into t1
    (vid, quantity)
values
    ('G1', 2);
insert into t1
    (vid, quantity)
values
    ('G2', 3);
insert into t1
    (vid, quantity)
values
    ('G3', 0);Now, I would like to create a simple batch job, that runs every month which is used to delete all vid that has a quantity of 0...How do i go about doing that. I have searched the web but can't seems to find any useful info so far..
Edited by: user13328581 on Sep 8, 2010 8:32 AM

you are missing a semi-colon in your package procedure call inside your begin ... end.
begin
dbms_scheduler.create_job
    job_name => 'jobdeletezeroquantity'
    , program_name => 'PLSQL_BLOCK'
    , job_action => 'BEGIN mfg.testpkg.deletezeroquantity END;'             <-- missing semi-colon on mfg.testpkg.deletezeroquantity
    , start_date => systimestamp
    , repeat_interval => 'freq=monthly; byday = 1'
    , end_date => null
    , enabled => True
    , comments => 'used to run a delete procedure every month'
end;should be:
begin
dbms_scheduler.create_job
    job_name => 'jobdeletezeroquantity'
    , program_name => 'PLSQL_BLOCK'
    , job_action => 'BEGIN mfg.testpkg.deletezeroquantity; END;'
    , start_date => systimestamp
    , repeat_interval => 'freq=monthly; byday = 1'
    , end_date => null
    , enabled => TRUE
    , comments => 'used to run a delete procedure every month'
end;

How to find out batch job failure and taking action:

Normally We will monitor the batch jobs through transaction code sm37 for job monitoring. In SM37 we will give a batch job name date and time as input. In the first step we will check the batch job for the reason failure or check the spool request for the batch job for failures an help in analyzing the error
I understand from the my experience is that the batch may fail due to below reasons.
1.,Data issues : ex: Invalid character in quantity (Meins) field >>>> We will correct the corresponding document with correct value or we will manually run or request the team to rerun the batch job by excluding the problematic documents from the batch job variant so that it may process other documents.
2.Configuration issues : Materials XXXX is not extended for Plant >>>> we will contact the material master team or business to correct the data or we will raise sub contract call with support team to correct he data. Once the data been corrected and will request the team to rerun the batch job.
3.Performance issues : Volume of the data being processed by the batch job ,network problems.>>>Normally these kind of issues we will encounter during the month end process as there will lot of accounting transactions or documents being posted business hence it may cause the batch job failure as there is enough memory to complete the program or select queries in the program will timeout because of volume of the records.
4.Network issues. : Temporary connectivity issues in other partner systems :Outage in other partner systems like APO or other system like GTS will cause the batch job failure as Batch job not in position to connect other system to get the inforamtion and proceed for further steps.Nornmally we will check RFC destination status by running a custom program weather connectivity between system are in progress or not. then intimate other partner system for the further actions, Once the partner system comes online then we will intimate the team to restart or manually submit batch job.
Some times we will create a manual job by transaction code SM36.

I'm not sure what the question is among all that but if you want to check on jobs that are viewable via SM37 and started via SM36. The tables are TBTCP -Background Job Step Overview and TBTCO - Job Status Overview Table.
You can use the following FM to get job details:
GET_JOB_RUNTIME_INFO - Reading Background Job Runtime Data

How to create a batch-job dynamically?

Hello,
I would like to create a batch-job from within my abap code.
Therefore I want to determine a abap-report to be executed every X-Hours.
Do you have some sample code for creating a batch job?
I need to do it in abap because I want to create a TN to trigger the code creating the job.
So the user interaction will be minimal to start a periodic batch job.
thanks.

Hi Daniel,
select options could be stored by export/import to/from database:
      DESCRIBE TABLE t_vkpos LINES n.
      READ TABLE t_vkpos INDEX n.
      lastmaxvbeln = t_vkpos-vbeln.
      PERFORM get_varimemkey USING p_vari CHANGING vkposmemkey.
      EXPORT lastmaxvbeln TO DATABASE indx(zk) ID vkposmemkey.
      IF NOT sy-subrc IS INITIAL.
      PERFORM get_varimemkey USING p_vari CHANGING vkposmemkey.
      IMPORT lastmaxvbeln FROM DATABASE indx(zk) ID vkposmemkey.
      IF sy-subrc IS INITIAL.
        ls_vbeln-sign   = 'I'.
        ls_vbeln-option = 'GT'.
        ls_vbeln-low    = lastmaxvbeln.
        APPEND ls_vbeln.
      ENDIF
FORM get_varimemkey USING    vari TYPE raldb_vari
                    CHANGING memkey TYPE char22.
IF vari IS INITIAL.
    RAISE get_varimemkey_unexepected.
ENDIF.
CONCATENATE 'Z_VKPOS_' vari INTO memkey.
ENDFORM.                    "
regards
Walter Habich

Reduce/Eliminate the start up time for a batch job

I'm looking for ideas on how I might be able to modify either my jobs, workflow or data flow in an effort to reduce the "start up" time that I'm encountering.
Currently I have a batch job we execute several hundred times a day. One of the parameters we pass to this job is a file path. The data flow reads a flat text file in this file path, processes the data through several transforms, and writes a flat text file back into the file path.
The actual processing time varies with the number of records being processed -- as one would expect.
However, even for the smallest - single record - input file the job takes between 40 and 50 seconds to complete. If I watch the trace file during this time it appears that 95% of this time is spent getting ready to run the job. The actual processing is almost instantaneous.
Is there anything that I can look at changing in order to improve the overall performance of these jobs?
I should mention that I am aware of setting up real time jobs, and how they would not require this "start up" time for each job submitted. However this jobs we process could between 1 to 1,000,000 records. Passing that many records in a single web service call does not seem reasonable. I would deal with issue on the submission side and then the potential timeout issue when waiting on the response -- which could also be quite large.

Thanks Michael --
I must admit I'm a bit lost when it comes to the issue of codepages.
Is there any documentation you could refer me to with regard to setting the correct codepage for the data sources and targets?
Also, from the trace log I see the following statement:
Changed engine runtime locale <eng_us.cp1252> to <Unicode (UTF-16)> because data flow <DF_Suppression> contains a transform that requires processing with the Unicode (UTF-16) code page.
Other than the data source and data targets I do not see any option to set, or change, the codepage within any of transforms that I am using. (URAC, Name Cleanse, Query, Case, Match transforms are used)
Right now my source and targets are set to "default", which would seem to need to be changed. Where is the "default" set?
Sorry for so many questions about this.

Performace tuning: how to pass data between different batch job programs?

Hi everyone,
    now i have one problem about performance tuning using threading in SAP programs: split one big program into two programs - one is main program and the other is sub program. using batch jobs, we can submit multi jobs of sub program at the same time.
    does anybody know how to pass data between different batch jobs? I don't want to use temp files. can ABAP memory can implement this?
    thanks!

Wei,
Yes we can transfer the data by using
SAP Memory OR ABAP Memory.
Ex: V_count TYPE i.
V_count = 100.
LOOP AT itab.
IF v_count EQ 25.
Here For every batch job
EXPORT data TO MEMORY ID 'ABC'
   Function module
    JOB_OPEN
   JOB_SUBMIT
   JOB_CLOSE.
ENDIF.
ENDLOOP .
IN your 2nd program.
INITIALIZATION.
IMPORT data FROM MEMORY IF 'ABC'.
FREE memory if .---When you free the memory you will get recent data.
Don't forget to reward if useful.

SAP threading-how to pass data between different batch job programs?

Hi everyone,
    now i have one problem about performance tuning using threading in SAP programs: split one big program into two programs - one is main program and the other is sub program. using batch jobs, we can submit multi jobs of sub program at the same time.
    does anybody know how to pass data between different batch jobs? I don't want to use temp files. can ABAP memory can implement this?
    thanks!

Passing Data Between Programs
[http://help.sap.com/saphelp_47x200/helpdata/en/9f/db9df735c111d1829f0000e829fbfe/frameset.htm|http://help.sap.com/saphelp_47x200/helpdata/en/9f/db9df735c111d1829f0000e829fbfe/frameset.htm]

Print selected pages of a PDF file to printer as a batch job using VB6

I have the following:
1) 1,000 multi-paged PDF files (say 00000001.pdf, 00000002.pdf, ..., 00160000.pdf)
2) A list of pages for each document to print
(e.g. 00000001.pdf: 2,3,5,7
00000002.pdf: 3-8, 10
00001000.pdf: 5,6,7,10)
I want to perform a single batch job to print the specified pages of all 1000 PDF documents to default printer.
I am using VB6, Acrobat Reader 9 installed.
I am thinking of writing a function like that:
Private Function PrintPDF(byval sPDFFilePath as string, byval iPageNum() as integer) as boolean
and then write a looping routine to print the files one by one.
Seems that Acrobat SDK has such feature.
Please advise how can I do this?

I downloaded the Arcobat SDK and now learning to use it with its sample.
However, I got stuck when I run the program BasicIacVB.
I got the following error message:
Cannot create ActiveX component.
When running the line
gApp = CreateObject("AcroExch.App")
I run this program on my PC with VS2005 installed.
What extra step do I need to do to use the sdk?
Seems there's no installation package there...
Sorry about my stupid question, I am a total newbie in AcrobatSDK.
P.S. My ultimate goal is to develop the program in VB6 (not using VB2005),
as my client PCs does not (and cannot for some reason) install .net framework.

RM30: batch job to send surveys?

Hi, Does anyone know off-hand the name of the batch job that has to be scheduled to make Risk Management 3.0 send surveys? Thanks.

Hello Alex,
you have to perform the Automatic Workflow customizing.
Highlight "Maintain Runtime Environment" and press the red-white ball in left upper corner.
The name of the job itself is SWWDHEX.
Best,
Jürgen

Batch job in a report program

Hi Abappers,
I have a peculiar scenario.
My report has 2 PERFORMS lets say
PERFORM1:
PERFORM2:
I have a DATE criterion in PERFORM 1.
Based on that date only I have to execute my PERFORM2.
Ex:   The Date consideration is Suspension of a Contract Date: which is 01/12/2010.(dd/mm/yyyy)
Now based on the this date I have to execute the 2nd PERFORM.
I mean I have to create a Sales Order on this date only.
To summarize,Execution of a program in a future date whose date is dynamic in nature
Can i achieve this using a BATCH JOB.
Please tell me how to schedule a BATCH JOB for a future date.
Can i call a BATCH JOB froma REPORT.
Please your inputs shall be of great help to me in solving my problem.
Thanks and Regards
Srikanth.P

Hi Sakshi,
In that case of using the FM
g_date is sy-datum.
g_time is sy-uzeit.
CALL FUNCTION 'JOB_OPEN'
EXPORTING
*   DELANFREP              = ' '
*   JOBGROUP               = ' '
    JOBNAME                = 'Z_BATCHJOB_TRGT'
    SDLSTRTDT              = g_date
   SDLSTRTTM               = g_time.
*   JOBCLASS               =
* IMPORTING
*   JOBCOUNT               =
* CHANGING
*   RET                    =
* EXCEPTIONS
*   CANT_CREATE_JOB        = 1
*   INVALID_JOB_DATA       = 2
*   JOBNAME_MISSING        = 3
*   OTHERS                 = 4
IF SY-SUBRC <> 0.
* MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
*         WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
ENDIF.
In the above case how would the Job name get mapped to the Report. / Or / How can we assign the Job Name to the Report.
There is no scope of giving the Report name in the FM.
Can you please elucidate.(EXP parameters)
By the way Thanks for your quick response.
Thanks and Regards
Srikanth

Recorded Actions and Batch jobs

Photoshop has an Actions panel feature that allows you to record and save a set of commands (eg: resize an image to 400 x 600, then turn it black & white). Anytime you want to perform the same set of commands, just select and run the saved Action. In addition, Photoshop has a Batch jobs feature that allows you to select a directory of files and run the saved action on all files in the directory. In this way, you could resize a thousand images to 400 x 600 and turn them all black & white with one click.
Does Captivate have any similar capability? I have about 3,000 swf files and I wish to set the preloader percentage on all of them to the same value. I would hate to open, edit, save, and close the files manually one at a time. If this feature does not exist, can you guys think of an efficient way to accomplish the task?
I have Captivate 5 but can upgrade if necessary.
Thanks for the help

Since you mention that you have both CPT and CPTX files it might be worth mentioning that your goal of just opening them, chaning the preloader percentage before save and close again might not work out as planned.
Have you considered the following issues?
Are you intending to open the CPT files in the same version of Captivate that created them, or were you thinking of upgrading them to a later version such as Cp5.5? If so, you'll need to do more than just change the preloader percentage. You'll also need to change the path to the preloader itself since each version of Captivate is installed in a different directory in Program Files.
Were all of the CPT projects set up as AS3 with all AS3 components, or were some of them AS2 and therefore would need to be reviewed and tested to make sure upgrading to a Cp5x version did not have other impacts? Projects that were set up as AS2 would need to change preloaders, skins, and any animations over to AS3 before the upgrade would be successful.
CPT to CPTX files will also experience font size issues since the font rendering technology changed in Cp5x. So you'll probably need to go through those projects and fix a lot of issues in text captions.
And another one to watch out for: After upgrading projects from CPT to CPTX the Continue button on the Quiz Results will often cease to work unless you remove and replace the Quiz Results slide.
This is not a complete list of potential upgrade issues. What I'm trying to sound here is a warning that, unless you use exactly the same version of Captivate that created them, making any changes to these files could turn into a much bigger job than you hope. There could be weeks of work involved trawling through each project file one by one checking for issues.

Siebel EDQ batch job performance

Similar Messages

Maybe you are looking for