Looking for best design approach for moving data from one db to another.

We have a very simple requirement to keep 2 tables synched up that live in 2 different databases. There can be up to 20K rows of data we need to synch up (nightly).
The current design:
BPEL process queries Source DB, puts results into memory and inserts into Target DB. Out of memory exception occurs. (no surprise).
I am proposing a design change to get the data in 1000 row chunks, something like this:
1. Get next 1000 records from Source DB. (managed through query)
2. Put into memory (OR save to file).
3. Read from memory (OR from a file).
4. Save into Target DB.
Question is:
1 Is this a good approach and if so, does SOA have any built in mechanisms to handle this? I would think so since I believe this is a common problem - we don't want to reinvent the wheel.
2. Is it better to put records into memory or writing to a file before inserting into the Target DB?
The implementation team told me this would have to be done with Java code, but I would think this would be out of the box functionality. Is that correct?
I am a SOA newby, so please let me know if there is a better approach.
Thank you very much for your valued input.
wildeman

Hi,
After going through your question, the first thing that came to my mind is what would be the size of the 20K records.
If this is going to be huge then even the 1000 row logic might take significant time to do the transfer. And I think even writing it to a file will not be efficient enough.
If the size is not huge then probably your solution might work. But I think you will need to decide on the chunk size based on how well your BPEL process will work. Possible you can try different size and test the performance to arrive at an optimal value.
But in case the size is going to be huge, then you might want to consider using ETL implementations. Oracle ODI does provide such features out of the box with high performance.
On the other hand, implementing the logic using the DBAdapter should be more efficient than java code.
Hope this helps. Please do share your thoughts/suggestions.
Thanks,
Patrick

Similar Messages

  • What are Parameters? How are they differenet from Variables? Why can't we use variables for passing data from one sequnece to another? What is the advantage of using Parameters instead of Variables?

    Hi All,
    I am new to TestStand. Still in the process of learning it.
    What are Parameters? How are they differenet from Variables? Why can't we use variables for passing data from one sequnece to another? What is the advantage of using Parameters instead of Variables?
    Thanks in advance,
    LaVIEWan
    Solved!
    Go to Solution.

    Hi,
    Using the Parameters is the correct method to pass data into and out of a sub sequence. You assign your data to be passed into or out of a Sequence when you are in the Edit Sequence Call dialog and in the Sequence Parameter list.
    Regards
    Ray Farmer

  • Best Way to port the data from one DB to another DB using Biztalk

    Hi,
    please suggest best way to move the data from one db to another DB using biztalk.
    Currently I am doing like that, for each transaction(getting from different source tables) through receive port, and do some mapping (some custom logic for data mapping), then insert to target normalized tables(multiple tables) and back to update the status
    of transaction in source table in sourceDB. It is processing one by one.
    How/best we we can do it using  bulk transfer and update the status. Since it has more than 10000 transaction per call.
    Thanks,
    Vinoth

    Hi Vinoth,
    For SQL Bulk inserts you can always use SQL Bulk Load
    adapter.
    http://www.biztalkgurus.com/biztalk_server/biztalk_blogs/b/biztalksyn/archive/2005/10/23/processing-a-large-flat-file-message-with-biztalk-and-the-sqlbulkinsert-adapter.aspx
    However, even though a SQL Bulk Load adapter can efficiently insert a large amount of data into SQL you are still stuck with the issues of transmitting the
    MessageBox database and the memory issues of dealing with really large messages.
    I would personally suggest you to use SSIS, as you have mentioned that records have to be processed in specific time of day as opposed to when the
    records are available.
    Please refer to this link to get more information about SSIS: http://msdn.microsoft.com/en-us/library/ms141026.aspx
    If you have any more questions related to SSIS, please ask it in
    SSIS 
    forum and you will get specific support.
    Rachit

  • BADI for transfering data from one modal to another modal within single appset

    Hallo Experts,
    My Business Requirement is Transfer of data from one modal to another in same environment and did this taking reference from below document.(How to custom badi for replicating destination app)
    http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/e04b5d24-085f-2c10-d5a2-c1153a9f9346?QuickLink=index&…
    Document contains TR for implementing BADI, but this document supports bpc version 7.0 and we are using is bpc 10.0.
    So i make all compatible changes in BADI implementation and activate it. Now i am testing using transaction UJKT using following script.
    and the result is records successfully written back to application MIS, but when i check data is not moved to target application MIS.
    I am facing stuck situation in my project work. Please Suggest. Hope for positive reply.
    Script:
    *XDIM_MEMBERSET WM_ACCOUNT = WM_041,
    *XDIM_MEMBERSET WM_UOM_02 = UOM_004
    *XDIM_MEMBERSET WM_UD_2 = WM_07
    *XDIM_MEMBERSET WD_EXT_MAT_GRP =CHALK-PH-I,CHALK-PH-II,CHAVN-PH-I,CHAVN-PH-II,NASHIK-WM,RAJASTHAN-WM,TAMILNADU-WM
    *XDIM_MEMBERSET CATEGORY= Plan
    *XDIM_MEMBERSET AUDITTRAIL=Input
    *XDIM_MEMBERSET P_ENTITY = SIL
    *XDIM_MEMBERSET RPTCURRENCY = LC
    *START_BADI DAPP
       DESTINATION_APP ="MIS"
       RENAME_DIM ="WD_EXT_MAT_GRP= PRODUCT"
       ADD_DIM ="PLANT=NO_PLANT","MIS_ACCOUNTS=CAIN0058040008","COST_CENTER=NO_COST_CENTER","FLOW=Opening","UOM=AMT","CUSTOMER_SALES_2=NO_CUSTOMER","CATEGORY=Plan","AUDITTRAIL=Input"             
       DEBUG = ON
       WRITE = OFF
       QUERY = ON
    *END_BADI
    Please find attached result.
    Regards,
    Dipesh Mudras.

    Hello,
    Here is the manual to copy data between apps (it works with BPC NW75):
    http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/b0480970-894f-2d10-f9a5-d4b1160be203?quicklink=index&overridelayout=true
    It works me, but now I need to modify the Script Logic to make that a "property" from the origin dimension has to be copied to the destination dimension as "id", like follow:
    //*XDIM_MEMBERSET CATEGORY = %CATEGORY_SET%
    //*XDIM_MEMBERSET TIME = %TIME_SET%
    //*XDIM_MEMBERSET ENTITY = %ENTITY_SET%
    //*XDIM_MEMBERSET RPTCURRENCY =  %RPTCURRENCY_SET%
    *START_BADI FiltroPD
         WRITE = OFF
         APPL = $APPLICATION$
         ADD_DIM = "ORIGEN = APPVENTAS"
         ADD_DIM ="O_COSTE = no_input"
         ADD_DIM="CECO = no_input"
         RENAME_DIM="P_ACCT = RATIOS.P_ACCT "
    *END_BADI

  • Moving Data from one table to another table

    Hi,
    I've a requirement to move large volumes (few billions) of data from one table to another empty target table. The target has more number of attributes and may not have same attribute name as the source. Please let me know what's the best method using oracle to move data in this case?
    Thanks in Advance,
    Tom

    You should use nologging and parallelism options
    Please refer to Tom's answers:
    http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:164612348068
    http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:6407993912330
    Kamran Agayev A. (10g OCP)
    http://kamranagayev.wordpress.com
    [Step by Step install Oracle on Linux and Automate the installation using Shell Script |http://kamranagayev.wordpress.com/2009/05/01/step-by-step-installing-oracle-database-10g-release-2-on-linux-centos-and-automate-the-installation-using-linux-shell-script/]

  • How to copy documents for InfoProvider data from one cube to another cube

    Hello Everyone,
    We are using standard document feature to store document/comment about the data shown in the report/planning query. Relevant characteristics have the property 'characteristics is document attrib' turned on/selected for this purpose.
    Looks like the documents created gets tagged to a particular InfoProvider along with characteristic value assignment.
    Is there a way to move or copy documents from one InfoProvider to another InfoProvider.
    As per the design requirement, we need to move data from a real time InfoProvider (transaction cube) to a basic reporting cube. The data in the first/source cube stays for a period of approx 3 months before moved to other cube. We want the documents associated with InfoProvider data also to be moved, so that users can continue to see the document/comments what they  had created earlier.  
    I looked at the Documents tab in DataWarehouse workbench. No options seem to be available to copy or move documents.
    Any help will be highly appreciated and points will be rewarded accordingly!
    Best regards,
    Sanjeeb

    Hi
    Have you got any answer to this? I would also be interested in a copy function but more from one query ID to another.
    Kind regards,
    Daniel Müller

  • Moving data from one table to another table in a diferent Database..

    I need to move the data of one table from one database to another table in a different database, is there a way to do this. Do I need the systemn password? or the password of the schema that own the table is enough? the tables are name different, same columns but different name. I know I can do it via sqlldr(creating a file...then runing the ctl file..etc), but I am wonder if it is a different way of doing it?
    Thank you

    Your question is more suitable for the Database-General forum @ General Database Discussions
    (You'll find more DBA's over there, although some of them take a glance over here as well every now and then ;) )
    Do not forget to mention the versions of both databases...

  • Best method for transferring data from one database to another?

    There is an 8i database I have to deal with on a regular basis. Besides being completely outdated and unsupported, in my opinion the database has not been well-maintained--for a table with 25 million records, should ‘select count(*) from table;’ take 5 minutes to run? I don’t really know, but that seems long to me. Many complex queries (most including only tables with less than a million records) take a ridiculously long time to run, to the point that I can’t even run some of them.
    I am not the DBA; I don't have the authority to fiddle with the database (nor would I feel comfortable doing that), and the powers that be will not put effort into improving functionality of this database due to an alleged plan to update/replace it within the next year. However, in the mean time, I still have to get data out of this database on a regular basis.
    I have XE 10g installed on my local machine, and I have set up a database link in it to the 8i database. I have found that I can pull in basic data (simple queries) from the 8i database into tables in my XE database (e.g. create table tbl1 as select data from tbl1@8idb) and then query those tables to get the information I need much, much faster (including creation of the tables). While this option does not allow me to create queries/reports that other people can run, it makes work I’m doing only for myself much faster.
    What I’m wondering is, what is the best way to bring the information I need over to my database? I usually don’t need entire tables, and I can probably identify a number of key tables (or parts of tables) I need. What I’ve been doing up until now is writing CREATE TABLE statements on the fly, but then I end up forgetting what all I’ve done, and each time I want up-to-date data, I have to drop the tables and re-create them. It seems to me that there should be an easier way to do this than to copy and paste from a text document into SQL*Plus.
    Does anyone have any suggestions for me on how best to do this?

    Sorry, I guess I posted this in the wrong forum. I re-posted in the database-general forum.

  • Moving data from one table to another

    Hi,
    I havent done CF in a log time and i have a question,
    My client has an admin site for his inventory in a db (MS SQL) one of the table (inventory) has 180 fields (which are options for the items) with around 100 records that are being added/deleted per week, he now whants to keep the deleted data has reference to what happened to them.
    What i want to do is move all the data berofe deleting the record of Inventory to inventory_deleted without having to specify all the 180 fields name to be moved :-)   yep lazy
    I already copied the table without the data all i need is some sort of query to move invetory_deleted before I delete it from inventory
    I'v tryed adding a active/deleted field to the table, but this is a huge site and it causes to many problems everywhere else when showing data, stats ...etc
    here is the delete code :-)
    <cfquery datasource="#clientds#">
          DELETE FROM inventory
          WHERE inv_number=#form.inv_num#
        </cfquery>
    and this is what i tested so far
    <cfquery datasource="#clientds">
    insert into inventory_deleted (*)
    select *
    from inventory
    WHERE inv_number=#form.inv_num#
    </cfquery>
    Help would be really apreciated

    Not i get what you mean by nothing but this is what i did
    <cfquery datasource="autoavantage" name="insertdata">
    insert into inventory_deleted (*) select * from inventory where inv_num = #inv_num#
    </cfquery>
    and this is the error i got
    Error Executing Database Query.
    [Macromedia][SequeLink JDBC Driver][ODBC Socket][Microsoft][ODBC Microsoft Access Driver] Syntax error in INSERT INTO statement.

  • Moving Itunes From one computer to another

    HI,
    Can anyone offer advice for moving Itunes from one laptop to another? The original laptop that has my itunes on it, has broke, it froze and will not operate. Rather that lose all my purchased music and playlists i would like to swap it to another laptop. Can anyone advise if this will be possible and how to do it? Both laptops have windows XP installed
    thanks
    Arlene

    Hi
    I am having a similar problem- when I record from Itunes to a cd it only works in "Data cd" format in the advanced tab. That's okay, but when I re-insert the cd and try to import it, it doesn't show anything- it plays in Itunes okay. There aren't any little boxes to check for selecting any songs and if I try to use the import function, it shows as blank.(file type shows as .txt) I am using Itunes 6- would Itunes 7 help this?
    thanks- I have a lot of music to transfer and this is a problem.
    Max

  • Which CKM is used for moving data from Oracle to delimited file ?

    Hi All
    Please let me know Which CKM is used for moving data from Oracle to delimited file ?
    Also is there need of defining each columns before hand in target datastore. Cant ODI take it from the oracle table itself ?

    Addy,
    A CKM is a Check KM which is used to validate data and log errors. It is not going to assist you in data movement. You will need an LKM SQL to File append as answered in another thread.
    Assuming that you have a one to one mapping, to make things simpler you can duplicate the Oracle based model and create a file based model. This will take all the column definitions from the Oracle based model.
    Alternatively, you can also use an ODI tool odiSQLUnload to dump the data to a file
    HTH

  • Moving time-dependant data from one table to another (archiving)

    Hello all
    I would like to know if there's an easier solution or a "best practice" to move data from one table to another. The context of this issue can be found within "archiving".
    More concretely: we have an application that uses several tables to log information to.
    These tables are growing like crazy, and we would like to keep only "relevant" data in those tables, so I was thinking about moving data from these tables that have been in there for, say 2 months, to "archiving" tables.
    I figured there must be some kind of "best practice" to get this done.
    I have already written a procedure that loops the table that has the time indicator and inserts the records from the normal tables into the archive tables (and afterwards delete this data), but it seems to be taking ages to get it done.
    Thanks in advance!
    Message was edited by:
    timschraepen

    There is nothing to do with PL/SQL.
    You can refer below links:
    http://www.lc.leidenuniv.nl/awcourse/oracle/server.920/a96524/c12parti.htm
    http://www.stanford.edu/dept/itss/docs/oracle/10g/server.101/b10739/partiti.htm#i1006727

  • What is the best way to move data from one array to another

    I'm going to be moving data from one array to a larger array on the same RAID but different controller. (I have some extra extra drives I'm also going to be installing Retrospect so I can't just restore from a backup.)
    The RAID has 450GB of production files, fonts etc.
    What is the best way to move the data over?
    I saw that someone had suggested using ditto. Would that be better than MacMV?
    I also own Bru LE so I could use that.
    Any advice would be appreciated.
    Thanks,
    Paul

    Ditto is a great option -- probably the best.
    ditto -rsrc src_folder /Volumes/targetvolume/targetfolder

  • Can anyone give me step by step for calling FM from one SAP to another SAP

    can anyone give me step by step for calling FM from one SAP to another SAP
    points will be rewarded,
    thank you,
    Regards,
    Jagrut BharatKumar Shukla

    *& Report  RFC_FOR_CUSTOMER_LIST
    *& RFC to a Fn. module from another system which implements BAPI
    REPORT  rfc_for_customer_list.
    DATA:
      fl_status TYPE i.
    DATA:
      fs_message TYPE bapiret2.
    DATA:
      BEGIN OF fs_customers,
        id   TYPE s_customer,
        name TYPE s_custname,
      END OF fs_customers.
    DATA:
      t_customers LIKE
         STANDARD TABLE
               OF fs_customers.
    CALL FUNCTION 'Z_BAPI_GET_CUSTOMER_LIST' DESTINATION 'R3N'
      IMPORTING
        return = fs_message
      TABLES
        customerlist = t_customers.
    IF sy-subrc EQ 0.
      fl_status = 1.
      LOOP AT t_customers INTO fs_customers.
        WRITE:
          /10 fs_customers-id,
           30 fs_customers-name.
        AT LAST.
          WRITE:
            /,/5 'No of customers in R3N = ', sy-tabix.
        ENDAT.
      ENDLOOP.
    ELSE.
      MESSAGE 'RFC failed' TYPE 'S'.
      EXIT.
    ENDIF.
                           TOP-OF-PAGE EVENT                            *
    TOP-OF-PAGE.
      IF fl_status NE 0.
        WRITE:
          /12 'ID'   COLOR 6,
           32 'NAME' COLOR 6.
        SKIP.
      ENDIF.
    This is one small example...
    Regards,
    Pavan

  • SCAT program for copying BOM from one material to another material??

    Hi Gurus,
    Tried creating SCAT program for copying BOM from one material to another material and founf that the copying function does not work.
    Please advise??
    Thank you.
    Naveen.

    HI Satish Ji,
    We are using version 6.0.
    We were able to create SCAT for creating new BOM but, for copying we are facing an issue.
    Kindly advice.
    Thank you.
    Naveen.

Maybe you are looking for