Best approach to transfer and transform data between two Oracle Instances

Let me first preface this post with the fact that I am an Oracle newb.
The current problem I am trying to solve is how to quickly transfer data from a customers legacy database into a new normalized destination that is modeled differently than the legacy data. So, the data not only has to be transferred, but also transformed (into normalized tables)
We have discovered, the hard way, that reading the data using a C++ application and performing inserts (even with indexing and constraints disabled) is just way too slow for what we need. We are dealing with around 20 million records here. I need to determine what the best approach extracting this data out of the source and inserting it into the destination. Any comments or tips are greatly appreciated.
Note: I have read about SQL*Loader and mentioned it to management, but they seem resistant to this approach. It's not totally out of the question though.

Oracle has a lot of technologies that fall under the general heading of "replication" to choose from. Going from a database to a flat file and back to another database is generally not the most efficient solution-- it's generally much easier to go straight from one database to another.
Oracle provides materialized views, Streams, and Change Data Capture (CDC)-- any of these would be able to give your new system incremental feeds of data from the old system's data and could be used by whatever routines you write to transform the data between data models.
Justin
Distributed Database Consulting, Inc.
http://www.ddbcinc.com/askDDBC

Similar Messages

  • Intervals, synchronize the data between two oracle tables located in diffe

    Hi Experts,
    Is there any way I can, at set intervals, synchronize the data between two oracle tables located in different servers? What is the software, if there is any? If there are steps I can follow, so much the better.
    Thanks!

    I'm not sure what you want to achieve here...
    I want to synchronize both the Servers , so that it can update HO servers as well as Factory Server with the complete data. Could you please elaborate a little bit more; basing on your description I could also tell you how to setup timesync for those two servers if I understand "complete data" as the current date and time...
    Also a lot of solutions are platform and version specific so it would help if you could indicate OS and Database Versions (4 digit).
    And I still fail to see what this has to do with forms. You might want to post your question to the Database General forum: General Database Discussions
    cheers

  • Best way to transfer single point data between loops on FPGA

    Hi,
        I used quite number of loops on FPGA and need to transfer single point datas between loops. Only the current value of data is needed so buffer is not necessary. I don't want use target scope FIFO since it require minimum 21 elements and I only need the current value. Is there other way other than local variables?
        Thanks for help!
        Regards,
       Tom

    Hi Godel,
    Since this discussion thread is over 3 years old, it would be better to start a new thread with your specific questions to get quicker help.
    I did a bit of research on your question, and found this helpful White Paper (http://www.ni.com/white-paper/7727/en/) that discusses the resources used for components in LV FPGA code. For local variables (although it depends on hardware), it looks like it uses Flip-flops and LUTs.
    As for using local variables with different clock rates, I found this KnowledgeBase article that might help shed some light on your question - some issues can arise from using them with more than one clock rate (http://digital.ni.com/public.nsf/allkb/C683585460E88508862570D1006B7434)
    Hope this helps! Again, if you have follow-up questions, I would definitely recommend creating a new thread
    Xavier
    Applications Engineering Specialist
    National Instruments

  • Best approach to synchronize data between two oracle database remotely

    Hi All,
    We have requirement to synchronize data between custom application database and EBS database. Thw two database is on different location connected by 128 kbps connection.
    The synchronization will do the following task :
    - Download customer table from EBS (I have checked there are only 2000 rows of it)
    - Up load billing data to EBS auto invoice interface table
    The question is :
    What approach is the best way to do this synchronization task ? some that come into my mind :
    - create database link and run stored procedure to do the batch update ?
    - for Customer synchronization, is it a good idea to create Materialized view so that any update to EBS customer data will be updated, refreshed nightly
    - or is ther any better approach ?
    Thank you very much,
    xtanto
    Edited by: xtanto on Oct 18, 2008 3:30 PM

    Maybe try Streams?

  • How can i swap the content such as messages, calendar entries, apps and app data between two iphone 4s?

    I have a Iphone 4s and my friend has one too and we want to switch phones but do not want to lose any data or message or calendar entries, we want the same exact content avaliable in my phone swapped to his and his to mine. how can this be done?

    Back up your phones onto your respective computers with iTunes, swap phones, Reset them and when you connect it to iTunes, instead of "Set up as a new iPhone", click on restore from back up

  • Definition of the best approach on how to do reporting between BPC and BW

    Hi,
    I need your opinion in the definition of the best approach on how to do reporting between BPC and BW.
    For example if we want to do reporting using BW on Actuals Vs Budget how should we manage this since technically BPC Model and BW InfoCube is different?
    BPC Models have the Budget and BW has the actuals, but the InfoObject that is used for Account is different. What is the best approach to do the reporting?
    Thanks in advance,
    JA

    Hi Gersh
    I already thought in that option, but the problem is the Yellow requests in the Infocube that are not used by VP.
    In the past I used Report RSAPO_CLOSE_TRANS_REQUEST_ALL3 in the virtual function module to close the requests, but now I didn't want to use VP based on function module.
    Is there any option to use data in Yellow requests in VP based on DTP?
    Best regards,
    JA

  • Transfer large data between two databases via ODI

    Hi, we want to transfer our 2 billion records (70gb) A database oracle server to B database oracle server via ODI LKM SQL to Oracle (SQLLDR).We use a filter like "prod_enter_date>sysdate-1" for daily  records(about 200.000).It takes about 1 our.How can we transfer data faster, thanks  for your advice.

    OK so assuming the Oracle DB's can see each on the network other have a word with your DBA's about creating a DBLink and test out the LKM DB Link KM.
    Also try Uli's amended LKM if dont want to be too intrusive on your souce system creating synonyms etc. : http://sonra.io/tag/lkm-db-links/
    I have used both and they work fine.
    Basically you are currently unloading a large amount of data, transfering to another location and then uploading that data with no transformation to the target environment, a DBLink LKM will transfer this data without the need to unload / upload , just the network transfer.
    Another thread here : https://forums.oracle.com/message/9630782
    To re-quote the that thread quoting the documentaion for that LKM : "As it says in the component description - "This module is recommended when developing interfaces between two Oracle servers."

  • Best way to store and retrieve data in a text file

    I want to write to a text file and then in another Applescript read the same data into an array
    I'll have 5 lines of text data
    What would be the best format?
    CSV?
    XML maybe?
    Just a plain file with each line of data separated by pressing enter?
    Thanks
    Omar

    I agree with Frank - for just 5 lines, simple text is probably sufficient.
    However, if the point is just to pass data between two AppleScripts you could even use native AppleScript objects - for example, you could write the list (array) to the file and then read it back in directly, e.g:
    Write the data:
    -- create a list of any types:
    set myList to {"a string", 3.14159, "foo", {name:"Joe", age:21}}
    -- create a file
    set f to open for access (choose file name) with write permission
    -- write your (binary) data to it
    write myList to f
    -- and close the file
    close access f
    Now, separately, you can read it back:
    set x to read (choose file) as list

  • Best way to transfer internal HD data in Mavericks to Mac with Yosemite ?

    Hello,
    I'm using a 2007 iMac with Mavericks and will be getting a new one which will presumably come with Yosemite installed. What's the best way to transfer all the data from my internal HD on the old system, to the new one ?
    I use SuperDuper to make backups to external HD's, so if I make a bootable copy of the mac HD to an ext HD using SuperDuper, will everything function fine despite the different OS's ?
    Thanks,
    Matrose.

    You can make a bootable copy of your system now, but you won't be able to boot from it with the new computer. They are not usually backwards compatible with the OS. But when you first boot into the new system, use SetUp Assistant to migrate the data from the cloned copy. That will work just fine.

  • Best approach to exporting and importing an environment

    Client is on 11.1.0.7. Windows 2008 R1. We have a need to export an environment that is using ASM and import that environment to another server that is not using ASM. Environment name will stay the same. Was hoping to get some advice on best approach for this and what would be involved. Thanks in advance.

    ...a need to export an environment...What do you mean by that? Are you referring to moving the data only or physically migrating the database? How big is the data/database size?
    If you only need to move data, export/import is transparent from ASM environment. If you need to migrate physical database from ASM to non-ASM in the same platform, you can use RMAN utility.

  • Use of SAP memory to transfer data between two different sessions.

    Hello experts,
    I wish to know how to use SAP memory to transfer data between two different sessions.
    The scenario is that when I run a report and change a variable, the value of changed variable should be availabe to another user on another terminal.
    Thanks & Regards!
    Sumit

    Hello,
    Just to add what Max has already mentioned. IMPORT TO / EXPORT FROM DATABASE statements can be used to store data in special "cluster" tables (you can't use any DDIC table) e.g., INDX.
    @OP: You can opt for Shared Memory(SHM) for this specific requirement as well. In my opinion SHM is a bit tricky to code, but it is easier to monitor. The opposite holds true for "data clusters".
    You should remember SHM is app-server specific. So if you've a load balancing scenario, using SHM can cause problems.
    Hope i'm clear.
    BR,
    Suhas
    Edited by: Suhas Saha on Nov 19, 2010 4:12 PM

  • Best practice for Plan and actual data

    Hello, what is the best practice for Plan and actual data?  should they both be in the same app or different?
    Thanks.

    Hi Zack,
    It will be easier for you to maintain the data in a single application. Every application needs to have the category dimension, mandatorily. So, you can use this dimension to maintain the actual and plan data.
    Hope this helps.

  • Can we cleanse and transform data at flat file or external table level?

    Hi,
    I have some data that I want to cleanse and transform. I don't want to cleanse it after i populate the external table, I want to get done with it at flat file level or while populating the external table. Can we cleanse and transform data at flat file or external table level through Oracle or OWB 11.2? Is it possible to run a conditional load (i.e. having a where clause or if-else-then) for an external table? Can we call oracle functions for an external table at the time of creation?
    Thanks in advance.
    Regards,
    Ann.

    Hi Oleg,
    Thanks a lot for the clarification. :)
    So is there a way that I can clease the data within the text file through Oracle or OWB? I have datatype mismatches in the data and most of my data is getting rejected because of that. The way I can think of, for solving this problem, is to create the external table with all fields with datatype varchar and then cleansing the data. But it doesn't seem very effecient plus it will get very complicated because I have almost 80-90 fields.
    Any help?
    Thanks and regards,
    Ann.

  • HCM errors in phase Identify Objects for Transfer and Start Data Selection.

    Hello Colleagues ,
    Background: We are using TDMS HCM PA PD Expert package to transfer all of the HCM data
    We are using TDMS 4.0 SP 5. We are trying to do the Configuraition in Identify Objects for Transfer and Start Data Selection.. In this phase, we are encountering the issue of 'choose to confirm to trigger data selection'.  The only option in this activit is'confirm batch' and 'confirm online'. Even if we press the button, we are still encountering issues. What selections then or values can we use?Thank you. Kindly see screen shot
    Regards,
    Meinard

    Hello Meinard,
    There could be two issues:
    First check if you have assigned the target ranges for the dialog user which you are using to select data from sender system. You need to maintain target ranges in activity 'Define target ranges for users' which you can find under activity "Configuration and Utilities in Control system".
    Secondly, there could be authorizaton issues for the sender system user. Check for any authorization issue in SU53 in sender system for the user which you used to logon to sender system.
    Thanks
    Anita

  • How can I share data between two forms on different lists

    Using a custom content type, I created two lists that I want to share the same data - one is a calendar.  Our employees complete a form from the "Out of Office Request" list that has workflow functionality that sends an email to that person's
    manager.  If the manager approves the request, the item automatically populates the "Out of Office Calendar."  The problem is that the only information from the request list that populates the calendar is the Title field and date/time fields. 
    I need the manager name in order to create a view for each manager.  
    How can I connect the other information in the request list to the calendar list.  It seems to me that if the title and date fields carry over the information, there should be a way to connect the other information.  I'm using Designer.
    I've tried to connect the two lists' webparts with the wizard, but when I get to the page that maps the two lists, there are no column names and the "Next >" button is grayed out.  This seems like the logical place to connect the two lists,
    but it isn't working.

    Hi,
    According to your post, my understanding is that you wanted share data between two forms on different lists.
    To show external  information on the calendar event, there are two methods: Calculated column, workflow. You can refer to:
    A Simple Guide to Show More Information on a Calendar Event
    I recommend to use workflow to achieve what you want. But you need to create a people column to display the manager.
    You can create a workflow associated to the "Out of Office Request" list, add action to Start Approve Process. If the manager approves the request, you can create a item in the calendar, and then update the people column and the title column.
    Then the calendar will display the Title, date/time and the manager.
    To create a view for each manager, you need to modify the Filter. You can use the people column is equal to the manager name or the Title contains the manager name.
    Thank you for your understanding.
    Best Regards,
    Linda Li
    Linda Li
    TechNet Community Support

Maybe you are looking for