How to transfer data between different tables in an Oracle 10g databse?

Question

I have to do the following: there are 5 database tables in our Oracle 10g database that store certain data. Then there are 2 new tables where we want the data to go to. My task is to somehow transfer all the data that is in the first 5 tables and insert it in the 2 new ones. All tables are in the same database. The challenge lies in the fact that the structures of the tables are very dissimilar so it won't be a matter of a simple INSERT SQL query.
The data access tier of our application is developed in JAVA. We use EJB. So one way I am thinking of doing the above mentioned data transfer is to simply code it in the JAVA backed. But that might be to slow, there are millions of records in those first 5 tables. What is the best way to do something like that? Perhaps a custom (Perl?) script or some such?
Thanks.

Answer 1

The problem is that the way the data is stored in the old tables you cannot write a query that would return it in such a format that you could immediately insert it in the new table without some formatting. Let me illustrate with an example. To pull the data from the old tables I use this query:
SELECT
     SXML_DOCUMENT_DATA.VALUE As DocumentValue,
SXML_ELEMENT.NAME As ElementName
FROM
     SXML_DOCUMENT,
     SXML_DOCUMENT_DATA,
     SXML_DOCUMENT_DETAIL,
     SXML_ELEMENT,
     SXML_TYPE
WHERE
     SXML_TYPE.XML_TYPE_KEY = SXML_DOCUMENT.XML_TYPE_KEY
AND     SXML_DOCUMENT.XML_DOCUMENT_KEY = SXML_DOCUMENT_DETAIL.XML_DOCUMENT_KEY
AND     SXML_DOCUMENT_DETAIL.XML_DOCUMENT_DETAIL_KEY = SXML_DOCUMENT_DATA.XML_DOCUMENT_DETAIL_KEY
AND     SXML_ELEMENT.XML_ELEMENT_KEY = SXML_DOCUMENT_DATA.XML_ELEMENT_KEY
AND     SXML_TYPE.NAME = 'DA_UNIT_COMMITMENT'
AND     (SXML_ELEMENT.NAME = 'resource'
     OR SXML_ELEMENT.NAME = 'resourceType'
     OR SXML_ELEMENT.NAME = 'commitmentType'
     OR SXML_ELEMENT.NAME = 'startTime'
     OR SXML_ELEMENT.NAME = 'endTime'
     OR SXML_ELEMENT.NAME = 'schedulingCoordinator')
ORDER BY
     SXML_DOCUMENT.NAME,
     SXML_DOCUMENT_DETAIL.XML_DOCUMENT_DETAIL_KEY,
     SXML_DOCUMENT_DATA.XML_DOCUMENT_DATA_KEY,
     SXML_ELEMENT.NAME;
The results from the SQL query above look like this:
DOCUMENTVALUE | ELEMENTNAME
1 | ALAMIT_7_UNIT_1 | resource
2 | GEN | resourceType
3 | BRS8 | schedulingCoordinator
4 | IFM | commitmentType
5 | 2008-07-29T18:00:00:00 | startTime
6 | 2008-07-30T00:00:00 | endTime
7 | ALAMIT_7_UNIT_1 | resource
8 | GEN | resourceType
9 | BRS8 | schedulingCoordinator
10 | IFM | commitmentType
11 | 2008-07-29T00:00:00 | startTime
12 | 2008-07-29T04:00:00 | endTime
and so on. The type of data repeats every 6 records. And the values of each 6 records corresponds to 1 row in the new table, which looks like this:
schedulingCoordinator | resource | resourceType | commitmentType | startDate | endDate
1| 1 | 27 | GEN | IFM | 2008-07-29T18:00:00:00 | 2008-07-30T00:00:00
2| 1 | 27 | GEN | | 2008-07-29T00:00:00 | 2008-07-29T04:00:00
So hopefully now you see the challenge. It is not as simple as writing a SQL query that returns result rows corresponding 1 to 1 to a row in the new table. Somehow I need to take the first 6 result rows from that big SQL query and put them in the first row of the new table. Then the next 6 and put them in the second row and so on. And also, we are talking about millions of records. What happens if I process the first 2 million and then some error occurs? Do I start from the beginning again? One idea I have is to process the data in chunks, say process 500 results, commit, process another 500, commit, and so on.
Edited by: 799984 on Oct 11, 2010 9:08 AM