Transporting large amounts of data from one database schema to another

Hi,
We need to move large amount of data from one 10.2.0.4 database schema to another 11.2.0.3 database.
Am currently using datapump but quite slow still - having to do in chunks.
Also the datapump files quite large so having to compress and move across the network.
Is there a better/quicker way?
Habe haerd about transportable tablespaces but never used them and don't know about speed - if quicker thana datapump.
tablespace names different in both databases.
Also source database on solaris opertaing system on sun box
target database on aix on ibm power series box.
Any ideas would be great.
Thanks
Edited by: user5716448 on 08-Sep-2012 03:30
Edited by: user5716448 on 08-Sep-2012 03:31

user5716448 wrote:
Hi,
We need to move large amount of data from one 10.2.0.4 database schema to another 11.2.0.3 database.
Pl quantify "large".
Am currently using datapump but quite slow still - having to do in chunks.
Pl quantify "quite slow".
Also the datapump files quite large so having to compress and move across the network.
Again, pl quantify "quite large".
Is there a better/quicker way?
Habe haerd about transportable tablespaces but never used them and don't know about speed - if quicker thana datapump.
tablespace names different in both databases.
Also source database on solaris opertaing system on sun box
target database on aix on ibm power series box.
It may be possible, assuming you do not violate any of these conditions
http://docs.oracle.com/cd/E11882_01/server.112/e25494/tspaces013.htm#ADMIN11396
Any ideas would be great.
Thanks
Edited by: user5716448 on 08-Sep-2012 03:30
Edited by: user5716448 on 08-Sep-2012 03:31Master Note for Transportable Tablespaces (TTS) -- Common Questions and Issues [ID 1166564.1]
HTH
Srini

Similar Messages

Copying large amount of data from one table to another getting slower

I have a process that copies data from one table (big_tbl) into a very big archive table (vb_archive_tbl - 30 mil recs - partitioned table). If there are less than 1 million records in the big_tbl the copy to the vb_archive_table is fast (-10 min), but more importantly - it's consistant. However, if the number of records is greater than 1 million records in the big_tbl copying the data into the vb_archive_tbl is very slow (+30 min - 4 hours), and very inconsistant. Every few days the time it takes to copy the same amount of data grows signicantly.
Here's an example of the code I'm using, which uses BULK COLLECT and FORALL INSERST to copy the data.
I occasionally change 'LIMIT 5000' to see performance differences.
DECLARE
TYPE t_rec_type IS RECORD (fact_id NUMBER(12,0),
store_id VARCHAR2(10),
product_id VARCHAR2(20));
TYPE CFF_TYPE IS TABLE OF t_rec_type
INDEX BY BINARY_INTEGER;
T_CFF CFF_TYPE;
CURSOR c_cff IS SELECT *
FROM big_tbl;
BEGIN
OPEN c_cff;
LOOP
FETCH c_cff BULK COLLECT INTO T_CFF LIMIT 5000;
FORALL i IN T_CFF.first..T_CFF.last
INSERT INTO vb_archive_tbl
VALUES T_CFF(i);
COMMIT;
EXIT WHEN c_cff%NOTFOUND;
END LOOP;
CLOSE c_cff;
END;
Thanks you very much for any advice
Edited by: reid on Sep 11, 2008 5:23 PM

Assuming that there is nothing else in the code that forces you to use PL/SQL for processing, I'll second Tubby's comment that this would be better done in SQL. Depending on the logic and partitioning approach for the archive table, you may be better off doing a direct-path load into a staging table and then doing a partition exchange to load the staging table into the partitioned table. Ideally, you could just move big_tbl into the vb_archive_tbl with a single partition exchange operation.
That said, if there is a need for PL/SQL, have you traced the session to see what is causing the slowness? Is the query plan different? If the number of rows in the table is really a trigger, I would tend to suspect that the number of rows is causing the optimizer to choose a different plan (with your sample code, the plan is obvious, but perhaps you omitted some where clauses to simplify things down) which may be rather poor.
Justin

Copying large amount of data from one table to another

We have a requirement to copy data from a staging area (consists of several tables) to the actual tables. The copy operation needs to
1) delete the old values from the actual tables
2) copy data from staging area to actual tables
3) commit only after all rows in the staging area are copied successfully. Otherwise, it should rollback.
Is it possible to complete all these steps in one transaction without causing problems with the rollback segments, etc.? What are the things that I need to consider in order to make sure that we will not run into production problems?
Also, what other best practices/alternative methods are available to accomplish what is described above.
Thanks,
Eser

It's certainly possible to do this in a single transaction. In fact, that would be the best practice.
Of course, the larger your transactions are, the more rollback you need. You need to allocate sufficient rollback (undo in 9i) to handle the transaction size you're expecting.
Justin
Distributed Database Consulting, Inc.
www.ddbcinc.com

How to Copy complete structure of a table with data from one database table to another databse table

I need a sql query to copy structure of table with data from production table of ONLINEBTREKDB database to production table of Archive database.
I tried this query
select * into Archive.dbo.Production from ONLINEBTREKDB.dbo.Production p
but problem is I am able to copy the table schema and data but not able to copy constraints(PK)
Any Help?
seema

You've multiple options
1. Use generate scripts wizard available in SQL management studio. This is particularly helpful when you want to script out lot of objects. You can also choose to script data as well inside this. This can be launched by right clicking the db, choosing tasks
-> generate scripts and then selecting required options inside the wizard
2. Use object explorer and right click and script out table. You can also use search functionality to find object you want inside object explorer
http://visakhm.blogspot.in/2013/02/object-filtering-using-ssms-object.html
3. Use query based on INFORMATION_SCHEMA views like TABLES,COLUMNS,CONSTRAINT_COLUMN_USAGE etc to generate the script
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Pass data from one web application to another web application

Hi,
Please provide suggestion for following scenario:
Scenario: Basically the aim is to transfer large amount of data from one application to another and display JSP of second application.
User enteres data on a JSP of one web application. When he submits that data, another web application opens in new window and data from first application should be passed to this second web application. Another web application will display that data on its own JSP. User can perform whatever he wants on second application screen.
Possible solutions:
1) response.sendRedirect(): This makes GET request. But, there is large data to send. So, GET is not effective.
2) forward(request, response): Can't use as I have to pass data to different application that is in another context.
3) URLConnection: Here, I can make POST request and set attributes in HTTP request and make connection to another application. I can pass data using output stream. But, I can't display second application JSP to user even if I use input stream to read response. Because control will be ultimately in first application only.
Is there any othe method to achieve this or any of the above options extendable?
Please give your inputs.

Hi,
According to your post, my understanding is that you want to migrate list data from one web application to another.
We can migrate list data programmatically, there are some articles for your reference.
http://blogs.msdn.com/b/tejasr/archive/2007/11/12/code-snippet-copy-list-data-between-sites-programmatically.aspx
http://www.fivenumber.com/copy-sharepoint-list-items-from-one-site-to-another-programmatically/
http://geekswithblogs.net/AnneBougie/archive/2009/01/23/copy-a-sharepoint-list.aspx
Thanks & Regards,
Jason
Jason Guo
TechNet Community Support

Power BI performance issue when load large amount of data from database

I need to load data set from my database, which have large amount of data, it will take so many time to initialize data before I can build report, is there any good way to process large amount of data for PowerBI? As I know many people analysis data based
on PowerBI, is there any suggestion for loading large amount of data from database?
Thanks a lot for help

Hi Ruixue,
We have made significant performance improvements to Data Load in the February update for the Power BI Designer:
http://blogs.msdn.com/b/powerbi/archive/2015/02/19/6-new-updates-for-the-power-bi-preview-february-2015.aspx
Would you be able to try again and let us know if it's still slow? With the latest improvements, it should take between half and one third of the time that it used to.
Thanks,
M.

How to move only subset of data from one database to another?

Both source & destination are Oracle11g databases.
The requirement details are as below,
1) The database contains static as well as transactional data for telecom domain.
2) We have to move region-wise data from one database to another.
3) There are around 10 regions.
4) The region wise data extraction from source db is based on around 40 to 50 tables. Rest of the tables contains
static data & it will not change.
5) The volume is around 1 million subscribers per region.
6) The migration is required because
a) The client is upgrading its base product which uses this database
b) There is a change in structure of static tables
c) Hardware upgrade
d) The client want to start with single region on new database & rest of the regions will be operated from old
database.
7) Keep execution time to very minimum.
I am thinking to have solution as below,
1) Create destination database with upgraded db structure (as mentioned in 6b)
2) Create database links to access source db in destination db.
3) Write SQL queries to fetch data from all the respective tables for a specific region
4) Write separate PL/SQL blocks for each table to fetch data from source db & insert into respective table in
destination db
a) Use FOR ALL & bulk collect to improve the performance
b) Divide table data into multiple chunks & execute parallel batches (around 10) to insert the data
5) Validate pre & post counts to verify the success of migration
Is there any other better way?
Regards,
Sandeep

How to move only subset of data from a partiular table by using transportable tablespace?
E.g. SUB table using SMP_SUB tablespace contains subscriber data in source database.
The indexes defined on SUB table are under SMP_IDX_SUB tablespace
The subscribers data can be categorized by different regions, say region_id column. Then how to move data & indexes of SUB table from source db to destination db?
any specific example would be helpful.

Different ways to Transfer data from one database to another database

Hi all,
What are the ways to transfer data from one database to another database. With the following options, i Can transfer data as far as i know. Please
correct me if i am wrong or tell me if there is any other options are available.
1) Create database link/connection string and using this string and COPY command, we can transfer data.
2) By using Export and Import utilities.
I told first one to my interviewer, he told, its strange, by using, COPY command also can we transfer data ? As far as i know, we can transfer data. Am i right ?
Thanks in advance,
Pal

transfer data from one database to another database.You mean store the data of one to another?
1) Create database link/connection string and using this string and COPY command, we can transfer data.every SELECT on a DB-link is transfering data. And you can have all kind of transfers and store on the e.g CTAS of materialized views or.... the SQL*PLUS COPY :
The COPY command is not being enhanced to handle datatypes or features introduced with, or after Oracle8i. The COPY command is likely to be made obsolete in a future release.
But there are many others. Check for ORACLE Streams, and the "COPY" your interviewer was mentioning is about the operating file system COPY right? That's transportable tablespaces.
http://download.oracle.com/docs/cd/E11882_01/server.112/e17120/tspaces013.htm#ADMIN11403
-- andy

Transfer data from one database to another without identities but keep the relation b/w PK and Foreign key

Hi,
I need to transfer data from one database to another database (both are identical databases).
1. Not transferring identity columns (primary keys). the destination table might have the same key.
2. keep the PK's and FK's relation b/w parent and child table
3. I have 4 levels
Example: tableA (col1 int identity(1,1) , col2, col3)
tableB (col1 int identity(1,1) ,
col2 , col3) -- col2 has the foreign key relation with tableA.col1
tableC (col1 int identity(1,1) ,
col2, col3) -- col2 has the foreign key relation with tableB.col1
tableD (col1 int identity(1,1) , col2, col3) -- col2 has the foreign key relation with tableC.col1
please advise me.
Thanks in advance

Try the below:
/********************************SAMPLE TARGET***************************************************************/
Use MSDNSamples
create table TableA(LevelValueId int identity(1,1) primary key, name varchar(100))
Insert into TableA(name) Select 'R1'
Insert into TableA(name) Select 'R2'
create Table TableB(ChildId int identity(100,1),name varchar(100), LevelValueID int references TableA(LevelValueId))
Insert into TableB(name,LevelValueID) Select 'Childname1',1
/********************************SAMPLE TARGET***************************************************************/
/********************************SAMPLE SOURCE***************************************************************/
Use Sample
create table TableA(LevelValueId int identity(1,1) primary key, name varchar(100))
Insert into TableA(name) Select 'C1'
Insert into TableA(name) Select 'C2'
create Table TableB(ChildId int identity(100,1),name varchar(100), LevelValueID int references TableA(LevelValueId))
Insert into TableB(name,LevelValueID) Select 'Kidname1',1
/********************************SAMPLE SOURCE***************************************************************/
USe MSDNSamples
/********************************MIGRATION INTERMEDIATE TABLE***************************************************************/
--Migration table
Create table Mg_TableA(LevelValueId int, NewValueId int)
/********************************MIGRATION INTERMEDIATE TABLE***************************************************************/
/********************************ACTUAL MIGRATION FOR MASTER TABLE***************************************************************/
MERGE INTO TableA
USING sample.dbo.TableA AS tv
ON 1 = 0
WHEN NOT MATCHED THEN
INSERT(name) Values(tv.name)
Output tv.levelValueId ,inserted.LevelValueid INTO
Mg_TableA;
/********************************ACTUAL MIGRATION FOR MASTER TABLE***************************************************************/
/********************************ACTUAL MIGRATION FOR CHILD TABLE***************************************************************/
Insert into TableB (name,LevelValueID)
Select A.name,B.NewValueId From sample.dbo.TableB A
Inner join Mg_TableA B on A.LevelValueID = B.LevelValueId
/********************************ACTUAL MIGRATION FOR CHILD TABLE***************************************************************/
/********************************TEST THE VALUES***************************************************************/
Select * From TableA
Select * From Mg_TableA
Select * From TableB
/********************************TEST THE VALUES***************************************************************/
Drop table TableB,Tablea,Mg_TableA
Use Sample
Drop Table TableB,Tablea

Couldn't copy large amount of data from enterprise DB to Oracle 10g

Hi,
I am using i-batis to copy data from enterprise DB to oracle and viceversa.
The datatype of a field on EDB is 'text' and the datatype on oracle is 'SYS.XMLTYPE'
i am binding these to a java string property in a POJO to bind values.
I could successfully copy limited amount of data from EDB to oracle but if there is more data, i am getting the following exceptions with different oracle drivers ( but i could read large amount of data from EDB):
--- Cause: java.sql.SQLException: ORA-01461: can bind a LONG value only for insert into a LONG column
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeUpdate(MappedStatement.java:107)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.update(SqlMapExecutorDelegate.java:457)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.update(SqlMapSessionImpl.java:90)
at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.update(SqlMapClientImpl.java:66)
at com.aqa.pojos.OstBtlData.updateOracleFromEdbBtlWebservice(OstBtlData.java:282)
at com.aqa.pojos.OstBtlData.searchEdbAndUpdateOracleBtlWebservice(OstBtlData.java:258)
com.ibatis.common.jdbc.exception.NestedSQLException:
--- The error occurred in com/aqa/sqlmaps/SQLMaps_OSTBTL_Oracle.xml.
--- The error occurred while applying a parameter map.
--- Check the updateOracleFromEDB-InlineParameterMap.
--- Check the parameter mapping for the 'btlxml' property.
--- Cause: java.sql.SQLException: setString can only process strings of less than 32766 chararacters
at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeUpdate(MappedStatement.java:107)
at com.iba
I have latest oracle 10g jdbc drivers.
remember, i could copy any amount of data from oracle to EDB but not otherway around.
PLease let me know if you have come across this issue, any recommendation is very much appreciated.
Thanks,
CK.

Hi,
I finally remembered how I solved this issue previously.
The jdbc driver isn't able to directly call the insert with a column xml_type. The solution I was using was to build a wrapper function in plSQL.
Here it is (for insert but I suppose tha update will be the same)
create or replace procedure insertXML(file_no_in in number, program_no_in in varchar2, ost_XML_in in clob, btl_XML_in in clob) is
begin
insert into AQAOST_FILES (file_no,program_no,ost_xml,btl_xml) values(file_no_in, program_no_in, xmltype(ost_XML_in), xmltype(btl_XML_in));
end insertXML;
here is the sqlmap file I used
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE sqlMap
PUBLIC "-//ibatis.apache.org//DTD SQL Map 2.0//EN"
"http://ibatis.apache.org/dtd/sql-map-2.dtd">
<sqlMap>
     <typeAlias alias="AqAost" type="com.sg2net.jdbc.AqAost" />
     <insert id="insert" parameterClass="AqAost">
          begin
               insertxml(#fileNo#,#programNo#,#ostXML:CLOB#,#bltXML:CLOB#);
          end;
     </insert>
</sqlMap>
an here is a simple program
package com.sg2net.jdbc;
import java.io.IOException;
import java.io.Reader;
import java.io.StringWriter;
import java.sql.Connection;
import oracle.jdbc.pool.OracleDataSource;
import com.ibatis.common.resources.Resources;
import com.ibatis.sqlmap.client.SqlMapClient;
import com.ibatis.sqlmap.client.SqlMapClientBuilder;
public class TestInsertXMLType {
     * @param args
     public static void main(String[] args) throws Exception {
          // TODO Auto-generated method stub
          String resource="sql-map-config-xmlt.xml";
          Reader reader= Resources.getResourceAsReader(resource);
          SqlMapClient sqlMap = SqlMapClientBuilder.buildSqlMapClient(reader);
          OracleDataSource dataSource= new OracleDataSource();
          dataSource.setUser("test");
          dataSource.setPassword("test");
          dataSource.setURL("jdbc:oracle:thin:@localhost:1521:orcl");
          Connection connection=dataSource.getConnection();
          sqlMap.setUserConnection(connection);
          AqAost aqAost= new AqAost();
          aqAost.setFileNo(3);
          aqAost.setProgramNo("prg");
          Reader ostXMLReader= Resources.getResourceAsReader("ostXML.xml");
          Reader bltXMLReader= Resources.getResourceAsReader("bstXML.xml");
          aqAost.setOstXML(readerToString(ostXMLReader));
          aqAost.setBltXML(readerToString(bltXMLReader));
          sqlMap.insert("insert", aqAost);
          connection.commit();
     public static String readerToString(Reader reader) {
          StringWriter writer = new StringWriter();
          char[] buffer = new char[2048];
          int charsRead = 0;
          try {
               while ((charsRead = reader.read(buffer)) > 0) {
                    writer.write(buffer, 0, charsRead);
          } catch (IOException ioe) {
               throw new RuntimeException("error while converting reader to String", ioe);
          return writer.toString();
package com.sg2net.jdbc;
public class AqAost {
     private long fileNo;
     private String programNo;
     private String ostXML;
     private String bltXML;
     public long getFileNo() {
          return fileNo;
     public void setFileNo(long fileNo) {
          this.fileNo = fileNo;
     public String getProgramNo() {
          return programNo;
     public void setProgramNo(String programNo) {
          this.programNo = programNo;
     public String getOstXML() {
          return ostXML;
     public void setOstXML(String ostXML) {
          this.ostXML = ostXML;
     public String getBltXML() {
          return bltXML;
     public void setBltXML(String bltXML) {
          this.bltXML = bltXML;
I tested the insert and it works correctly
ciao,
Giovanni

Transfer of Data from one Database to Another

Respected Guys
Is there any way to transfer few fields from different tables located on a Database Server used by SAP to an other database used by an another application. The other database also have same fields and and Tables structure.
Both databases are located on different servers and are oracle9i.
Is there any way to transfer real time data from one database to another. If not then any way by which we can preiodically can transfer the data.
Thnx

Hi
U can use a <a href="http://help.sap.com/saphelp_nw04s/helpdata/en/9c/d736b880c34f76b507bac7751a0474/frameset.htm">DBCON (Database Connection) tool</a>.
Here you should call another database and write SQL native code in order to transfer/read the data.
Max

How to copy the data from one database to another database without DB link

Good Day,
I want to copy the data from one database to another database with out DB link but the structure is same in both.
Thanks
Nihar

You could use SQL*Plus' COPY command
http://technology.amis.nl/blog/432/little-gold-nugget-sqlplus-copy-command
http://download.oracle.com/docs/cd/B19306_01/server.102/b14357/apb.htm#sthref3702
Edited by: Alex Nuijten on Sep 16, 2009 8:13 AM

Table Data From One DataBase to another

Dear Fellows,
There are two DataBases running in my NT Server. I want to pull data of a table from other database. Does any body know how can I do it if I dont want to export and then import.
Thanks in advance
Danish

Please find the help for SQL*PLUS command COPY.
It will be useful when you want to copy datas from one database from another database without using import & export.

Move all data from one database to other using oracle expdp and impdp tool

I need to move all of the data from one database to another. For that I am using oracle expdp and impdp tool.
I have the full database export dump file. I moved that dump file to my other database.
Now when I try to use impdp tools it gives lots of error regarding path mismatch,user not exists, role not exists.
As my datafile path of source database does not match with the target database path , also the users are not there in target database. I dont want to create all the tablespaces that in my source database as there are 82 tablespaces.Is there any way by which I can move my all data without using remap option for tablespaces or users and grants.

The FULL parameter indicates that a complete database export is required. The following is an example of the full database export and import syntax.
The user must have the privielge of EXP_FULL_DATABASE
expdp system/password@db10g full=Y directory=TEST_DIR dumpfile=DB10G.dmp logfile=expdpDB10G.log
impdp system/password@db10g full=Y directory=TEST_DIR dumpfile=DB10G.dmp logfile=impdpDB10G.logGO through below linkfor more details...
http://www.oracle-base.com/articles/10g/OracleDataPump10g.php
Regards
Umi
Edited by: Umi on Feb 4, 2011 2:27 AM

How do i transfer data from one internal tabe to another.

Hi All,
         How do i transfer data from one internal tabe to another.
         Can i do it ebven if he tables are different in structure.
Please Advice.
Thanks in advance.

Hi Saket Tiwari,
I hope the earlier post by kashyap is good enough an answer. anywas in addition to it let me give a detailed
explanation of how you can populate an internal table.
1) Append data line by line.
     Syntax : APPEND [<wa> TO / INITIAL LINE TO] <itab>.
this appends new line to internal table <itab>.
2) Using COLLECT statement.
             COLLECT is another form of statement used for populating the internal tables. Generally COLLECT is used while inserting lines into an internal table with unique standard key. The syntax for COLLECT statement is as shown
     Syntax : COLLECT [<wa> INTO] <itab>.
3) Using INSERT statement
     Syntax INSERT [<wa> INTO / INITIAL LINE INTO] <itab> [index <idx>].
INSERT statement adds a line/work area to the internal table. You can specify the position at which the new line is to be added by using the INDEX clause with the INSERT statement.
Now coming to your request..
To append part or all of an internal table
     Syntax
              APPEND LINES OF <itab1> [FROM <n1>] [TO <n2>] TO <itab2>.
*     Note:
Without the FROM and TO options, this statement appends the entire table <itab1> to <itab2>.*
b) To insert part or all of an internal table into another internal table
     Syntax
          INSERT LINES OF <itab1> [FROM <n1>] [TO <n2>]
          INTO <itab2> [INDEX <idx>].
c) Using Move statement.
To copy entire contents of one table into another in one execution
     Syntax MOVE <itab1> To <itab2>.
               OR
          <itab1> = <itab2>.
but u hav to be careful because he contents of itab2 will eb overwritten on the execution of this statement.
These copy the contents of ITAB1 to ITAB2. Incase of internal tables with header line we have to use [] inorder to distinguish from work area. So, to copy contents of internal tables with header line the syntax becomes,
ITAB1[] = ITAB2[].
Coming to the letter part of your question, Yes, we can copy values between tables having different structures.
for this we use
                            MOVE-CORRESPONDING <itab1> TO <itab2>
    this executes the statement for their header lines. Searches for the sub-fields which occur both in itab1 and itab2 and then generates, for all relevant field pairs which correspond to the
        sub-fields ni , statements of the form MOVE itab1-ni TO itab2-ni. The other fields remain unchanged.
I hope the information provided has been of your help.
Reward if useful.
Regards,
Jose

Transporting large amounts of data from one database schema to another

Similar Messages

Maybe you are looking for