Loading from multiple flat files to same table using SQL Loader

Hi Gurus,
Can anyone please brief me the pros and cons of kicking of multiple sql loader sessions that reads multiple flat files but inserting it into just one table.
The table is not partitioned. Avg record counts for each flat file is about 5-6 million.
Oracle 11g,
OS: Linux
Regards
Cherrish Vaidiyan

Vaidiyan wrote:
Hi Gurus,
Can anyone please brief me the pros and cons of kicking of multiple sql loader sessions that reads multiple flat files but inserting it into just one table.Cherrish,
Pros -> Faster loading of more data
Cons -> Potential performance degradation
Test to see how much resource consuming this task would be and do a priority comparison of that multi-multi load task with other stuff that will be happening in the database in the multi-multi load time so you could decide how to share resources in that time.

Similar Messages

Load XML File into temporary tables using sql loader

Hi All,
I have an XML file as below. I need to insert the contents into a temporary staging table using sql loader. Please advice how I need to do that.
For example Portfolios should go into a seperate table, and all the tags inside it should be populated in the columns of the table.
Family should go into a seperate table and all the tags inside it should be populated in the columns of the table.
Similarly offer, Products etc.
- <ABSProductCatalog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
- <ProductSalesHierachy>
- <Portfolios>
- <Portfolio productCode="P1">
<Attribute name="CatalogProductName" value="Access" />
<Attribute name="Status" value="Active" />
</Portfolio>
- <Portfolio productCode="P2">
<Attribute name="CatalogProductName" value="Data" />
<Attribute name="Status" value="Active" />
</Portfolio>
- <Portfolio productCode="P3">
<Attribute name="CatalogProductName" value="Voice" />
<Attribute name="Status" value="Active" />
</Portfolio>
- <Portfolio productCode="P4">
<Attribute name="CatalogProductName" value="Wireless" />
<Attribute name="Status" value="Active" />
</Portfolio>
</Portfolios>
- <Families>
- <Family productCode="F1">
<Attribute name="CatalogProductName" value="Internet Access Services" />
<Attribute name="Status" value="Active" />
- <ParentHierarchy>
<Item productCode="P1" modelType="Portfolio" />
</ParentHierarchy>
</Family>
- <Family productCode="F2">
<Attribute name="CatalogProductName" value="Local Access Services" />
<Attribute name="Status" value="Active" />
- <ParentHierarchy>
<Item productCode="P2" modelType="Portfolio" />
</ParentHierarchy>
</Family>
</Families>
- <SubFamilies>
- <SubFamily productCode="SF1">
<Attribute name="CatalogProductName" value="Business Internet service" />
<Attribute name="Status" value="Active" />
- <ParentHierarchy>
<Item productCode="F1" modelType="Family" />
</ParentHierarchy>
</SubFamily>
</SubFamilies>
- <ProductRefs>
- <ProductRef productCode="WSP1" modelType="Wireline Sales Product">
<ActiveFlag>Y</ActiveFlag>
- <ProductHierarchy>
<SalesHierarchy family="F1" subFamily="SF1" portfolio="P1" primary="Y" />
<SalesHierarchy family="F2" portfolio="P2" primary="N" />
<FinancialHierarchy quotaBucket="Voice" strategicProdCategory="Local Voice" />
</ProductHierarchy>
</ProductRef>
- <ProductRef productCode="MSP2" modelType="Handset">
<ActiveFlag>Y</ActiveFlag>
- <ProductHierarchy>
<SalesHierarchy portfolio="P4" primary="Y" />
</ProductHierarchy>
</ProductRef>
</ProductRefs>
</ProductSalesHierachy>
- <Offers>
- <Offer productCode="ABN">
<OfferName>ABN</OfferName>
<OfferDescription>ABN Description</OfferDescription>
- <Segments>
<Segment>SCG</Segment>
<Segment>PCG</Segment>
</Segments>
<OfferUpdateDate>2009-11-20</OfferUpdateDate>
<ActiveFlag>Y</ActiveFlag>
</Offer>
- <Offer productCode="OneNet">
<OfferName>OneNet</OfferName>
<OfferDescription>OneNet Description</OfferDescription>
- <Segments>
<Segment>SCG</Segment>
<Segment>PCG</Segment>
<Segment>PCG2</Segment>
</Segments>
<OfferUpdateDate>2009-11-20</OfferUpdateDate>
<ActiveFlag>Y</ActiveFlag>
</Offer>
</Offers>
- <Products>
- <Product productCode="WSP1" modelType="Wireline Sales Product">
<ProductName>AT&T High Speed Internet</ProductName>
<ProductDescription>High Speed Internet</ProductDescription>
<LegacyCoProdIndicator>SBC</LegacyCoProdIndicator>
<RevenueCBLCode>1234B</RevenueCBLCode>
<VolumeCBLCode>4567A</VolumeCBLCode>
<SAARTServiceIDCode>S1234</SAARTServiceIDCode>
<MarginPercentRequired>Y</MarginPercentRequired>
<PercentIntl>%234</PercentIntl>
<UOM>Each</UOM>
<PriceType>OneTime</PriceType>
<ProductStatus>Active</ProductStatus>
<Compensable>Y</Compensable>
<Jurisdiction>Everywhere</Jurisdiction>
<ActiveFlag>Y</ActiveFlag>
- <Availabilities>
<Availability>SE</Availability>
<Availability>E</Availability>
</Availabilities>
- <Segments>
<Segment>SCG</Segment>
<Segment>PCG</Segment>
</Segments>
<VDIndicator>Voice</VDIndicator>
<PSOCCode>PSOC 1</PSOCCode>
<USBilled>Y</USBilled>
<MOWBilled>N</MOWBilled>
<ProductStartDate>2009-11-20</ProductStartDate>
<ProductUpdateDate>2009-11-20</ProductUpdateDate>
<ProductEndDate>2010-11-20</ProductEndDate>
- <AliasNames>
<AliasName>AT&T HSI</AliasName>
<AliasName>AT&T Fast Internet</AliasName>
</AliasNames>
- <OfferTypes>
<OfferType productCode="ABN" endDate="2009-11-20" />
<OfferType productCode="OneNet" />
</OfferTypes>
- <DynamicAttributes>
- <DynamicAttribute dataType="String" defaultValue="2.5 Mbps" name="Speed">
<AttrValue>1.5 Mbps</AttrValue>
<AttrValue>2.5 Mbps</AttrValue>
<AttrValue>3.5 Mbps</AttrValue>
</DynamicAttribute>
- <DynamicAttribute dataType="String" name="TransportType">
<AttrValue>T1</AttrValue>
</DynamicAttribute>
</DynamicAttributes>
</Product>
- <Product productCode="MSP2" modelType="Handset">
<ProductName>Blackberry Bold</ProductName>
<ProductDescription>Blackberry Bold Phone</ProductDescription>
<LegacyCoProdIndicator />
<RevenueCBLCode />
<VolumeCBLCode />
<SAARTServiceIDCode />
<MarginPercentRequired />
<PercentIntl />
<UOM>Each</UOM>
<PriceType />
<ProductStatus>Active</ProductStatus>
<Compensable />
<Jurisdiction />
<ActiveFlag>Y</ActiveFlag>
- <Availabilities>
<Availability />
</Availabilities>
- <Segments>
<Segment>SCG</Segment>
<Segment>PCG</Segment>
</Segments>
<VDIndicator>Voice</VDIndicator>
<PSOCCode />
<USBilled />
<MOWBilled />
<ProductStartDate>2009-11-20</ProductStartDate>
<ProductUpdateDate>2009-11-20</ProductUpdateDate>
- <AliasNames>
<AliasName />
</AliasNames>
- <OfferTypes>
<OfferType productCode="ABN" />
</OfferTypes>
- <DynamicAttributes>
- <DynamicAttribute dataType="String" name="StlmntContractType">
<AttrValue />
</DynamicAttribute>
- <DynamicAttribute dataType="String" name="BMG 2 year price">
<AttrValue>20</AttrValue>
</DynamicAttribute>
- <DynamicAttribute dataType="String" name="MSRP">
<AttrValue>40</AttrValue>
</DynamicAttribute>
- <DynamicAttribute dataType="String" name="BMGAvailableType">
<AttrValue />
</DynamicAttribute>
- <DynamicAttribute dataType="String" name="ProductId">
<AttrValue>123456</AttrValue>
</DynamicAttribute>
- <DynamicAttribute dataType="String" name="modelSource">
<AttrValue>product</AttrValue>
</DynamicAttribute>
</DynamicAttributes>
</Product>
</Products>
<CatalogChanged>Y</CatalogChanged>
</ABSProductCatalog>

Two options that come to mind. Others exist.
#1 - {thread:id=474031}, which is basically storing the XML in an Object Relational structure for parsing
#2 - Dump the XML into either an XMLType based table or column and use SQL (with XMLTable) to create a view that parses the data. This would be the same as the view shown in the above post.
Don't use sql*loader to parse the XML. I was trying to find a post from mdrake about that but couldn't. In short, sql*loader was not build as an XML parser so don't try to use it that way.

How to Load Multiple Files in Oracle Database using Sql Loader

Hi All,
I want to import multiple files in my DB using Sql*Loader. Please tell me the Syntax, how can I import multiple files using one Control File?
Thanks & Regards,
Imran

Hi,
You might get a good response to your post in the forum dedicated to data movement , including SQL*Loader . You can find it here Export/Import/SQL Loader & External Tables
Regards,

Loading the data from a text file to a table using pl/sql

Hi Experts,
I want to load the data from a text (sample1.txt) file to a table using pl/sql
I have used the below pl/sql code
declare
f utl_file.file_type;
s varchar2(200);
c number := 0;
begin
f := utl_file.fopen('TRY','sample1.txt','R');
loop
utl_file.get_line(f,s);
insert into sampletable (a,b,c) values (s,s,s);
c := c + 1;
end loop;
exception
when NO_DATA_FOUND then
utl_file.fclose(f);
dbms_output.put_line('No. of rows inserted : ' || c);
end;
and my sample1.txt file looks like
1
2
3
The data is getting inserted, with below manner
select * from sampletable;
A     B     C
1     1     1
2     2     2
3     3     3
I want the data to get inserted as
A     B     C
1     2     3
The text file that I have is having three lines, and each line's first value should go to each column
Please help...
Thanks

declare
f utl_file.file_type;
s1 varchar2(200);
s2 varchar2(200);
s3 varchar2(200);
c number := 0;
begin
f := utl_file.fopen('TRY','sample1.txt','R');
utl_file.get_line(f,s1);
utl_file.get_line(f,s2);
utl_file.get_line(f,s3);
insert into sampletable (a,b,c) values (s1,s2,s3);
c := c + 1;
utl_file.fclose(f);
exception
when NO_DATA_FOUND then
if utl_file.is_open(f) then utl_file.fclose(f); ens if;
dbms_output.put_line('No. of rows inserted : ' || c);
end;SY.

Loading from a Flat file: Binary and Text File

Hi,
Does anyone know what the difference is between loading from a binary file or text file?
Ramon

Hi,
the difference is that text files contain lines (or records) of text and each of these has an end-of-line marker automatically appended to the end of it whenever you indicate that you have reached the end of a line.
So what happens when we read from a text file is that the end-of-line character for the operating system that we are using gets converted into a low value end-of-string indicator and when we write to a file the appropriate end-of-line character(s) get written when we indicate the end of the line. This makes the reading and writing of text files much easier because the appropriate end-of-line markers are handled for us.
With a binary file none of these conversions take place. When we read a binary file the end-of-line characters for our operating system will be read into the string and treated no different than any other character. When we write to a binary file the only end-of-line markers that are written will be those that we code into the output ourselves and hence will be exactly as we code it regardless of the operating system that we are running on. This makes it much easier for us when the file does not contain straight text and the end-of-line marker does not separate lines of text but rather appears as part of the non-text data content of the file.

Extract Data from XML and Load into table using SQL*Loader

Hi All,
We have a XML file (sample.xml) which contains credit card transaction information. We have a standard SQL*Loader control file which loads the data from a flat file and the control file code is written as position based method. Our requirement is to use this control file as per our requirement(i.e) load the data into the table from our XML file), But we need help in converting the XML to a flat file or Extract the data from the XML tags and pass the information to the control file and in turn it loads the table.
Your suggestion is highly appreciated.
Thanks in advance

Hi,
First of all go to PSA maintanance ( Where you will see PSA records ).
Goto list---> Save-> File---> Spreadsheet (Choose Radio Button)
> Give the proper file name where you want to download and then-----> Generate.
You will get ur PSA data in Excel Format.
Thanks
Mayank

Map FLAT file to oracle table using 9.04 version - PLS HELP!!!!

Hello all
I am having a problem with mapping a flat file to oracle table. The validation is successful, when I go to Project/Deployment manager. Try to deploy the mapping itself and the target table. It said succesful, and the last step is another "Deploy", this one is fail. Saying could not locate the file (which is a flat file) , but it is there on the server.
I have read all the help on line and follow what they show me, but still not work
Any ideas? Please provide detail answer if you know it.
Thank in advance

Hallo,
just give a rights on connector
Variant 1
1. connect to user sys
2. grant read,write on directory <connector_name> to <target_schema>;
or
Variant 2
1. as user sys or system give CREATE_ANY_DIRECTORY to <target_schema>
2. manualy make CREATE DIRECTORY <connector_name> as '<full_path_to_directory>';
and enjoy :)
PS: <connector_name> you can take from script CREATE_TABLE wisch in Generation phase was created!
Kirill

How can i store a picture file in a table using sql

can anyone help me
to store a pic file in a table using sql

You can find an example in this link
http://www.orafaq.com/forum/t/38347/0/

How to extract data from multiple flat files to load into corresponding tables in SQL Server 2008 R2 ?

Hi,
          I have to implement the following scenario in SSIS but don't know how to do since I never worked with SSIS before. Please help me.
          I have 20 different text files in a single folder and 20 different tables corresponding to each text file in SQL Server 2008 R2 Database. I need to extract the data from each text file and
load the data into corresponding table in Sql Server Database. Please guide me in how many ways I can do this and which is the best way to implement this job. Actually I have to automate this job. Few files are in same format(with same column names
and datatypes) where others are not.
1. Do I need to create 20 different projects ?
               or
    Can I implement this in only one project by having 20 packages?
             or
    Can I do this in one project with only one package?
Thanks in advance.

As I said I don't know how to use object data type, I just given a shot as below. I know the following code has errors can you please correct it for me.
Public
Sub Main()
' Add your code here
Dim f1
As FileStream
Dim s1
As StreamReader
Dim date1
As
Object
Dim rline
As
String
Dim Filelist(1)
As
String
Dim FileName
As
String
Dim i
As
Integer
i = 1
date1 =
Filelist(0) =
"XYZ"
Filelist(1) =
"123"
For
Each FileName
In Filelist
f1 = File.OpenRead(FileName)
s1 = File.OpenText(FileName)
rline = s1.ReadLine
While
Not rline
Is
Nothing
If Left(rline, 4) =
"DATE"
Then
date1 (i)= Mid(rline, 7, 8)
i = i + 1
Exit
While
End
If
rline = s1.ReadLine
End
While
Next
Dts.Variables(
"date").Value = date1(1)
Dts.Variables(
"date1").Value = date1(2)
Dts.TaskResult = ScriptResults.Success
End
Sub

Loading data from two flat files into one table

Hi All,
I want to load few fields from one file and few fields from another file and load the data into one table in the database.
For Ex: I have two flat files file1.csv and file2.csv and file1.csv containes 25 columns and file2.csv containes 12 columns, i want to load 20 columns from file1.csv and 7 columns from file2.csv into the table
how to combine the columns from both the files and load them into one table at a time?
Any help is appriciated
Thanks
R.G

Use external tables.
If using Oracle9 or higher connect both your csv with database as separate external tables and load the requited fields in new/requited table.

Problem to Load the data Flat file to Oracle table on Linux Machine

hi,
when i change the path of source flat file from windows to Linux machine. Now it gives the error. Basicalyy this file will load the files names into Oracle table. but in windows path its fine. All things r running fine in Linux. when i run the interface it gives this>>>>>>>>
SQL*Loader: Release 11.2.0.1.0 - Production on Wed Dec 19 16:40:16 2012
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
SQL*Loader-704: Internal error: ulconnect: OCIServerAttach [0]
ORA-12162: TNS:net service name is incorrectly specified
Thanks
Regards,
Edited by: AMSI on Dec 19, 2012 6:01 AM

I assume you're using the "LKM file to Oracle SQL Loader".
For test purpose, can you just try with "LKM file to SQL" (slower) ?
We could check if the problem comes from the SQL Loader.
Your unix server has access to the oracle database ?

Loading multiple .csv files into a table using SSIS

Hi,
I have a requirement where I have 200+ csv files to be loaded into Netezza table using SSIS.
Issue I am facing is all columns have different number of columns, for ex, file 1 has columns A,B,C and file 2 has columns C,D,E. My target table has all columns from A to E in it.
But, when I am using for each loop container, only the file for which I have specified filepath+filename in loop variable, that is getting loaded. Rest all files, no data is getting loaded from them and package is executing successfully.
Any help is appreciated.
Regards,
VT

if you want to iterate through files then all files should be in same folder and you should use file enumerator type within ForEach loop. Then inside loop you might need a script task to generate the data flow on the fly based on the avialble input columns
from the file and do the mapping in the destination. So I assume you put NULLs (or some default value) for missing columns from the file
http://blog.quasarinc.com/ssis/best-solution-to-load-dynamically-change-csv-file-in-ssis-etl-package/
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Sqlldr flat file to external table using shell scripts

Hi,
Has anyone done this before? Please give me a hand.
Thanks!

Thanks Justin.
When do I need to create the external table EMP_STAGING ?
These are my steps so far:
- shell script to crate the flat file (but I need to change the table name to EMP_STAGING)
- use a script to call sqlldr to load the flat file into the external table
- then the script will call the MERGE sql script to merge the data from the external table into the database table
Am I on the right track?
In which stage should I create and drop the external table?
Thanks!

How to have more than one condition on same column --- using SQL Loader

Hi All,
I am stuck with SQL Loader..
How do I filter records before loading in the table using when clause..
i should load data only when
field1 = 'AC' or 'VC'
field2 is NULL
i used various combinations in when clause like
a) when field1='AC' or field1='VC' and field2 = BLANKS
b) when (field1='AC') and (field2 = BLANKS )
& similar...
In all the cases I tried I could not implement OR condition with field1 and null condition with field2
but my main concern is can we use OR or IS NULL things in when clause of SQL Loader..
is it possible to check this anywhere??
any alternate solution u could suggest??
Thanks
Dikshit

Ok I'll try that, although I did try it earlier when I had iTunes 5.xx loaded, I think.
As to size of playlists, I have a master (900 songs) that defines what will fit onto the ipod , I then generate all the others as subsets of the master (not of the library)- hence I know they will all fit
Can you also clarify something for me: the dialogue box we are discussing is intended, I think, so that one can set the automatic synching of certain playlists between the PC & the ipod. Is that the only way one can select other playlists to go to the ipod - i.e. as static once-offs, not synchronised ?
Apple' docs, I think, are poor in this regard - they assume most ipods are bigger then the users song library and they gloss over the details of this alternate mode of playlist synching.
Thanks - Nick

Multiple flat file in and multiple target tables

Hi,
How can we have multiple flat file into multiple targets.
I am trying to load data from multiple flat files into respective tables. But it gives error like
VLD-2411: Cannot handle two file structures
Make sure that only one file structure is used in a SQL*Loader map
Can anyone help.
Regards
Rakesh Kumar

I donot thing in one mapping you can take multiple sqlloader file.
If want to load data form multiple file use External table.

Loading from multiple flat files to same table using SQL Loader

Similar Messages

Maybe you are looking for