Multiple Files Loading

Hi
I have requirement to load multiple files into oracle database. I am in confusion whether i should use Sql Loader scripts to load or create a mapping in OWB for better performance.
Could someone suggest which one gives best solution?
P.S: All files are same format

If you use a file as a source in a mapping OWB will generate a SQL*Loader mapping. You can configure the file operator to load many files, it can also be supplied as a parameter.
Cheers
David

Similar Messages

Error in loading the multiple file loading

hi,
i am following this blog to load the multiple files into rdbms table.
http://www.odigurus.com/2011/05/multiple-files-single-target-table.html
the problem is that when i assign the source Flat file Datastore
Model Resource Name #Project_Name.FILE_NAME
But when i point the Single File in the Resource Name:EMP.DMP. its working fine. loaded suceessfully.... But problem in Multiple loading???
it gives me error
SQL*Loader-500: Unable to open file (E:/File/#Project_Name.FILE_NAME)
SQL*Loader-553: file not found
SQL*Loader-509: System error: The system cannot find the file specified.
SQL*Loader-2026: the load was aborted because SQL Loader cannot continue.
********************Task Error Code In Loading**********************************************
import java.lang.String
import java.lang.Runtime as Runtime
from jarray import array
import java.io.File
import os
import re
ctlfile = r"""E:/File/Med_CDR_File.ctl"""
logfile = r"""E:/File/Med_CDR_File.log"""
outfile = r"""E:/File/Med_CDR_File.out"""
oracle_sid=''
if len('')>0: oracle_sid = '@'+''
loadcmd = r"""sqlldr 'test/<@=snpRef.getInfo("DEST_PASS") @>%s' control='%s' log='%s' > "%s" """ % (oracle_sid,ctlfile, logfile, outfile)
rc = os.system(loadcmd)
if rc <> 0 and rc <> 2:
     raise "Load Error", "See %s for details" % logfile
# Init Vars
nbIns = 0
nbRej = 0
nbNull = 0
strprt = ""
maxAllowedError = r"""0"""
c = 0
flag = 0
# Open log file
f = open(logfile, "r")
try:
     lines = f.readlines()
     for line in lines:
          if line.rstrip().upper().endswith(r"""TEST.TC$_0MSC_SMS:""".upper()):
               flag = 1
               c = 0
          if flag == 1:
               if c > 0 and c <= 4:
                    if c == 1 :
                         nbIns = int(re.findall("\d+", line)[0])
                    elif c == 2:
                         nbRej = int(re.findall("\d+", line)[0])
                    elif c == 4:
                         nbNull = int(re.findall("\d+", line)[0])
                         break
          c+=1
     strprt = "\n\tIns:\t%s\n\tReject:\t%s\n\tNullField:\t%s" % (nbIns, nbRej, nbNull)
finally:
     f.close()
# if some rows has been rejected due to invalide data, check KM option LOA_ERRORS
if rc == 2:
     if nbRej > int(maxAllowedError):
          raise strprt
          break
waiting
Regards,
Edited by: AMSI on Nov 10, 2012 1:40 AM

Hi AMSI,
Did you try to run only the interface? It might not work.
Try executing your interface from within a package. A previous step should refresh / set the variable.
Regards,
Jerome Fr

Multiple files load

hello all,
maybe you could help me with the following problem. I have several files of almost 2GB having the exact structure that are to be loaded in a certain table. the filenames are file1, file2, .. filen where n has always a certain constant value.
question: Is there a way I could load all these files using only one interface and define the file format as a datastore only once?
thanks in advance for any suggestion

Hi shugyosha,
We can do this in two method,
Method1
1) Create the File datastore in model for the file to be uploaded(in this datastore creation name should be kept like 'File#FNumber
Where #FNumber is a variable
2) create an interface(File_Upload),Where soure datastore is file and the target is the Oracle table(is a table where the data in file to be loaded)
3)create a variable as FNumber with the sql query of 'select #FNumber from dual' and select the increment option
4) create the Package for file upload, In that package place the variable #FNumber and interface
5) keep the flow like #Fnumber variable as declare variable and take a copy of this variable and use it as refresh variable and that take a copy again and use that copy as evaluate variable
Check the condition like < 'maximum file number'
if it is true means goto interface
6) keep the false codition blank
Thanks
Madha

Unable to load multiple files to Essbase using MaxL and wildcards

I have multiple data files to load:
Files:
Filename.txt
Filename_1.txt
Filename_2.txt
According to the following link, Essbase is able to load multiple files to BSO databases via MaxL by using wildcards:
http://docs.oracle.com/cd/E17236_01/epm.1112/esb_tech_ref/frameset.htm?launch.html
However, when I try to run the following I receive the following error:
MaxL:
import database MyApp.DB data from server text data_file "../../MyApp/Filename*.txt" using server rules_file "L_MyRule" on error append to "\\Server\Folder\L_MyRule.err";
Error:
ERROR - 1003027 - Unable to open file [DB01/oracleEPM/user_projects/epmsystem2/EssbaseServer/essbaseserver1/app/MyApp/DB/../../Filename*.txt].
ERROR - 1241101 - Unexpected Essbase error 1003027.
I can run the following fine without any problem, but this only deals with the first of several files, and I’d rather not hard-code multiple files since the number might vary in the future:
MaxL:
import database MyApp.DB data from server text data_file "../../MyApp/Filename.txt" using server rules_file "L_MyRule" on error append to "\\Server\Folder\L_MyRule.err";
Any ideas? And how about ASO databases?

JamesD wrote:
According to the following link, Essbase is able to load multiple files to BSO databases via MaxL by using wildcards:
http://docs.oracle.com/cd/E17236_01/epm.1112/esb_tech_ref/frameset.htm?launch.html
That is the tech ref for 11.1.2.2, are you on 11.1.2.2?
Import Data
11.1.2.2 - http://docs.oracle.com/cd/E17236_01/epm.1112/esb_tech_ref/frameset.htm?maxl_imp_data.html
11.1.2.1 - http://docs.oracle.com/cd/E17236_01/epm.1112/esb_tech_ref_1112100/frameset.htm?maxl_imp_data.html
Cheers
John
http://john-goodwin.blogspot.com/

Reader XI: how to load multiple files so that I can show them in one instance of full screen mode?

It is possible to show a pdf file in full screen mode in acrobat reader XI (windows).
It is indicated that this is also possible with multiple files.
I do not understand how to accomplish this; How can I load more than one file to show in the full screen mode?
can anyone explain this to me?
Thanks in advance!

frankvanrooij wrote:
It is indicated that this is also possible with multiple files.
Where do you see that?

Load multiple files using the same data load location

has anybody tried loading multiple files using the same load locations. I need to do this as the data in these multiple files will need to be exported from FDM as a single export file. the problem i am facing is more user related. since these files will be received at different points of time, users will need a way to tell them what has been loaded and what is yet to be loaded.
is it possible to throw a window on the web broser with OK and Cancel buttons from an event script?
any pointers to possible solutions will be helpful

was able to resolve this. the implementation method is as follows
take a back up of previously imported data in the befcleardata event script. then in the beffileimport event append the data to the import file. there are many other intricacies but this is the broad implementation logic. it allowed my users to load multiple files without worrying about append or replace import type choices

Loading multiple files with SQL Loader

Hello.
I will appreciate your recommendation about the way to load multiple files (to multiple tables) using SQL Loader with only one Control file.
file1 to load to Table1, file2 to load to Table2 etc.
How the Control file should look like?
I was looking on Web, but didn't find exactly what I need.
Thanks!

Ctl File : myctl.ctl
---------- Start ---------
LOAD DATA
INFILE 'F:\sqlldr\abc1.dat'
INFILE 'F:\sqlldr\abc2.dat'
INTO TABLE hdfc1
(TRANS_DATE CHAR,
NARRATION CHAR,
VALUE_DATE CHAR,
DEBIT_AMOUNT INTEGER,
CREDIT_AMOUNT INTEGER,
CHQ_REF_NUMBER CHAR,
CLOSING_BALANCE CHAR)
INTO TABLE hdfc2
(TRANS_DATE CHAR,
NARRATION CHAR,
VALUE_DATE CHAR,
DEBIT_AMOUNT INTEGER,
CREDIT_AMOUNT INTEGER,
CHQ_REF_NUMBER CHAR,
CLOSING_BALANCE CHAR)
-----------End-----------
Sqlldr Command
sqlldr scott/tiger@dbtalk control=F:\sqlldr\myctl.ctl log=F:\sqlldr\ddl_file1.txt
Regards,
Abu

SQL Loader to Load Multiple Tables from Multiple Files

Hi
I wish to create a control file to load multiple tables from multiple files
viz.Emp.dat into emp table and Dept.dat into Dept table and so on
How could I do it?
Can I create a control file like this:
OPTIONS(DIRECT=TRUE,
SKIP_UNUSABLE_INDEXES=TRUE,
SKIP_INDEX_MAINTENANCE=TRUE)
UNRECOVERABLE
LOAD DATA
INFILE 'EMP.dat'
INFILE 'DEPT.dat'
INTO TABLE emp TRUNCATE
FIELDS TERMINATED BY "|" OPTIONALLY ENCLOSED BY '"'
(empno,
ename,
deptno)
INTO TABLE dept TRUNCATE
FIELDS TERMINATED BY "|" OPTIONALLY ENCLOSED BY '"'
(deptno,
dname,
dloc)
Appreciate a Quick Reply
mailto:[email protected]

Which operating system? ("Command Prompt" sounds like Windows)
UNIX/Linux: a shell script with multiple calls to sqlldr run in the background with "&" (and possibly nohup)
Windows: A batch file using "start" to launch multiple copies of sqlldr.
http://www.pctools.com/forum/showthread.php?42285-background-a-process-in-batch-%28W2K%29
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/start.mspx?mfr=true
Edited by: Brian Bontrager on May 31, 2013 4:04 PM

How to Load Multiple Files in Oracle Database using Sql Loader

Hi All,
I want to import multiple files in my DB using Sql*Loader. Please tell me the Syntax, how can I import multiple files using one Control File?
Thanks & Regards,
Imran

Hi,
You might get a good response to your post in the forum dedicated to data movement , including SQL*Loader . You can find it here Export/Import/SQL Loader & External Tables
Regards,

SQL*Loader with multiple files

Gurus,
I search the documentation and this forum and haven't found a solution to my issue yet...
I am not expert of SQL*Loader. I have used SQL*Loader to copy from one file to a table many times. But I have not copied multiple files into one table especially with different names.
More specifically....
I need to load data from multiple files into a table. But the file names will be different each time. A file will be created every hour. The file name will consist of the root file name appended by a time stamp. For example, a file created on 10/07/2010 at 2:15 P.M. would be filea100720101415.txt while a file created on 10/08/2010 at 8:15 A.M. would be filea100820100815.txt. All the files will be in one directory.How can I load the data from the files using SQL*Loader?
My database: Oracle 10g Release 2
Operating System: Windows 2003 Server
Please assist.
Robert

sect55 wrote:
Gurus,
I search the documentation and this forum and haven't found a solution to my issue yet...
I am not expert of SQL*Loader. I have used SQL*Loader to copy from one file to a table many times. But I have not copied multiple files into one table especially with different names.
More specifically....
I need to load data from multiple files into a table. But the file names will be different each time. A file will be created every hour. The file name will consist of the root file name appended by a time stamp. For example, a file created on 10/07/2010 at 2:15 P.M. would be filea100720101415.txt while a file created on 10/08/2010 at 8:15 A.M. would be filea100820100815.txt. All the files will be in one directory.How can I load the data from the files using SQL*Loader?
My database: Oracle 10g Release 2
Operating System: Windows 2003 Server
Please assist.
RobertToo bad this isn't in *nix, where you get a powerful shell scripting capability.
That said, here is the core of the solution .... you will also need a way to identify files that have been processed vs. new ones. Maybe rename them, maybe move them. But with this sample you can see the basics. From there it is really an issue of DOS scripting, which would better be found by googling around a bit.
cd c:\loadfiles
FOR %%datfile IN (*.txt) DO SQLLDR CONTROL=sample.ctl, LOG=sample.log, BAD=baz.bad, DATA=%%datfileTry googling "dos scripting language". You'll find lots of tutorials and ideas on "advanced" (well, as advanced as DOS gets) techniques to solve your problem.
Edited by: EdStevens on Dec 1, 2010 5:03 PM

Sql*loader map multiple files to multiple tables

Can a single control file map multiple files to multiple different tables? If so, what does the syntax look like? I've tried variations of the following, but haven't hit the jackpot yet.
Also, I understand that a direct load will automatically turn off most constraint checking. I'd like to turn this back on when I'm done loading all tables. How/when do I do that? I can find multiple references to 'REENABLE DISABLED CONSTRAINTS', but I don't know where to say that.
TIA.
LOAD DATA
INFILE 'first.csv'
TRUNCATE
INTO TABLE first_table
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
(a,b,c)
INFILE 'second.csv'
TRUNCATE
INTO TABLE second_table
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
(x,y,z,xx,yy,zz)
etc.

Here you go what you want..
http://www.psoug.org/reference/sqlloader.html
LOAD DATA
INFILE 'c:\temp\demo09a.dat'
INFILE 'c:\temp\demo09b.dat'
APPEND
INTO TABLE denver_prj
WHEN projno = '101' (
projno position(1:3) CHAR,
empno position(4:8) INTEGER EXTERNAL,
projhrs position(9:10) INTEGER EXTERNAL)
INTO TABLE orlando_prj
WHEN projno = '202' (
projno position(1:3) CHAR,
empno position(4:8) INTEGER EXTERNAL,
projhrs position(9:10) INTEGER EXTERNAL)
INTO TABLE misc_prj
WHEN projno != '101' AND projno != '202' (
projno position(1:3) CHAR,
empno position(4:8) INTEGER EXTERNAL,
projhrs position(9:10) INTEGER EXTERNAL)
Thanks
Aravindh

SQL Loader - Load multiple files in UNIX

HI all, I'm looking for a bit of help with using SQL LOADER to load multiple files into one table. I've had a look on the forums but still struggling with this one.
What I want to do is basically upload everything thats in /home/ib. I know you can use INFILE for several files in the control file but I have several hundred files to upload so this isn't practical. Can I pass the directory name as an INFILE parameter?
Any help would be appreciated.

On Unix you shouldn't worry about that. See this example :
[ora102 work db102]$ cat test11.dat
aaaaa,bbbbb
ccccc,ddddd
eeeee,fffff
[ora102 work db102]$ cat test12.dat
ggggg,hhhhh
jjjjj,kkkkk
lllll,mmmmm
[ora102 work db102]$ cat test13.dat
nnnnn,ooooo
ppppp,qqqqq
rrrrr,sssss
[ora102 work db102]$ cat load.sh
CTL=load.ctl
echo "load data" > $CTL
for DAT in test1*.dat
do
        echo "INFILE "$DAT >> $CTL
done
echo "replace"                  >> $CTL
echo "INTO TABLE test1"         >> $CTL
echo "fields terminated by ','" >> $CTL
echo "trailing nullcols"        >> $CTL
echo "( a, b )"                 >> $CTL
sqlldr test/test control=load.ctl
[ora102 work db102]$ ./load.sh
SQL*Loader: Release 10.2.0.1.0 - Production on Mon Oct 2 11:45:44 2006
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Commit point reached - logical record count 3
Commit point reached - logical record count 6
Commit point reached - logical record count 9
[ora102 work db102]$ sqlplus test/test
SQL*Plus: Release 10.2.0.1.0 - Production on Mon Oct 2 11:45:49 2006
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
TEST@db102 SQL> select * from test1 order by a,b;
A                    B
aaaaa                bbbbb
ccccc                ddddd
eeeee                fffff
ggggg                hhhhh
jjjjj                kkkkk
lllll                mmmmm
nnnnn                ooooo
ppppp                qqqqq
rrrrr                sssss
9 rows selected.
TEST@db102 SQL>

Failed to load multiple files using theMgr.createLocalFileInputStream

Hi
I am using 2.3.8 and having trouble with putting multiple files to the container using XmlInputStream.
I have hundreds of documents in 'output' folder from my project root, and I was able to load them by using other signature - reading each line from each file via BufferedReader.
However, if I switch to XmlInputStream, the first putDocument() is successful but I get an error an XmlException at the second iteration.
container = theMgr.createContainer(theContainer, config);
XmlUpdateContext updateContext = theMgr.createUpdateContext();
Transaction dbtxn = environment.beginTransaction(null, null);
txn = theMgr.createTransaction(dbtxn);
Iterator filesIterator = files2add.iterator();
while (filesIterator.hasNext()) {
File file = (File) filesIterator.next();
String docName = fileNameToDocName(file);
XmlInputStream theStream = theMgr.createLocalFileInputStream(file.getName());
container.putDocument(docName,
theStream,
updateContext,
null);
com.sleepycat.dbxml.XmlException: Error: The resource does not exist: malformed or non-existent stream source, errcode = INVALID_VALUE
     at com.sleepycat.dbxml.dbxml_javaJNI.XmlContainer_putDocument__SWIG_5(Native Method)
     at com.sleepycat.dbxml.XmlContainer.putDocument(XmlContainer.java:405)
     at com.sleepycat.dbxml.XmlContainer.putDocument(XmlContainer.java:77)
     at com.expd.arch.persistence.BerkeleyTestCase.loadXmlFilesViaStream(BerkeleyTestCase.java:85)
The second document failing looks like this - nothing wrong
<?xml version="1.0" encoding="UTF-8"?>
<consol version="0">
<import-file-number>21345</import-file-number>
<ownership-location>LAX</ownership-location>
<id>0</id>
</consol>
Can someone tell what's wrong in my code?

Hello,
The error is indicating something wrong with the file name or location, not the content. Try verifying that the file name passed to createLocalFileInputStream() is in good shape and refers to a regular file with permission to read. If that looks OK, there's a small chance you need to explicitly delete the XmlInputStream between calls. That should not be necessary, because the putDocument() call should consume it, but it does not hurt.
Regards,
George

Sql loader multiple files

How can I load multiple input files ( monthwise 200601 to 200907) using sqlloader and a batchfile into tables like xxxx200601 etc
any suggestions are welcome,
thanks in advance
Edited by: sunnysmiles on Jul 10, 2009 2:14 PM

sunnysmiles wrote:
How can I load multiple input files ( monthwise 200601 to 200907) using sqlloader and a batchfile
LOAD DATA
INFILE mon200601.dat
INFILE mon200602.dat
INFILE mon200603.dat
INFILE mon200604.dat
INFILE mon200605.dat
INFILE mon200606.dat
INFILE mon200607.dat
APPEND
INTO TABLE emp
( empno POSITION(1:4)   INTEGER EXTERNAL,
    ename POSITION(6:15) CHAR,
    deptno POSITION(17:18) CHAR,
    mgr    POSITION(20:23) INTEGER EXTERNAL
any suggestions are welcome,[loading data from multiple files|http://www.orafaq.com/wiki/SQL*Loader_FAQ#Can_one_load_data_from_multiple_files.2F_into_multiple_tables_at_once.3F], [official documentation|http://download.oracle.com/docs/cd/B19306_01/server.102/b14215/ldr_concepts.htm#i1004677]
thanks in advance

Loading data from multiple files to multiple tables

How should I approach on creating SSIS package to load data from multiple files to multiple tables. Also, Files will have data which might overlap so I might have to create stored procedure for it. Ex. 1st day file -data from au.1 - aug 10 and 2nd day
file might have data from aug.5 to aug 15. So I might have to look for max and min date and truncate table with in that date range.

thats ok. ForEachLoop would be able to iterate through the files. You can declare a variable inside loop to capture the filenames. Choose fully qualified as the option in loop
Then inside loop
1. Add execute sql task to delete overlapping data from the table. One question here is where will you get date from? Does it come inside filename?
2. Add a data flow task with file source pointing to file .For this add a suitable connection manager (Excel/Flat file etc) and map the connection string property to filename variable using expressions
3. Add a OLEDB Destination to point to table. You can use table or view from variable - fast load option and map to variable to make tablename dynamic and just set corresponding value for the variable to get correct tablename
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

Multiple Files Loading

Similar Messages

Maybe you are looking for