SQL Loader - CSV Data file with carraige returns and line fields

Hi,
I have a CSV data file with occasional carraige returns and line feeds in between, which throws my SQL loader script off. Sql loader, takes the characters following the carraige return as a new record and gives me error. Is there a way I could handle carraige returns and linefeeds in SQL Loader.
Please help. Thank you for your time.
This is my Sql Loader script.
load data
infile 'D:\Documents and Settings\user1\My Documents\infile.csv' "str '\r\n'"
append
into table MYSCHEMA.TABLE1
fields terminated by ','
OPTIONALLY ENCLOSED BY '"'
trailing nullcols
( NAME CHAR(4000),
field2 FILLER,
field3 FILLER,
TEST DEPT CHAR(4000)
)

You can "regexp_replace" the columns for special characters

Similar Messages

How to load unicode data files with fixed records lengths?

Hi!
To load unicode data files with fixed records lengths (in terms of charachters and not of bytes!) using SQL*Loader manually, I found two ways:
Alternative 1: one record per row
SQL*Loader control file example (without POSITION, since POSITION always refers to bytes!)<br>
LOAD DATA
CHARACTERSET UTF8
LENGTH SEMANTICS CHAR
INFILE unicode.dat
INTO TABLE STG_UNICODE
TRUNCATE
A CHAR(2) ,
B CHAR(6) ,
C CHAR(2) ,
D CHAR(1) ,
E CHAR(4)
) Datafile:
001111112234444
01NormalDExZWEI
02ÄÜÖßêÊûÛxöööö
03ÄÜÖßêÊûÛxöööö
04üüüüüüÖÄxµôÔµ Alternative2: variable length records
LOAD DATA
CHARACTERSET UTF8
LENGTH SEMANTICS CHAR
INFILE unicode_var.dat "VAR 4"
INTO TABLE STG_UNICODE
TRUNCATE
A CHAR(2) ,
B CHAR(6) ,
C CHAR(2) ,
D CHAR(1) ,
E CHAR(4)
) Datafile:
001501NormalDExZWEI002702ÄÜÖßêÊûÛxöööö002604üuüüüüÖÄxµôÔµ Problems
Implementing these two alternatives in OWB, I encounter the following problems:
* How to specify LENGTH SEMANTICS CHAR?
* How to suppress the POSITION definition?
* How to define a flat file with variable length and how to specify the number of bytes containing the length definition?
Or is there another way that can be implemented using OWB?
Any help is appreciated!
Thanks,
Carsten.

Hi Carsten
If you need to support the LENGTH SEMANTICS CHAR clause in an external table then one option is to use the unbound external table and capture the access parameters manually. To create an unbound external table you can skip the selection of a base file in the external table wizard. Then when the external table is edited you will get an Access Parameters tab where you can define the parameters. In 11gR2 the File to Oracle external table can also add this clause via an option.
Cheers
David

SQL*Loader Sequential Data File Record Processing?

If I use the conventional path will SQL*Loader process a data file sequentially from top to bottom? I have a file comprised of header and detail records with no value found in the detail records that can be used to relate to the header records. The only option is to derive a header value via a sequence (nextval) and then populate the detail records with the same value pulled from the same sequence (currval). But for this to work SQL*Loader must process the file in the exact same sequence that the data has been written to the data file. I've read through the 11g Oracle® Database Utilities SQL*Loader sections looking for proof that this is what will happen but haven't found this information and I don't want to assume that SQL*Loader will always process the data file records sequentially.
Thank you

Oracle Support responded with the following statement.
"Yes, SQL*LOADER process data file from top to bottom.
This was touched in the note below:
SQL*Loader - How to Load a Single Logical Record from Physical Records which Include Linefeeds (Doc ID 160093.1)"
Jason

How can we load a flat file with very, very long lines into a table?

Hello:
We have to load a flat file with OWB. The problem is that each of the lines in the file might be up to 30.000 characters long (up to 1.000 units of information in each line, 30 characters long each)
Of course, our mapping should insert these units of information as independent rows in a table (1.000 rows, in our example).
We do not know how to go about it. We usually load flat files using table functions, but we am not sure that they will be able to cope with these huge lines. And how should we pivot those lines? Will the Pivot operator do the trick? Or maybe we should pivot those lines outside the database before loading them?
We are a bit lost. Any suggestion would be appreciated.
Regards
Edited by: [email protected] on Oct 29, 2008 8:43 AM
Edited by: [email protected] on Oct 29, 2008 8:44 AM

Yes, well, we could define a 1.000 column external table, and then map those 1.000 columns to the Pivot operator… perhaps it would work. But we have been investigating a little bit, and we think that we have found a better solution: there is a unix utility called “fold”. This utility can split our 30.000 character lines in 1.000 lines, 30 characters long each: just what we needed. Then we can load the resulting file using an external table.
We think this is a much better solution that handling 1.000 columns in the external table and in the Pivot operator.
Thanks for your help.
Regards
Edited by: [email protected] on Oct 29, 2008 10:35 AM

Sql loader maximum data file size..?

Hi - I wrote sql loader script runs through shell script which will import data into table from CSV file. CSV file size is around 700MB. I am using Oracle 10g with Sun Solaris 5 environment.
My question is, is there any maximum data file size. The following code from my shell script.
SQLLDR=
DB_USER=
DB_PASS=
DB_SID=
controlFile=
dataFile=
logFileName=
badFile=
${SQLLDR} userid=$DB_USER"/"$DB_PASS"@"$DB_SID \
          control=$controlFile \
          data=$dataFile \
          log=$logFileName \
          bad=$badFile \
          direct=true \
          silent=all \
          errors=5000Here is my control file code
LOAD DATA
APPEND
INTO TABLE KEY_HISTORY_TBL
WHEN OLD_KEY <> ''
AND NEW_KEY <> ''
FIELDS TERMINATED BY ','
TRAILING NULLCOLS
        OLD_KEY "LTRIM(RTRIM(:OLD_KEY))",
        NEW_KEY "LTRIM(RTRIM(:NEW_KEY))",
        SYS_DATE "SYSTIMESTAMP",
        STATUS CONSTANT 'C'
)Thanks,
-Soma
Edited by: user4587490 on Jun 15, 2011 10:17 AM
Edited by: user4587490 on Jun 15, 2011 11:16 AM

Hello Soma.
How many records exist in your 700 MB CSV file? How many do you expect to process in 10 minutes? You may want to consider performing a set of simple unit tests with 1) 1 record, 2) 1,000 records, 3) 100 MB filesize, etc. to #1 validate that your shell script and control file syntax function as expected (including the writing of log files, etc.), and #2 gauge how long the processing will take for the full file.
Hope this helps,
Luke
Please mark the answer as helpful or answered if it is so. If not, provide additional details.
Always try to provide actual or sample statements and the full text of errors along with error code to help the forum members help you better.

SQL Loader: Multiple data files to Multiple Tables

How do you create one control file that refrences multiple data file and each file loads data in a different table.
Eg.
DataFile1 --> Table 1
DataFile2 --> Table 2
Contents and Structure of both data files are different. Data file is comma seperated.
Below example is for 1 data file to 1 table. Need to modify this or create a wrapper that would call multiple control files.
OPTIONS (SKIP=1)
LOAD DATA
INFILE DataFile1
BADFILE DataFile1_bad.txt'
DISCARDFILE DataFile1_dsc.txt'
REPLACE
INTO TABLE Table1
FIELDS TERMINATED BY ","
TRAILING NULLCOLS
Col1,
Col2,
Col3,
create_dttm sysdate,
MySeq "myseq.nextval"
Welcome any other suggestions.

I was thinking if there is a way to indicate what file goes with what table (structure) in one control file.
Example ( This does not work but wondering if something similar is allowed..)
OPTIONS (SKIP=1)
LOAD DATA
INFILE DataFile1
BADFILE DataFile1_bad.txt'
DISCARDFILE DataFile1_dsc.txt'
REPLACE
INTO TABLE Table1
FIELDS TERMINATED BY ","
TRAILING NULLCOLS
Col1,
Col2,
Col3,
create_dttm sysdate,
MySeq "myseq.nextval"
INFILE DataFile2
BADFILE DataFile2_bad.txt'
DISCARDFILE DataFile2_dsc.txt'
REPLACE
INTO TABLE "T2"
FIELDS TERMINATED BY ","
TRAILING NULLCOLS
T2Col1,
T2Col2,
T2Col3
)

SQL Loader reads Data file Sequentially or Randomly?

Will SQL Loader loads the data read from file Sequentially or Randomly?
I have the data file like the below
one
two
three
four
and my control file is
LOAD DATA
INFILE *
TRUNCATE
INTO TABLE T TRAILING NULLCOLS
x RECNUM,
y POSITION (1:4000)
so my table will be polulated like
X Y
1 one
2 Two
3 Three
4 Four
Will this happend sequentially even for the large data sets? say i have from one to one million datas in my data files.
Please clarify.
Thanks,
Rajesh.

SQL Loader may read the file sequentially, but you should not rely on the physical ordering of the rows in the table.
It looks like that's what you were hinting at.

Report to a PDF file with Ole Obj and codebar fields

Im writing a report with some ole objects (images) and a codebar Field. When I run the report saving the result to a PDF file the Ole objects and the codebar fields doesnt appears in it. But, when run it directly in the report builder it does appears. Some idea about it? I would apreciate it.
Luis

You mean a barcode? That is a font, not an OLE object. If you have the font installed on your PC, you can have barcodes in your pdf report.
PDF also has the possibility on embedded fonts, so the font doesn't have to be on every PC. There are several threads about this. Search this forum for 'embedded font', e.g. Font embedding in reports

XML is cluttered with carriage returns and line feeds - how to suppress?

Hi all,
at least within text paragraphs we would prefer 'clean' XML without carriage returns. Can Framemaker be configured to avoid these control character?
Below is an example of a DITA file produced by Framemaker. Evidently this is related to the Pretty Printing feature and I do admit that it increases the readability. However they cause some issues for us. Note that the carriage returns in the paragraph are at arbitrary positions, not at the end of the lines as they appear in the Framemaker editor (I inserted the word "End" at the end of the first line).
Robert

By default, FrameMaker adds a CR at around 70-80 characters unless instructed to do otherwise.
Also, soft returns (SHIFT+ENTER) are lost unless you're in a pre formatted element like a <codeblock> (which is probably why there's no CR after your "End" text.
If you want to limit the seemingly arbitrary CRs, you must tell FM where to put them. Add the following rule to the structure application rules file ..
element "p"
writer line break is 99999 characters;
…scott

Uploading data file with specific encoding and other metadata

Hello everyone.
I've been recently testing Microsoft Codename "Data Hub". When select the option to upload a CSV or XLSX file to populate my database, it basically has two problems, in my opinion:
1. It would be great if I could select the file's encoding. If my file is in Spanish, it won't properly load latin characters (e.g. accents, or special characters). The information ends up showing with a bunch of weird characters.
2. It also doesn't let me select which character to use as data separator. It uses the comma as the default. Some of the data might contain commas, so I usually change the separator to a pipe in my CSV files.
Does anybody know any workarounds?
Thanks.

Hey there, and apologies. I wanted to reach out and see if this is still a problem for you.
I also wanted to check if this is a question about Azure SQL Database or something else?
Thanks Guy

UNICODE data files with SQLLDR

how can i load UNICODE data files with SQLLDR.
my Oracle instance is on UNIX with NLS_CHARACTERSET WE8ISO8859P1.
I have .dat files extracted from SQL Server using bcp utility with -w option.
When i use -c option i'm not getting the european characters correctly like the a and e with 2 dots on top....
when i load UNICODE (-w) file with CHARACTERSET UTF8 in my control file, it doesnt go thru. Any solution for this ? Thanks !

I just created a unicode textfile on windows with some westeuropean characters and imported it into we8iso8859p1 database on linux using controlfile parameter CHARACTERSET UTF16.
They got all properly converted.
As Justin mentioned, unicode on windows means generally UTF16 Little Endian.
Best regards
Maxim

Crystal Report with text(csv) data file, can we set it as input parameter?

Hi,
I am a new user of Crystal Reports 2008.
I have created a report with charts in it. The input data comes from a csv text file.
Can I set the name of this text file as an input parameter?
as I need to generate 44 similar reports with different text filenames(and data)?
Thank you.
Regards

Brian,
Thanks much.
I did exactly what you said.
Just to see any change, I first gave a bad report file name just to see if I am accidentally pointing to a different file,
but I got an error saying report not found.
Then I renamed my original datafile name and generated a report and it still generated one without giving an error.
Then I also gave a junk name to the logoninfo and printed that name, the new name was assigned to logoninfo, but the code did not error out.
It ended up generating the report.
Now here is what I think is happening,
1) The save data in report option seems to be still on even though I have turned it off in 2 locations
a) file -> Report Options
b) file -> Options -> Reporting tab.
2) For some reason the logoninfo is getting ignored as well.
Since I did not see any answers yesterday I posted a link to this thread on the .Net forum
Crystal Report with text(csv) data file, can we set it as input param? C#
and Ludek Uher says that I am connecting to the text file via a DAO database engine and so need to use the same code for changing the text file as for changing an Access database.
But the link he gave me tells me to try the same thing that we have been trying..
Here is my plan,
1) I will first try and find out why my save data with report option is still on ( but it shows off in Crystal ).
2) why is LogonInfo getting ignored.
Meanwhile any suggestions from anyone are welcome.

Crystal Report with text(csv) data file, can we set it as input param? C#

Hi,
I am new to the forums and posted a question which belonged to the .net - SAP Crystal reports group.
Can someone help me with my problem? following is the thread that I have started.
Crystal Report with text(csv) data file, can we set it as input parameter?
Thank you in advance.

Looking at the original thread, you are connecting to the text file via the DAO database engine:
"I added the text file as follow, new connection -> Access/Excell (DAO) -> select the file and the database type as text"
Thus I would use the same code for changing the text file as for changing an Access database. See Kbase [1218178 - Error: "Logon failed" when connecting to Access database in .NET application|http://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/com.sap.km.cm.docs/oss_notes_boj/sdn_oss_boj_bi/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/scn_bosap/notes%7B6163636573733d36393736354636443646363436353344333933393338323636393736354637333631373036453646373436353733354636453735364436323635373233443330333033303331333233313338333133373338%7D.do] for more information. If that does not work, you may want to consider connecting via ODBC or feeding the data from the text file to and ADO .NET dataset and pointing the report at the dataset.
Ludek
Follow us on Twitter http://twitter.com/SAPCRNetSup
Got Enhancement ideas? Try the [SAP Idea Place|https://ideas.sap.com/community/products_and_solutions/crystalreports]

Load one text file with 12 periods' data into 12 different periods at once?

Hi guys,
In one swoop, can we load one .txt file with 12 periods of data into 12 different periods?
The scenario:
Budget data is required for monthly comparative reporting with actuals, so we have 12 periods in our Budget version.
From a non-SAP system we get one .txt file containing 12 periods worth of budget data,
- it is in the correct format (and we don't want to create risk by opening and editing it) so it is loaded into each period and the other 11 periods of irrelevant data are obviously ignored.
Some extra tasks (such as validation, cashflow calculations) are then performed per period.
Because this has to be repeated 12 times, this can be a time consuming process
We now have multiperiod monitor functionality (from EHP2) and I see how it works for the automatic tasks (very well).
I'm aware that the guide says that manual tasks will be ignored during the automatic run, this is true.
However, if I remember correctly EC-CS used to allow it with upoad files, so i was expecting it in BCS 6.02
- is there anyway to load a file containing 12 periods worth of data into 12 individual periods in all at once?
(NB we still have an improvement to the previous situation, the user can scroll between periods more quickly and load the file 12 times, then go back to the start and run all auto tasks at once)..
One thought was to use a file server location with a hardcoded filename but his would require work beyond my expertise.
All suggestions welcome

Hi
I wil suggest mapping the path based on a common location is the best way and then map the same logically in the Upload method
If you are using Citrix then it becomes all the more easy to have a commmon location
Rgds
Dheeraj

FND function to create .csv data file on client machine?

Hello All,
I am trying to create a concurrent request which when run by the user's creates a .csv file with data. I created a procedure in the database which run's when they run the conc.request.
However is there any FND function or anyother function available that creates the data file on the client machine (either in their 'C:\') or in a share drive etc?
Currently through the procedure, I am able to write the .csv data file to a unix directory to which users do not have access.
Thanks,
Chiru

Chiru,
Several options for you to consider are :
1. If you want to have the csv on client machine: Make the program to create the csv in server with a constant name, eg : test.csv and create a dos command in client's machine that is could FTP to server, rename the existing csv file in client's machine and get test.csv from server.
2. If you want to do in share drive : As long as the share drive exists in PC that can be used for FTP, you can create unix host script that can FTP the file to the windows server. For instance, create a generic unix host script that accepts parameter such as filename, and server destination, destination folder, unix login ,password, etc. This program will run in a request set after your concurrent program. If you search google I'm sure you will find samples of unix script that does this kind of thing.
3. As per what Kevin said ,another options is to send the csv file using email to the recipients.
HTH

SQL Loader - CSV Data file with carraige returns and line fields

Similar Messages

Maybe you are looking for