SQL LOADER / INFILE filename as variable in .ctl file

I stumbled over several threads in the OTN-forums regarding this problem, but neither was it finaly solved nor did I find a FAQ that answered my question. Soooo:
We get several datafiles from several sources and process them via SQL Loader and store 'em in the DB.
This is done via a CHRON job and a PERL skript, for all datafiles in a specific directory.
We need the information which file on which date generated the data INSIDE the DB as well.
So I want to store the filename || SYSDATE combination as well.
I know, I could parse the .ctl file and replace a key-string with the actual filename and so have it in the input.
But this seems a bit dirty to me. Isn't there some way, i.e. a keyword or variable for the infile-filename within the SQLLoader that I can access in the .ctl file? Something like:
INTO TABLE processed_files
FIELDS TERMINATED BY ';'
WHEN LOWER(LTRIM(RTRIM(hdr_ftr))) = 'ftr' -- FOOTER??
(hdr_ftr VARCHAR2(100),
source INFILE||' on '||TO_CHAR(SYSDATE, 'MM/DD/YYYY'),
realm VARCHAR2(100),
version VARCHAR2(20)
I would be greatfull if you?d share your wisdom with me. ;-))
Oliver

I passed this quite similar to 'Ask Tom' and got the advice to put the .ctl's content as a string variable into a Shell skript.
This shell skript (which had to be written anyway to loop over the datafiles an subsequently call the sqlldr) should then replace the INFILE parameter and the CONSTANTs for the filenames and generate a 'temporarry' .ctl before calling sqlldr!
That's it, no better and safer way!

Similar Messages

SQL Loader and foreign characters in the data file problem

Hello,
I have run into an issue which I can't find an answer for. When I run SQL Loader, one of my control files is used to get file content (LOBFILE) and one of the fields in the data file has a path to that file. The control file looks like:
LOAD DATA
INFILE 'PLACE_HOLDER.dat'
INTO TABLE iceberg.rpt_document_core APPEND
FIELDS TERMINATED BY ','
doc_core_id "iceberg.seq_rpt_document_core.nextval",
-- created_date POSITION(1) date "yyyy-mm-dd:hh24:mi:ss",
created_date date "yyyy-mm-dd:hh24:mi:ss",
document_size,
hash,
body_format,
is_generic_doc,
is_legacy_doc,
external_filename FILLER char(275) ENCLOSED by '"',
body LOBFILE(external_filename) terminated by EOF
A sample data file looks like:
0,2012-10-22:10:09:35,21,BB51344DD2127002118E286A197ECD4A,text,N,N,"E:\tmp\misc_files\index_testers\foreign\شیمیایی.txt"
0,2012-10-22:10:09:35,17,CF85BE76B1E20704180534E19D363CF8,text,N,N,"E:\tmp\misc_files\index_testers\foreign\ลอบวางระเบิด.txt"
0,2012-10-22:10:09:35,23552,47DB382558D69F170227AA18179FD0F0,binary,N,N,"E:\tmp\misc_files\index_testers\foreign\leesburgis_á_ñ_é_í_ó_ú_¿_¡_ü_99.doc"
0,2012-10-22:10:09:35,17,83FCA0377445B60CE422DE8994900A79,binary,N,N,"E:\tmp\misc_files\index_testers\foreign\làm thế nào bạn làm ngày hôm nay"
The problem is that whan I run this, SQL Loader throws an error that it can't find the file. It appears that it can't interpret the foreign characters in a way that allows it to find that path. I have tried adding a CHARACTERSET (using AL32UTF8 or UTF8) value in the control file but that only has some success with Western languages, not the ones listed above. Also, there is no set of defined languages that could be found in the data file. It essentaially could be any language.
Does anyone know if there is a way to somehow get SQL Loader to "understand" the file system paths when a folder and/or file name could be in some other langauge?
Thanks for any thoughts - Peter

Thanks for the reply Harry. If I try to open the file in various text editors like Wordpad, Notepad, GVIM, andTextpad, they all display the foreign characters differently. Only Notepad comes close to displaying the characters properly. I have a C# app that will read the file and display the contents and it renders it fine. If you look at the directory of files in Windows Explorer, they all are displayed properly. So it seems things like .Net and Windows have some mechanism to understand the characters in order to render them properly. Other applications, again like Wordpad, do not know how to render them properly. It would seem that whatever SQL Loader is using to "read" the data files also is not rendering the characters properly which prevents it from finding the directory path to the file. If I add "CHARACTERSET AL32UTF8" in the control file, all is fine when dealing with Western langauges (ex, German, Spanish) but not for the Eastern languages (ex. Thai, Chinese). So .... telling SQL Loader to use a characterset seems to work, but not in all cases. The AL32UTF8 is the characterset that the Oracle database was created with. I have not had any luck if I try to set the CHARACTERSET to whatever the Thai character set is, for example. There problem there though is that even if that did work, I can't target specific lagauages because the data could come from anywhere. It's like I need some sort of global "super set" characterset to use. It seems like the CHARACTERSET is the right track to follow but I am not sure, and even if it is, is there a way to handle all languages.
Thanks - Peter

Sql Loader INFILE name value in table column Value

Hi,
Here is my Sql Loader Script
LOAD DATA
infile '%1'
APPEND INTO TABLE XX_SUPPLIER_UPD
FIELDS TERMINATED BY ";" OPTIONALLY ENCLOSED BY '"' TRAILING NULLCOLS
ACTION Char
,ADDRESS_TYPE Char
,REGION Char "LTRIM(RTRIM(:REGION))"
,PO_BOX Char
,,WWW_ADDRESS Char
,status Char "NVL(:status,'X')"
,filename Char "replace(:infile,'\"','')"
I am getting the infile name as the parameter and i want to insert that parameter value in the column name fillename. Can any one guide me to how to do.
Cheers!
Jayaraj.S

If you were to use external tables instead of SQL*Loader, you can dynamically change the location of the external table (i.e. the filename) using a simple ALTER TABLE statement.
External tables also mean that all the control is inside the database rather than relying on external utilities and external scripts.
;)

Sql Loader with TSN environment variable.

Sql loader command works fine if I have TSN entry in my system,Can I use SQL Loader with in any other way ie by use of some environment variable???
What are the different ways I can use sql ldr ???

952189 wrote:
Sql loader command works fine if I have TSN entry in my system,Can I use SQL Loader with in any other way ie by use of some environment variable???
What are the different ways I can use sql ldr ???
>Sql loader command works fine if I have TSN entry in my system,Can I use SQL Loader with in any other way ie by use of some environment variable???
What are the different ways I can use sql ldr ???
if sqlldr is run on DB Server system, then no tnsnames.ora file is required.

Error while load the data from CSV with CTL file..?

Hi TOM,
When i try to load data from CSV file to this table,
CTL File content:
load data
into table XXXX append
     Y_aca position char (3),
     x_date position date 'yyyy/mm/dd'
NULLIF (x_date = ' '),
X_aca position (* + 3) char (6)
"case when :Y_aca = 'ABCDDD' and :XM_dt is null then
decode(:X_aca,'AB','BA','CD',
'DC','EF','FE','GH','HG',:X_aca)
else :X_aca
end as X_aca",
Z_cdd position char (2),
     XM_dt position date 'yyyy/mm/dd'
NULLIF XM_dt = ' ',
When I try the above CTL file; geting the following error..
SQL*Loader-281: Warning: ROWS parameter ignored in parallel mode.
SQL*Loader-951: Error calling once/load initialization
ORA-02373: Error parsing insert statement for table "XYZ"."XXXX".
ORA-00917: missing comma

Possible Solutions
Make sure that the data source is valid.
Is a member from each dimension specified correctly in the data source or rules file?
Is the numeric data field at the end of the record? If not, move the numeric data field in the data source or move the numeric data field in the rules file.
Are all members that might contain numbers (such as "100") enclosed in quotation marks in the data source?
If you are using a header, is the header set up correctly? Remember that you can add missing dimension names to the header.
Does the data source contain extra spaces or tabs?
Has the updated outline been saved?

SQL Loader - want to access data from .txt file but don't want to insert

Hello,
I have situation where I have some data in .txt file and I have to load that data into one of my table.
My problem is I have fundtion get_id(a,b,c) in database which I have to call while loading the data to find out value of x.
I have value of a, b and c in txt file which I can access it using POSITION(N:M) in sql loader, but After accessing that value I just want to past to get_id() function and dont want to insert into sql loader.
any help

My control file is as below
Following field is not in database but I need to pass the value to the function to get a ESC_USER_ID. any help
OFFICE_NUM      filler POSITION(32:35) ,
STATION_DESK      filler POSITION(45:48),
LOAD DATA
INFILE REFERRALs.TXT
BADFILE REFERRAL_LOAD.BAD
DISCARDFILE REFERRAL_LOAD.DIS
INSERT
INTO TABLE REFERRAL
EVALUATE CHECK_CONSTRAINTS REENABLE DISABLED_CONSTRAINTS
TRAILING NULLCOLS
ID                    sequence (1,1),
JOB_ID                    POSITION (16:22) "GET_JOB_ID(:JOB_ID)"      ,
CUS_ID                    POSITION (23:31) "GET_CUS_ID(:CUS_ID)"     ,
STATUS_LU               CONSTANT 'F'                    ,
REQUEST_USER_TYPE_LU          CONSTANT 'S'                    ,
OFFICE_NUM      filler POSITION(32:35) ,
STATION_DESK      filler POSITION(45:48),
REQUEST_USER_ID          POSITION(143:149) "GET_ESCUSER_ID(OFFICE_NUM,STATION_DESK,:REQUEST_USER_ID)"     ,
REQUEST_DT               POSITION (37:44) "to_date(:REQUEST_DT,'YYYY/MM/DD')"     ,
REVIEW_USER_ID               FILLER                         ,
REVIEW_DT               FILLER                         ,
REFUSE_REASON_LU          FILLER                         ,
RESULT_LU               POSITION (97:97)               ,
NOTIFY_STATUS_LU          FILLER                         ,
NOTIFY_DT               FILLER                         ,
APPOINTMENT_DT               POSITION (106:113) "to_date(:APPOINTMENT_DT,'YYYY/MM/DD')"     ,
RESULT_DT               POSITION (98:105) "to_date(:RESULT_DT,'YYYY/MM/DD')"     ,
NO_MATCH_IND               CONSTANT 'N'
)

SQL*Loader with ZONED EXTERNAL in a delimited file

I have a text file delimited with '|' and numeric data with trailing signs, e.g somedata|12.00-|13.01|42.01-
My table columns are defined as number(9,2) and I defined the datatype in the control file as zoned external. However, I get the dreaded ORA-01722: invalid number error.
Does anyone have any suggestions?

As Bravid says, you can't do that from within SQL*Loader, unless the control files are dynamically generated by a script and the filename included as the default value for that particular column as a hardcoded literal.
It's easier if you use external tables as the table definition contains the "location" which is the list of filenames it is accessing, and the value can be queried as well as being dynamically changed (though if you're dynamically changing it, then you'll programatically know the filename anyway. hint hint!)
;)

SQL Loader: refer datafile name in the control file

Hi All:
Database: 10GR2
I have a table as following:
test (
filename varchar2(100),
mydata varchar2(1000)
I am using sql loader to populate data for this table. I also need store the datafile name to column test.filename as part of dataload. How to refer datafile name in the control file?
Again, I am processing multiple files in a batch mode. I need store datafile name in filename column so that I can know which "mydata" come from which datafile.
Thanks in advance!
Kevin

As Bravid says, you can't do that from within SQL*Loader, unless the control files are dynamically generated by a script and the filename included as the default value for that particular column as a hardcoded literal.
It's easier if you use external tables as the table definition contains the "location" which is the list of filenames it is accessing, and the value can be queried as well as being dynamically changed (though if you're dynamically changing it, then you'll programatically know the filename anyway. hint hint!)
;)

How to refer the SQL*Loader command line parameters in the control file

The problem I am trying to resolve is to insert the input file name when SQL*Loader is loading the record from the file. I have multiple files to load and it is nice to associate a record with a particular input file.

Create the control file dynamically (and fill one of the columns with the file name as a constant) before you start SQL*Loader
Another option is to use external tables
Re: Data Loading
Message was edited by:
Jens Petersen

SQL*Loader and filenames

Is it possible to pick up the name of the file being loaded from within the control file.
I am building a process that automatically loads a file when it is placed in a particular directory. I want to be able to create a 'load history' record. i.e. as well as loading the data from the file into a particular table I want to create a record in another table with the load date/time and the name of the file loaded.
The filename will be of a fixed format with a date embedded.
I will be using Oracle 8.1.7. so no flashy 9i facilities will be available!
Any ideas out there
Thanks
Steve

If you execute sqlloader without any parameters it'll show all that are available. In your case you could write it like this:
<loop>
sqlldr control=<yourctlfile> data=$var_file bad=$var_bad
<end loop>
not tested
Not sure about the "$", I usually don't write scripts in ksh.
C.

SQL*Loader . A column value in data file contains comma(,)

Hi Friends,
I am getting an issue while loading a csv file to database.
A column in datafile contains a comma .. how to load such data?
For ex, a record in data file is :
453,1,452,N,5/18/2006,1,"FOREIGN, NON US$ CORPORATE",,,310
Here "FOREIGN, NON US$ CORPORATE" is a column and contains a , in the value.
I have specified optionally enclosed with " also.. but still not working
Here is my control file:
options (errors=100)
load data
infile 'TAX_LOT_DIM_1.csv'
badfile 'TAX_LOT_DIM_1.bad'
replace
into table TAX_LOT_DIM
fields terminated by ',' optionally enclosed by '"'
trailing nullcols
TAX_LOT_DIM_ID ,
TAX_LOT_NBR ,
TAX_LOT_ODS_ID ,
RESTRICTION_IND ,
LAST_UPDATE_DTM ,
TRAN_LOT_NBR integer,
MGR_GRP_CD optionally enclosed by '"' ,
RESTRICTION_AMT "TO_NUMBER(:RESTRICTION_AMT,'99999999999999999999.999999999999')" ,
RESTRICTION_INFO ,
SRC_MGR_GRP_CD
Problem is with MGR_GRP_CD column in ctrl file.
Please reply asap.
Regards,
Kishore

Thanks for the response.
Actually my ctrl file is like this with some conversion functions:
replace
into table TAX_LOT_DIM
fields terminated by ',' optionally enclosed by '"'
trailing nullcols
TAX_LOT_DIM_ID "TO_NUMBER(:TAX_LOT_DIM_ID ,'999999999999999.99999999')",
TAX_LOT_NBR ,
TAX_LOT_ODS_ID "to_number(:TAX_LOT_ODS_ID ,'999999999999999.999999')",
RESTRICTION_IND ,
LAST_UPDATE_DTM "to_date(:LAST_UPDATE_DTM ,'mm/dd/yyyy')",
TRAN_LOT_NBR integer, --"TO_NUMBER(:TRAN_LOT_NBR,'999999999999999.99999999999999999')",
MGR_GRP_CD char optionally enclosed by '"' ,
RESTRICTION_AMT "TO_NUMBER(:RESTRICTION_AMT,'99999999999999999999.999999999999')" ,
RESTRICTION_INFO ,
SRC_MGR_GRP_CD
For char columns , even i dont give any datatype, i think it will work.
And pblm is not with this hopefully.
Thanks,
Kishore

SQL Loader : Loading multiple tables using same ctl file

Hi ,
We tried loading multiple tables using the same ctl file but the data was not loaded and no errors were thrown.
The ctl file content is summarised below :
LOAD DATA
APPEND INTO TABLE TABLE_ONE
when record_type ='EVENT'
TRAILING NULLCOLS
record_type char TERMINATED BY ',' ,
EVENT_SOURCE_FIELD CHAR TERMINATED BY ',' ENCLOSED BY '"',
EVENT_DATE DATE "YYYY-MM-DD HH24:MI:SS" TERMINATED BY ',' ENCLOSED BY '"',
EVENT_COST INTEGER EXTERNAL TERMINATED BY ',' ENCLOSED BY '"',
EVENT_ATTRIB_1 CHAR TERMINATED BY ',' ENCLOSED BY '"',
VAT_STATUS INTEGER EXTERNAL TERMINATED BY ',' ENCLOSED BY '"',
ACCOUNT_REFERENCE CONSTANT 'XXX',
bill_date "to_date('02-'||to_char(sysdate,'mm-yyyy'),'dd-mm-yyyy')",
data_date "trunc(sysdate)",
load_date_time "sysdate"
INTO TABLE TABLE_TWO
when record_type ='BILLSUMMARYRECORD'
TRAILING NULLCOLS
RECORD_TYPE char TERMINATED BY ',' ,
NET_TOTAL INTEGER EXTERNAL TERMINATED BY ',' ENCLOSED BY '"',
LOAD_DATE_TIME "sysdate"
INTO TABLE BILL_BKP_ADJUSTMENTS
when record_type ='ADJUSTMENTS'
TRAILING NULLCOLS
RECORD_TYPE char TERMINATED BY ',' ,
ADJUSTMENT_NAME CHAR TERMINATED BY ',' ENCLOSED BY '"',
LOAD_DATE_TIME "sysdate"
INTO TABLE BILL_BKP_CUSTOMERRECORD
when record_type ='CUSTOMERRECORD'
TRAILING NULLCOLS
RECORD_TYPE char TERMINATED BY ',' ,
GENEVA_CUSTOMER_REF CHAR TERMINATED BY ',' ENCLOSED BY '"',
LOAD_DATE_TIME "sysdate"
INTO TABLE TABLE_THREE
when record_type ='PRODUCTCHARGE'
TRAILING NULLCOLS
RECORD_TYPE char TERMINATED BY ',' ,
PROD_ATTRIB_1_CHRG_DESC CHAR TERMINATED BY ',' ENCLOSED BY '"',
LOAD_DATE_TIME "sysdate"
Has anyone faced similar errors or are we going wrong somewhere ?
Regards,
Sandipan

This is the info on the discard in the log file :
Record 1: Discarded - failed all WHEN clauses.
Record 638864: Discarded - failed all WHEN clauses.
While some of the records were loaded for one table.
Regards,
Sandipan

SQL Loader combine fields into one.

Hi i am using sql loader to convert a db from sql server to oracle.
my SQL Server DB has a field for the date and another for the time, i want to convert this into on filed called date. I cannot seem to figure out how to do this.
my ctl file looks like this.
load data
infile '[Cell_Phones].[dbo].[Account_Details].dat' "str '<EORD>'"
into table MD_CELLPHONE.Account_Details
fields terminated by '<EOFD>'
trailing nullcols
Account_Number ,
Phone_Number ,
Call_temp FILLER,
Call_Date "TO_DATE(:Call_temp ||' ' ||':Call_Date', 'mm/dd/yyyy HH:miam')"
BEGINDATA
3817913.0<EOFD>1234567890.0<EOFD>2007-03-31 00:00:00<EOFD>4:25 PM<EOFD>3817913.0<EOFD>1234567890.0<EOFD>2007-03-24 00:00:00<EOFD>8:19 PM<EOFD>3817913.0<EOFD>1234567890.0<EOFD>2007-03-31 00:00:00<EOFD>4:25 PM<EOFD>
But if i run this i get the following error
SQL*Loader-291: Invalid bind variable CALL_TEMP in SQL string for column CALL_DATE.
basically i cant find a way to do this to combine 2 columns into one and discard the one i am not using.
Any help would be appreciated.
Jeff

Hello,
You can load the data as in a stg table and the move from stg table to regular table concatenating both call_temp and call_date column together.
load data
infile 'Cell_Phones.dbo.Account_Details.dat' "str '<EORD>'"
into table MD_CELLPHONE.Account_Details_stg
fields terminated by '<EOFD>'
trailing nullcols
Account_Number ,
Phone_Number ,
Call_temp,
Call_Date
Convert to right date format
Insert into account_details (col1 , col2, col3, ...) select val1 , val2 , to_date ( val3 || ' ' || val4) ...
from account_details_stg;
Regards
Edited by: OrionNet on Feb 2, 2009 5:11 PM

Sql loader shell scripting

Hi All,
I need help on sql loader shell scripting basically since my input file is csv file will load into some target table say xx_90 along defining with control and data file. what where the data that goes into the bad file a segment of scripting that keep watching on badfile directory path if any bad file it finds it should move to a sperate database error table.
Plz any sqlloader shellscripting experts can provide me the solution for this. Its a very urgent code i need to deliver I need the sample code for this your help is highy appreciated
thanku
sahana

thanku..............when the load fails it moves to bad file script something like filewatcher it sees the data in the badfile and loads into the database error table........as i am basically businessobjects developer as for time being they have assinged me the task i did few manual loading on sql loader but i never have any experince on writing unix scripting i am quite worried to get this script done........
can i use the samecode for the failed bad records into seperate external table......
here i would change the path location and the rest goes as it isright............................i have the script for loading the datainto the targetsomething like this:
sqlldr userid=ops$NGPTQMS/[email protected] \
control=${evncpaq1/apps/cp/cm/r11.5.10/geae/gl/include/PAY0001_test.ctl} \
data=${evncpaq1/data/infile/mon_gross_det.dat} \
log=${evncpaq1/data/infile/mon_gross_det.log} \
discard=${evncpaq1/data/infile/mon_gross_det.dis} \
errors=999
#bad=${BAD_FILE} \
if [ $? -ne 0 ]; then
echo "Error ! The sql loader call failed "
exit 1
fi
IF any error is there in the script plz modify it and plz provide the complete code where to incorporate the code which you have written for bad file. Would be set it in seqence the steps so that i can run it in cron tab basically i have no other support for me to assist me on the job.......
my dba has sent me the mail something like this:
Please see the attached script to archive the source file once we are done with loading the data to the database. You already have the script for loading the data to the database. Please incorporate the attached script with your loading script. This script is generic and can be used to archive the bad file also once it has been loaded into the database (in case of any failures). This script send an email in case of any failures to the defined email distribution list. Right now the assumption is to run this through the cron. It we need to run this through the Appworx I will have to do some changes in the script. Also we will have to install the appworx agent on the server. Please let me if we need to need to run this through appworx I will create the appworx module for the same.
and the archive script he sent is:
. $HOME/.profile
# Script to Archive SysLog files
# This script accepts 2 parameters
# - 1: Script Name 2: File Name
# Created By : Pravin Darbare
# Creation Date : 03/19/2008
# Modified Date :
# Modification History :
echo "Starting SysLog_archiving Process"
# Set Environmental Variables
echo "Invoke Set Variable Script `date '+%H%M%S'`"
# Set Local Script Variables
#SCRIPT_NAME=$1
SOURCE_DIR=$1
FILE_NAME=$2
# Zipping and Archive Source File
echo "Zip Source file `date '+%H%M%S'`"
#cp $LOAD_SRC_FILES/$SOURCE_DIR/$FILE_NAME $LOAD_SRC_FILES/syslog/Old_$FILE_NAME
gzip /$SOURCE_DIR/$FILE_NAME
check_rtn_status=$?
if [ $check_rtn_status != 0 ]; then
echo "Zipping source file Syslog process unsuccessfull"
mailx -s "Zipping source file during Syslog unsuccessfull" $PR_PAGER_EMAIL_LIST </dev/null
exit 1
fi
echo "Archive Source File `date '+%H%M%S'`"
mv /$SOURCE_DIR/$FILE_NAME.gz /$SOURCE_DIR/archive/$FILE_NAME.gz$FILE_EXT
check_rtn_status=$?
if [ $check_rtn_status != 0 ]; then
echo "Archiving source file during Syslog load process unsuccessfull"
mailx -s "Archiving source file during Syslog load process unsuccessfull" $PR_PAGER_EMAIL_LIST </dev/null
exit 1
fi
# example of calling file Syslog_archiving_Files.sh /apps/syslog/source_dir a.txt
# example of calling file (Bad file) Syslog_archiving_Files.sh /apps/syslog/source_dir_bad a.txt
# You need to have directory "archive" in both pathes .
based on the complete detailed mail must having some view plz provide me something like exact code the rest path directories i will change it........
for me its highly urgent to keep up my job
Your support is highly appreciated
Thanks&Regards
sahana

SQL*Loader - How to load only a few columns from a .csv file to ora table

Hi there,
Could anyone please let me know how to load few columns from a .csv
file into a oracle table using SQL*Loader.
I know how to create a .dat and .ctl file and run the sql loader.
Suppose I have a .csv file with
col1, col2, col3, col4
and I only need to load col1 and col3 into col_a and col_b respectively
in table_a?
structure of table_ a
col_a,
col_b
Please advice

Try like..it i will work..
LOAD DATA
INFILE 'test.txt'
LOAD DATA
TRUNCATE INTO TABLE T1
FIELDS TERMINATED BY ','
(col1,
col2 FILLER,
col3,
col4 FILLER
)

SQL LOADER / INFILE filename as variable in .ctl file

Similar Messages

Maybe you are looking for