External table filenames

I am loading hundreds/thousands of files of delimited data via external tables. The record counts for verifying the data is being provided on a file by file level from the mainframe system sending the data, example file prn00532.dat has 78,322 records.
I can add all the files to the create external table DDL no problem, but I then can not verify that each file that is loaded has the correct number of records as the files all get loaded as a group.
My question is, without having to create an table for each external file, is there a way to have a calculated column that oracle adds, or a bind or something I can reference to insert into a another table/same table that shows the external data file that oracle is currently loading?
Basically when oracle switches from file prn00532.dat to prn00533.dat I can capture that...

Instead of disturbing parallel coordinators you could just submit n jobs each doing a simple <b>insert into big_table select * from external_table_n</b> or something similar - all external_tables_i having the same structure just different location parameters.
Regards
Etbin
Edited by: Etbin on 26.2.2011 12:25
possibly grouping text data files according to partitions the data would end within

Similar Messages

How to insert filename as column in external table.

I have an external table that is made up by number of different files. I would like to add a new column in the table to store the actual external filename. How can this be done?
thanks in advanced

That's going to be a problem, then. The ISO 8859-1 character set doesn't support that character, so you cannot store a ™ in a CHAR or VARCHAR2 column in your database.
Potentially, you could use an NCHAR or NVARCHAR2 column, depending on your national character set. Using the national character set, though, can cause other problems, particularly for front-end applications that may not fully support those data types.
You could also potentially change your existing character set to something that does support that character, but that may be a rather involved sort of change.
Justin

Variable filename external table

I have written a script to import data from a .csv into a table in Apex:
CREATE TABLE APEX_USER_002.INK_KASSA_LOAD
KASS_DTM      VARCHAR2(255),
       KASS_ID       VARCHAR2(255),
       KASS_VKP_COD VARCHAR2(255),
       KASS_VKP_OMS VARCHAR2(255),
       KASS_COD VARCHAR2(255),
       KASS_OMS   VARCHAR2(255),
       KASS_PRS_COD VARCHAR2(255),
       KASS_PRS_OMS VARCHAR2(255)
ORGANIZATION EXTERNAL
TYPE ORACLE_LOADER
DEFAULT DIRECTORY AFM_KASSA_IN
ACCESS PARAMETERS
    RECORDS DELIMITED BY NEWLINE CHARACTERSET WE8MSWIN1252
      LOGFILE AFM_KASSA_LOG: 'log.log'
    BADFILE AFM_KASSA_BAD: 'badfile.bad'
    NODISCARDFILE
    SKIP 1
    FIELDS TERMINATED BY ';'
    OPTIONALLY ENCLOSED BY '"'
    LRTRIM
    MISSING FIELD VALUES ARE NULL
KASS_DTM,
       KASS_ID,
       KASS_VKP_COD,
       KASS_VKP_OMS,
       KASS_COD,
       KASS_OMS,
       KASS_PRS_COD,
       KASS_PRS_OMS
    LOCATION ('variable_filename.csv')
It works fine when I put the name of the .csv hardcoded in the script. Like this: LOCATION ('Kassa_20130406130014.csv')
I have written a function which returns the filename, but when I use the function in LOCATION (function()) I get an error: ora-00905: missing keyword
How can I call a function in LOCATION ()?

Hi,
I'm not sure but I think you need re-create external table.
Create procedure that drops and re-creates external table dynamically with new file name.
Or use e.g. DBMS_SCHEDULER to call OS script that renames file always to same name for external table.
Regards,
Jari

External table - fetch location ?

Using Oracle 10.2.0.5
An external table is a construct that gives me SQL access to a file.
Is it possible to know the name of the file somehow inside the select? Like Add a column with the file name?
pseudo example
CREATE TABLE EXT_DUMMY
    "RECORDTYPE" VARCHAR2(100 BYTE),
    "COL1" VARCHAR2(100 BYTE),
    "COL2" VARCHAR2(100 BYTE),
    "FILE" VARCHAR2(100 BYTE)
ORGANIZATION EXTERNAL
    TYPE ORACLE_LOADER DEFAULT DIRECTORY "IMPORT_BAD_FILE"
    ACCESS PARAMETERS (
             records delimited BY newline
             FIELDS TERMINATED BY ';'
             MISSING FIELD VALUES ARE NULL
               ( RECORDTYPE CHAR
               , COL1 CHAR
               , COL2 CHAR
               , FILE CHAR FILLER
    LOCATION ( 'Testfile1.txt, Testfile2.txt' )
    reject limit 10
;The result could look like this:
RECORDTYPE   COL1       COL2      FILE
SAMPLE           DUMMY    DUMMY Testfile1.txt
SAMPLE           DUMMY1   DUMMY Testfile1.txt
SAMPLE           DUMMY2   DUMMY Testfile1.txt
SAMPLE           DUMMY3   DUMMY Testfile1.txt
SAMPLE           DUMMY1   DUMMY1 Testfile2.txt
SAMPLE           DUMMY1   DUMMY2 Testfile2.txt
SAMPLE           DUMMY2   DUMMY1 Testfile2.txtI would like to know from which file a certain row is read. Maybe I missed an option in the documentation. In this example I have two different files as the source for the external table.
Another use case could be this:
If I enable a user to switch the external table to a different file alter table EXT_DUMMY location ('Testfile3.txt' ). How can we know which file is read during the select on the table? When userA does the select, maybe userB just altered the location before the select was started. Therefore userA would read in a different file then expected.
Edited by: Sven W. on May 26, 2011 4:48 PM
Edited by: Sven W. on May 26, 2011 4:51 PM
Edited by: Sven W. on May 26, 2011 5:11 PM

Hi Sven,
I'm not sure how much we can rely on this, but let's consider the following :
create table test_xt (
rec_id number
, message varchar2(100)
organization external (
default directory test_dir
access parameters (
    records delimited by newline
    fields terminated by ';'
location (
    'marc5.txt'
, 'test1.csv'
, 'test2.csv'
, 'test3.csv'
);I always thought the ROWID doesn't hold much sense for an external table, but...
SQL> select t.rowid
2       , dump(t.rowid) as rowid_dump
3       , regexp_substr(dump(t.rowid,10,9,1),'\d+$') as file#
4       , t.*
5 from test_xt t
6 ;
ROWID              ROWID_DUMP                                                FILE#      REC_ID MESSAGE
(AADVyAAAAAAAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,0,0,0,0,0,0,0,0,0     0               1 this is a line from marc5.txt
(AADVyAAAAAAAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,0,0,0,0,0,0,0,0,33    0               2 this is a line from marc5.txt
(AADVyAAAAAAAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,0,0,0,0,0,0,0,0,66    0               3 this is a line from marc5.txt
(AADVyAAAAAAAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,0,0,0,0,0,0,0,0,99    0               4 this is a line from marc5.txt
(AADVyAAAAAEAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,1,0,0,0,0,0,0,0,0     1               1 this is a line from test1.csv
(AADVyAAAAAEAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,1,0,0,0,0,0,0,0,33    1               2 this is a line from test1.csv
(AADVyAAAAAEAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,1,0,0,0,0,0,0,0,66    1               3 this is a line from test1.csv
(AADVyAAAAAEAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,1,0,0,0,0,0,0,0,99    1               4 this is a line from test1.csv
(AADVyAAAAAIAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,2,0,0,0,0,0,0,0,0     2               1 this is a line from test2.csv
(AADVyAAAAAIAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,2,0,0,0,0,0,0,0,33    2               2 this is a line from test2.csv
(AADVyAAAAAIAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,2,0,0,0,0,0,0,0,66    2               3 this is a line from test2.csv
(AADVyAAAAAMAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,3,0,0,0,0,0,0,0,0     3               1 this is a line from test3.csv
(AADVyAAAAAMAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,3,0,0,0,0,0,0,0,33    3               2 this is a line from test3.csv
(AADVyAAAAAMAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,3,0,0,0,0,0,0,0,66    3               3 this is a line from test3.csv
(AADVyAAAAAMAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,3,0,0,0,0,0,0,0,99    3               4 this is a line from test3.csv
(AADVyAAAAAMAAAAAA Typ=208 Len=17: 4,0,0,213,200,0,0,0,3,0,0,0,0,0,0,0,132   3               5 this is a line from test3.csv
16 rows selected
Then with a join to EXTERNAL_LOCATION$ :
SQL> with ext_loc as (
2    select position-1 as pos
3         , name as filename
4    from sys.external_location$
5    where obj# = ( select object_id
6                   from user_objects
7                   where object_name = 'TEST_XT' )
8 )
9 select x.filename,
10         t.*
11 from test_xt t
12       join ext_loc x on x.pos = to_number(regexp_substr(dump(t.rowid,10,9,1),'\d+$'))
13 ;
FILENAME       REC_ID MESSAGE
marc5.txt           1 this is a line from marc5.txt
marc5.txt           2 this is a line from marc5.txt
marc5.txt           3 this is a line from marc5.txt
marc5.txt           4 this is a line from marc5.txt
test1.csv           1 this is a line from test1.csv
test1.csv           2 this is a line from test1.csv
test1.csv           3 this is a line from test1.csv
test1.csv           4 this is a line from test1.csv
test2.csv           1 this is a line from test2.csv
test2.csv           2 this is a line from test2.csv
test2.csv           3 this is a line from test2.csv
test3.csv           1 this is a line from test3.csv
test3.csv           2 this is a line from test3.csv
test3.csv           3 this is a line from test3.csv
test3.csv           4 this is a line from test3.csv
test3.csv           5 this is a line from test3.csv
Seems to work... assuming the files are always read in the order specified through the LOCATION parameter, and the generated ROWID actually means what I think it means.

Error when loading from External Tables in OWB 11g

Hi,
I face a strange problem while loading data from flat file into the External Tables.
ORA-12899: value too large for column EXPIRED (actual: 4, maximum: 1)
error processing column EXPIRED in row 9680 for datafile <data file location>/filename.dat
In a total of 9771 records nearly 70 records are rejected due to the above mentioned error. The column (EXPIRED) where the error being reported doesn't have value greater than 1 at all. I suspect it to be a different problem.
Example: One such record that got rejected is as follows:
C|234|Littérature commentée|*N*|2354|123
highlightened in Bold is the EXPIRED Column.
When I tried to insert this record into the External Table using UTL_FILE Utility it got loaded successfully. But when I try with the file already existing in the file directory it again fails with the above error, and I would like to mention that all the records which have been loaded are not Ok, please have a look at the DESCRIPTION Column which is highlightened. The original information in the data file looks like:
C|325|*Revue Générale*|N|2445|132
In the External Table the Description Value is replaced by the inverted '?' as follows:
Reue G¿rale
Please help.
Thanks,
JL.

user1130292 wrote:
Hi,
I face a strange problem while loading data from flat file into the External Tables.
ORA-12899: value too large for column EXPIRED (actual: 4, maximum: 1)
error processing column EXPIRED in row 9680 for datafile <data file location>/filename.dat
In a total of 9771 records nearly 70 records are rejected due to the above mentioned error. The column (EXPIRED) where the error being reported doesn't have value greater than 1 at all. I suspect it to be a different problem.
Example: One such record that got rejected is as follows:
C|234|Littérature commentée|*N*|2354|123
highlightened in Bold is the EXPIRED Column.
When I tried to insert this record into the External Table using UTL_FILE Utility it got loaded successfully. But when I try with the file already existing in the file directory it again fails with the above error, and I would like to mention that all the records which have been loaded are not Ok, please have a look at the DESCRIPTION Column which is highlightened. The original information in the data file looks like:
C|325|*Revue Générale*|N|2445|132
In the External Table the Description Value is replaced by the inverted '?' as follows:
Reue G¿rale
Please help.
Thanks,
JL.sorry, couldnt see the highlighted test.could you plesae enclsoe it in tags
also post the table definition with attributes. BTW..Whats your NLS_LANGUAGE SET TO?

Comparison of Data Loading techniques - Sql Loader & External Tables

Below are 2 techniques using which the data can be loaded from Flat files to oracle tables.
1)     SQL Loader:
a.     Place the flat file( .txt or .csv) on the desired Location.
b.     Create a control file
Load Data
Infile "Mytextfile.txt" (-- file containing table data , specify paths correctly, it could be .csv as well)
Append or Truncate (-- based on requirement) into oracle tablename
Separated by "," (or the delimiter we use in input file) optionally enclosed by
(Field1, field2, field3 etc)
c.     Now run sqlldr utility of oracle on sql command prompt as
sqlldr username/password .CTL filename
d.     The data can be verified by selecting the data from the table.
Select * from oracle_table;
2)     External Table:
a.     Place the flat file (.txt or .csv) on the desired location.
abc.csv
1,one,first
2,two,second
3,three,third
4,four,fourth
b.     Create a directory
create or replace directory ext_dir as '/home/rene/ext_dir'; -- path where the source file is kept
c.     After granting appropriate permissions to the user, we can create external table like below.
create table ext_table_csv (
i Number,
n Varchar2(20),
m Varchar2(20)
organization external (
type oracle_loader
default directory ext_dir
access parameters (
records delimited by newline
fields terminated by ','
missing field values are null
location ('file.csv')
reject limit unlimited;
d.     Verify data by selecting it from the external table now
select * from ext_table_csv;
External tables feature is a complement to existing SQL*Loader functionality.
It allows you to –
•     Access data in external sources as if it were in a table in the database.
•     Merge a flat file with an existing table in one statement.
•     Sort a flat file on the way into a table you want compressed nicely
•     Do a parallel direct path load -- without splitting up the input file, writing
Shortcomings:
•     External tables are read-only.
•     No data manipulation language (DML) operations or index creation is allowed on an external table.
Using Sql Loader You can –
•     Load the data from a stored procedure or trigger (insert is not sqlldr)
•     Do multi-table inserts
•     Flow the data through a pipelined plsql function for cleansing/transformation
Comparison for data loading
To make the loading operation faster, the degree of parallelism can be set to any number, e.g 4
So, when you created the external table, the database will divide the file to be read by four processes running in parallel. This parallelism happens automatically, with no additional effort on your part, and is really quite convenient. To parallelize this load using SQL*Loader, you would have had to manually divide your input file into multiple smaller files.
Conclusion:
SQL*Loader may be the better choice in data loading situations that require additional indexing of the staging table. However, we can always copy the data from external tables to Oracle Tables using DB links.

Please let me know your views on this.

Showing Text file as external table without trimming spaces

Hi all:
I am trying to display a text file from db sever with original spacing.
Well it's not entirely related to APEX.
This is how external table is created:
create table u_x
(rowline varchar2(255))
     ORGANIZATION EXTERNAL
       TYPE ORACLE_LOADER
       DEFAULT DIRECTORY ext_tab         <-- this is a dir i have created
       access parameters
       records delimited by newline
       fields notrim
       LOCATION ('aaa.txt')
REJECT LIMIT UNLIMITED;
on the first line of the aaa.text (its a report), it shows something like:
Sat Dec 29                                                                Page 1
but when I do select * from u_x , the result shows:
Sat Dec 29 Page 1
all above is done in sql workshop /sql command in APEX 4.2.1 (db 10.2.0.5)
Is there any way to preserve the spaces between "29" and "Page"?
(the result compressed all spaces in-between to 1 space).
I think the ideal way is to invoke client's notepad to open the server's text file.
anyway to generate the link?
Thanks
John

Well, I have tried to download the file using below procedure.
create or replace PROCEDURE download_bfile (
      directory_in   IN   VARCHAR2,
      file_in        IN   VARCHAR2
   AS
      lob_loc    BFILE;
      v_mime     VARCHAR2 (48);
      v_length   NUMBER(16);
   BEGIN
      lob_loc := BFILENAME (UPPER (directory_in), file_in);
      v_length := DBMS_LOB.getlength (lob_loc);
      -- set up HTTP header
      -- use an NVL around the mime type and
      -- if it is a null set it to application/octect
      -- application/octect may launch a download window from windows
      OWA_UTIL.mime_header (nvl(v_mime, 'application/octet'), FALSE);
      -- set the size so the browser knows how much to download
      HTP.p ('Content-length: ' || v_length);
      -- the filename will be used by the browser if the users does a save as
      HTP.p (   'Content-Disposition:attachment; filename="'
             || SUBSTR (file_in, INSTR (file_in, '/') + 1)
             || '";'
      -- close the headers
      OWA_UTIL.http_header_close;
      -- download the BLOB
      WPG_DOCLOAD.download_file (lob_loc);
      apex_application.stop_apex_engine;
   END download_bfile;
I run it inside sql command in apex, but it does not show a pop window to save the file,
it displays the whole text content of the file in the result window.
Any idea why the save-as window does not pop up?
thanks,
John.

Doubt in external table.

I want to read the csv file and load into oracle table.But I am getting file with filename_<today date> for every day.
Is it possible to use single External table to read file in dynamic.
or what is the best way to do this?
My oracle version 10g in windows OS.
Thanks in advance,

>
I want to read the csv file and load into oracle table.But I am getting file with filename_<today date> for every day.
Is it possible to use single External table to read file in dynamic.
>
Yes - it is possible. You need to issue an ALTER TABLE command to provide the new filename before you SELECT from the table.
alter table myExternalTable location ('myNewFileName.ext'); See the Location of Datafiles and Output Files section of Chapter 12 External Tables Concepts in the Utilities doc
http://docs.oracle.com/cd/B28359_01/server.111/b28319/et_concepts.htm#i1009094
Also see the Altering External Tables section of Managing External Tables in the DBA Guide
http://docs.oracle.com/cd/E14072_01/server.112/e10595/tables013.htm
That sections has examples for all of the ALTER statements you can use. For example you can also alter the DIRECTORY object being used.
ALTER TABLE admin_ext_employees
DEFAULT DIRECTORY admin_dat2_dir;

How to overwrite a log and bad file in external table in oracle 10g

Hi,
I have used external table in oracle 10g.whenever use select query in external table orace internally create one log file in specified directory,
but this log file is growing.How can i overwrite the log file(old to replace with new).I need overwrite a log and bad file in external table.
kindly give the solutions.
By
Siva

I don't believe that is possible with the LOGFILE clause, but it may be with the BADFILE clause. Here is an excerpt from the documentation :
The LOGFILE clause names the file that contains messages generated by the external tables utility while it was accessing data in the datafile. If a log file already exists by the same name, the access driver reopens that log file and appends new log information to the end. This is different from bad files and discard files, which overwrite any existing file. NOLOGFILE is used to prevent creation of a log file.
If you specify LOGFILE, you must specify a filename or you will receive an error.
If neither LOGFILE nor NOLOGFILE is specified, the default is to create a log file. The name of the file will be the table name followed by _%p and it will have an extension of .log.

SQL Loader versus External Table

If anyone has worked on external tables please let me know your views.

user637544 wrote:
for sqlldr i follow the following approach
1. truncate table
2. call sqlldr and load records in staging table
3. check for bad records
how do i tuncate, call external table and check bad records for external table.As part of the SQL*Loader control file you can tell it to truncate the table as part of the process before it loads in the data.
The key differences between SQL*Loader and External Tables are:
1. SQL*Loader is an external utility run from the o/s command line, whereas External tables are usable from within SQL and PL/SQL.
2. SQL*Loader can only load data up into the database, whereas External tables can allow you to read files and write files (10g onwards)
3. SQL*Loader has limited control on skipping rows etc. whereas External tables can be queried using SQL and has all the flexibility that SQL can offer.
4. SQL*Loader requires connection information to the database, whereas External tables are already part of the database
5. SQL*Loader can only change the source filename through dynamic o/s level scripts, whereas External tables can have the filename changed using an ALTER TABLE ... LOCATION ... command.
If you want to truncate your staging table before loading it with data from the external table, then you will just have to do that as a seperate step, because the external table is not associated with other tables. Simply truncate the other table and then query the data from the external table into the staging table.
External tables offer a lot more flexibility than using SQL*Loader.

How to use external table - creating NFS mount -the details involved

Hi,
We are using Oracle 10.2.0.3 on Solaris 10. I want to use external tables to load huge csv data into the database. This concept was tested and also found to be working fine. But my doubt that : since ours is a J2EE application, the csv files have to come from the front end- from the app server. So in this case how to move them to the db server?
For my testing I just used putty to transfer the file to db server, than ran the dos2unix command to strip off the control character at the end of file. but since this is to be done from the app server, putty can not be used. In this case how can this be done? Are there any risks or security issues involved in this process?
Regards

orausern wrote:
For my testing I just used putty to transfer the file to db server, than ran the dos2unix command to strip off the control character at the end of file. but since this is to be done from the app server, putty can not be used. In this case how can this be done? Are there any risks or security issues involved in this process? Not sure why "putty" cannot be used. This s/w uses the standard telnet and ssh protocols. Why would it not work?
As for getting the files from the app server to the db server. There are a number of options.
You can look at it from an o/s replication level. The command rdist is common on most (if not all) Unix/Linux flavours and used for remote distribution and sync'ing of files and directories. It also supports scp as the underlying protocol (instead of the older rcp protocol).
You can use file sharing - the typical Unix approach would be to use NFS. Samba is also an option if NTLM (Windows) is already used in the organisation and you want to hook this into your existing security infrastructure (e.g. using Microsoft's Active Directory).
You can use a cluster file system - a file system that resides on shared storage and can be used by by both app and db servers as a mounted/cooked file system. Cluster file systems like ACFS, OCFS2 and GFS exist for Linux.
You can go for a pull method - where the db server on client instruction (that provides the file details), connects to the app server (using scp/sftp/ftp), copy that file from the app server, and then proceed to load it. You can even add a compression feature to this - so that the db server copies a zipped file from the app server and then unzip it for loading.
Security issues. Well, if the internals is not exposed then security will not be a problem. For example, defining a trusted connection between app server ad db server - so the client instruction does not have to contain any authentication data. Letting the client instruction only specify the filename and have the internal code use a standard and fixed directory structure. That way the client cannot instruct something like +/etc/shadow+ be copied from the app server and loaded into the db sever as a data file. Etc.

How do you set up csv as an oracle external table

How do you set up csv as an oracle external table?

Hi,
>>where do I specify which csv file to read from the external table..
Please, read either the link provided by Andrew or the link that I have posted in my previous post.
"# DEFAULT DIRECTORY - specifies the default location of files that are read or written by external tables. The location is specified with a directory object, not a directory path. See Location of Datafiles and Output Files for more information.
# LOCATION - specifies the location of the external data. The location is specified as a list of directory objects and filenames. If the directory object is not specified, then the default directory object is used as the file location."
Cheers
Legatti

External table and XML

Hi.
I have this XML-structure:
<order>
<orderhead>
<orderno>123</orderno>
<orderline>
<name>name_of_what_i_order</name>
</orderline>
<orderline>
<name>name_of_what_i_order</name>
</orderline>
</orderhead>
</order>
How can I get the <orderhead><orderno> when I'm making an external table for the <ordeline> elements? I use 'records delimited by "</orderline>"' to get all the orderlines in one tablerow each. But how can I insert the orderno in each of these rows?

Hi,
You need to practice or test by youself. Fortunately, I have a posting at http://www.oraclepoint.com/topic.php?filename=98&extra=page%3D1, which is a similar case I've done couple of years ago.
What I did are:
1. formating XML
2. Load to oracle table by using SQL*Loader
3. Parse elements one by one from entire xml large object from within oracle table
4. insert the parsed elements to oracle table
Hope it helps. By the way, please register and then log in at oraclepoint.com to get the solution package becuase it's only available for members.
R.Wang
http://www.oraclepoint.com

External Table Preprocessor

Dear Experts,
I was trying to explore the "preprocessor" command in external table, but I was not able to succeed.
I just followed the steps mentioned in, latest Oracle Magazine.
http://www.oracle.com/technetwork/issue-archive/2012/12-nov/o62asktom-1867739.html
But finally when I do the select from table df, I am getting the following error.
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-29400: data cartridge error
KUP-04095: preprocessor command /export/csrmgr/ripple_dms/run_df.sh encountered error "error during exec: errno is 2
PS: The output of v$version;
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
"CORE     11.2.0.2.0     Production"
TNS for IBM/AIX RISC System/6000: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production
Could anyone help me to resolve this issue.

I'm not able to test Tom's example at the minute.
However, that new feature was discussed in the March/April 2011 edition of Oracle Magazine too.
Oracle Magazine March/April 2011 article on new feature in 11gR2 called the external table preprocessor.
by Arup Nanda
CHANGE IN PROCESS
Now suppose that with your external table and its text file in place, the input file for the external table is compressed to reduce the volume of data transmitted across the network. Although compression helps the network bandwidth utilization, it creates a challenge for the ETL process. The file must be uncompressed before its contents can be accessed by the external table.
Rather than uncompress the file in a separate process, you can use the preprocessor feature of Oracle Database 11g Release 2 with external tables to uncompress the file inline. And you will notneed to change the ETL process.
To use the preprocessor feature, first you need to create a preprocessor program. The external table expects input in a text format but not necessarily in a file. The external table does not need to read a file; rather, it expects to get the file contents “fed” to it. So the preprocessor program must stream the input data directly to the external table—and not
create another input file. The input to the pre-processor will be a compressed file, and the output will be the uncompressed contents.
The following is the code for your new preprocessor program, named preprocess.bat:
@echo off
C:\oracle\product\11.2.0\dbhome_1\BIN\unzip.exe -qc %1The first line, @echo off, suppresses the output of the command in a Windows environment. The remaining code calls the unzip.exe utility located in the Oracle home. The utility needs an input file, which is the first (and only) parameter passed to it, shown as %1. The options q and c tell the utility to uncompress quietly (q) without producing extraneous output such as “Inflating” or “%age inflated” and match filenames case-insensitively (c), respectively.
Next you need to create the directory object where this preprocessor program is located. Logging in as SYS, issue
create directory execdir as 'c:\tools';And now grant EXECUTE permissions on the directory to the ETLADMIN user:
grant execute on directory execdir to etladmin;Finally, create the new external table:
create table indata1
cust_id number,
cust_name varchar2(20),
credit_limit number(10)
organization external
type oracle_loader
default directory etl_dir
access parameters
    records delimited by newline
    preprocessor execdir:'preprocess.bat'
    fields terminated by ","
location ('indata1.txt')
/It calls the preprocess.bat executable in the directory specified by EXECDIR before the external table accesses the indata1.txt file in the location specified by the ETL_DIR directory. Remember, indata1.txt is now a compressed file. So, in effect, the external table reads not the actual specified input file but rather the output of preprocess.bat, which is the uncompressed data from the indata1.txt file.
If you select from the external table now, the output will be similar to that of the earlier select * from indata1; query. The preprocessor passed the uncompressed contents of the indata1.txt (compressed) file on to the external table. There was no need to uncompress the file first—saving significant time and the intermediate space required and making it unnecessary to change the ETL process.
This inline preprocessing unzip example uses a script, but that is not always necessary. An executable can be used instead. For example, in Linux you can use /bin/gunzip. However, the utility can’t accept any parameters. So if you pass parameters (as in this article’s example), you must use a script.
Security Concerns
=================
The EXECUTE privilege on a directory is a new feature introduced in Oracle Database 11g Release 2. It enables the DBA to grant EXECUTE permissions only for certain directories and only to certain users. Without WRITE privileges, users will not be able to
update the executables inside a directory to insert malicious code, but users will be able to execute the “approved” code accessible in a single location. The DBA can put all the necessary preprocessor tools into a single directory and grant EXECUTE privileges there to the users who may need them. And, of course, the executables and data should be in different directories.
Preprocessing also requires some special precautions on the part of the DBA. Because the executables called by preprocessing programs will be executed under the privileges of the Oracle software owner and malicious executable code can cause a lot of damage, the DBA should be extremely careful in monitoring executables for potentially harmful code.
The directory containing the preprocessor executables needs to be accessible to the Oracle software owner for EXECUTE operations only, not for WRITE activity. Therefore, as an added precaution, the system administrator can remove WRITE access to that directory from all users, including the Oracle software owner. This significantly reduces the chance of damage by malicious code.
Other Uses
==========
Compression is not the only use for inline preprocessing, although it certainly is the most widely used. You can, for example, use this preprocessing technique to show the output of a program as an external table. Consider, for instance, the dir command in Windows for listing the contents of a directory. How would you like to get the output as a table so that you can apply predicates?
Getting and using this output is quite simple with the preprocessor functionality. Remember, the preprocessor does not actually need a file but, rather, requires the output of the preprocessor program. You can write a preprocessor program to send the
output of a dir command. The new preprocessor program, named preproc_dir.bat, has only the following two lines:
@echo off
dirYou will also need a file for the external table. The contents of the file are irrelevant, so you can use any file that the Oracle software owner can read in a directory to which that owner has read access. For this example, the file is dirfile.txt, and although the contents of the file are immaterial, the file must exist, because the external table will access it. Listing 1 shows how to create the table.
Because the dir command displays output in a prespecified manner, the external table easily parses it by reading the fields located in specific positions. For example, positions 1 through 10 display the date, 11 through 20 display the time, and so on. The dir command produces some heading and preliminary information that the external table has to ignore, so there is a skip 5 clause in Listing 1 that skips the first five lines of the output. The last few lines of the output show how many files and directories are present and how much free space remains. This output must be skipped as well, so the external table displays records only when the date column has a value.
Listing 1 also shows the result of a query against the external table. Because the MOD_DT column is of the date datatype, you can also apply a WHERE condition to select a specified set of records.
Code Listing 1:
===============
create table dir_tab
mod_dt date,
mod_time char(10),
file_type char(10),
file_size char(10),
file_name char(40)
organization external
type oracle_loader
default directory etl_dir
access parameters
    records delimited by newline
    preprocessor execdir:'preproc_dir.bat'
    skip 5
    load when (mod_dt != blanks)
    fields
      mod_dt position (01:10) DATE mask "mm/dd/yyyy",
      mod_time position (11:20),
      file_type position (21:29),
      file_size position (30:38),
      file_name position (39:80)
location ('dirfile.txt')
reject limit unlimited
-- select from this table
SQL> select * from dir_tab;
MOD_DT        MOD_TIME       FILE_TYPE       FILE_SIZE      FILE_NAME
16-DEC-10     10:12 AM       <DIR> .
16-DEC-10     10:12 AM       <DIR> ..
22-MAY-10     09:57 PM       <DIR> archive
22-MAY-10     10:27 PM                                2,048 hc_alap112.dat
05-DEC-10     07:07 PM                                   36 indata1.txt
22-DEC-05     04:07 AM                               31,744 oradba.exe
16-DEC-10     09:58 AM                                1,123 oradim.log
28-SEP-10     12:41 PM                                1,536 PWDALAP112.ora
16-DEC-10     09:58 AM                                2,560 SPFILEALAP112.ORA
9 rows selected.
-- select a file not updated in last 1 year
SQL> select * from dir_tab where mod_dt < sysdate - 365;
MOD_DT        MOD_TIME       FILE_TYPE       FILE_SIZE      FILE_NAME
22-DEC-05     04:07 AM                               31,744 oradba.exe

External table wont display

db=xe
Hello,
I am attempting to create an external table to reference a csv file that contains exif data from images. The exif data presents the date in the following form 2011:12:03 16:47:27
I have constructed the external table like this;
CREATE TABLE metadata_external
( sourcefile VARCHAR2 (255 CHAR)
, createdate VARCHAR2 (255 CHAR)
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "UPLOAD_DIR"
ACCESS PARAMETERS
    ( RECORDS DELIMITED BY NEWLINE
      NOBADFILE
      NOLOGFILE
      FIELDS TERMINATED BY ','
      MISSING FIELD VALUES ARE NULL
      ( sourcefile
      , createdate
LOCATION
    ( "UPLOAD_DIR":'metadata_external.csv'
) REJECT LIMIT UNLIMITED;You will note that I have given the createdate column a datatype of varchar2. This was to avoid any problems with formating so that I could use the to_date function when inserting into other tables.
But the following code will not display anything but throws the following error;
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-00554: error encountered while parsing access parameters
KUP-01005: syntax error: found "minussign": expecting one of: "binary_double, binary_float, comma, char, date, defaultif, decimal, double, float, integer, (, nullif, oracle_date, oracle_number, position, raw, recnum, ), unsigned, varrawc, varchar, varraw, varcharc, zoned"
KUP-01007: at line 8 column 1
ORA-06512: at "SYS.ORACLE_LOADER", line 19I suspect that the : separators in the date are causing the problems.
Would anyone have some suggestions to fix this problem?
Thank You
Ben

Hello Mahir,
Here is the csv contents;
SourceFile,CreateDate,DOF,ShutterSpeed,Aperture,ISO,MeteringMode,ExposureCompensation,FileName,LightValue
C:/temp/_DSC6473.NEF,2011:12:03 16:47:27,3.38 m (5.04 - 8.42),1/1600,5.0,100,Multi-segment,-1,_DSC6473.NEF,15.3
C:/temp/_DSC6474.NEF,2011:12:03 16:47:53,3.38 m (5.04 - 8.42),1/2000,5.0,100,Multi-segment,-1,_DSC6474.NEF,15.6
C:/temp/_DSC6475.NEF,2011:12:03 16:48:22,3.38 m (5.04 - 8.42),1/800,5.0,100,Multi-segment,-1,_DSC6475.NEF,14.3
C:/temp/_DSC6476.NEF,2011:12:03 16:48:32,3.38 m (5.04 - 8.42),1/1250,5.0,100,Multi-segment,-1,_DSC6476.NEF,14.9
C:/temp/_DSC6477.NEF,2011:12:03 16:48:57,3.38 m (5.04 - 8.42),1/500,5.0,100,Multi-segment,-1,_DSC6477.NEF,13.6Please note I have only referenced the first two columns in the ddl for the external table because the second column is continueing to return errors.
Thank You
Ben
Edited by: benton on Dec 9, 2011 7:36 AM

External table filenames

Similar Messages

Maybe you are looking for