How to read any file using external tables.

Question

Hi folks,
I have written an application that reads a series of csv files using external tables which works fine as long as I specify each file name in the directory i.e.......
CREATE TABLE gb_test
(file_name varchar2(10),
rec_date date
rec_name VARCHAR2(20),
rec_age number,
ORGANIZATION EXTERNAL
TYPE ORACLE_LOADER
DEFAULT DIRECTORY GB_TEST
ACCESS PARAMETERS
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
LOCATION ('data1.csv','data2.csv','data3.csv','data4.csv')
PARALLEL 5
REJECT LIMIT 20000;
However I have discovered that I may not know the name of the files to be processed prior to the program being run so just want to read any file regardless of it's name (although it will always be a .csv file).
Is there a way to ensure that you don't need to specify the files to be read in the LOCATION part of the syntax.
Thanks in advance.
Graham.

Answer 1

Right, I have now completed this, however it's currently only working as SYS as opposed to any user, however here is a detail of the scenario and the steps required in case any of you guys need in the future ......
The problem was I needed to search for csv files on my hard-drive. These files would be stored in a series of directories (a through to z), so I needed a way to read all 26 directories and process all files in these directories.
The problem was, prior to running the program, the user would remove all the files in the directories and insert new ones, but it was never known how many he would decide to do each time.
Solution: I created a table called stock_data_directories as follows ...
create table stock_data_directories(sdd_rec_no number,
sdd_table_name varchar2(50),
sdd_directory_name varchar2(50),
sdd_directory_path varchar2(100));
Then inserted 26 records like ...
insert into stock_data_directories(sdd_rec_no,sdd_table_name,sdd_directory_name,sdd_directory_path)
values(1,'rawdata_a','KPOLLOCKA','C:\KPOLLOCK\A')
insert into stock_data_directories(sdd_rec_no,sdd_table_name,sdd_directory_name,sdd_directory_path)
values(2,'rawdata_b','KPOLLOCKB','C:\KPOLLOCK\B');
etc...etc...
Then created 26 DIRECTORIES E.G.
CREATE OR REPLACE DIRECTORY KPOLLOCKA AS 'C:\KPOLLOCK\A';
CREATE OR REPLACE DIRECTORY KPOLLOCKB AS 'C:\KPOLLOCK\B';
Then created 26 external tables like the following ...
CREATE TABLE rawdata_a
(stock varchar2(1000),
stock_date varchar2(10),
stock_open VARCHAR2(20),
stock_high varchar2(20),
stock_low varchar2(20),
stock_close VARCHAR2(30),
stock_qty varchar2(20) )
ORGANIZATION EXTERNAL
TYPE ORACLE_LOADER
DEFAULT DIRECTORY KPOLLOCKA
ACCESS PARAMETERS
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
LOCATION ('AA.csv')
PARALLEL 5
REJECT LIMIT 20000
This basically says in directory rawdata_a it currently has 1 file called AA.csv.
Then wrote a procedure as follows ...
procedure p_process_files(pv_return_message OUT varchar2)is
cursor c_get_stock_data_directories is
select distinct sdd_directory_path,
sdd_table_name
from stock_data_directories
order by sdd_table_name;
vv_return_message varchar2(1000);
begin
-- here get the files for each directory
for r_get_stock_directories in c_get_stock_data_directories loop
p_build_external_table(r_get_stock_directories.sdd_directory_path,
     r_get_stock_directories.sdd_table_name,
     vv_return_message);
end loop;
end;
then wrote a procedure called p_build_external_table as follows ...
procedure p_build_external_table(pv_directory_path IN stock_data_directories.sdd_directory_path%type, -- e.g. 'C:\kpollock\A\
pv_table_name IN stock_data_directories.sdd_table_name%type, -- e.g. rawdata_a
pv_return_message OUT varchar2) is
vv_pattern VARCHAR2(1024);
ns VARCHAR2(1024);
vv_file_name varchar2(4000);
vv_start_string varchar2(1) := '''';
vv_end_string varchar2(3) := ''',';
vn_counter number := 0;
vv_err varchar2(2000);
BEGIN
vv_pattern := pv_directory_path||'*';
SYS.DBMS_BACKUP_RESTORE.searchFiles(vv_pattern, ns);
FOR each_file IN (SELECT FNAME_KRBMSFT AS name FROM X$KRBMSFT) LOOP
if each_file.name like '%.CSV' then
vv_file_name := vv_file_name||vv_start_string||substr(each_file.name,instr(each_file.name,'\',1,3)+1)||vv_end_string;
     vn_counter := vn_counter + 1;
end if;
END LOOP;
vv_file_name := substr(vv_file_name,1,length(vv_file_name)-1); -- remove final , from string
execute immediate 'alter table '||pv_table_name||' location('||vv_file_name||')';
pv_return_message := 'Successfully changed '||pv_table_name||' at '||pv_directory_path||' to now have '||to_char(vn_counter)||' directories';
exception
when others then
vv_err := sqlerrm;
pv_return_message := ' Error found updating directories. Error = '||vv_err;
END;
This reads every file in the directory and appends it to a list, so if it finds A.csv and ABC.csv, then using the dynamic sql, it alters the location to now read 'a.csv','abc.csv',
It ignores all other file extentions.