How to use external table - creating NFS mount -the details involved

Hi,
We are using Oracle 10.2.0.3 on Solaris 10. I want to use external tables to load huge csv data into the database. This concept was tested and also found to be working fine. But my doubt that : since ours is a J2EE application, the csv files have to come from the front end- from the app server. So in this case how to move them to the db server?
For my testing I just used putty to transfer the file to db server, than ran the dos2unix command to strip off the control character at the end of file. but since this is to be done from the app server, putty can not be used. In this case how can this be done? Are there any risks or security issues involved in this process?
Regards

orausern wrote:
For my testing I just used putty to transfer the file to db server, than ran the dos2unix command to strip off the control character at the end of file. but since this is to be done from the app server, putty can not be used. In this case how can this be done? Are there any risks or security issues involved in this process? Not sure why "putty" cannot be used. This s/w uses the standard telnet and ssh protocols. Why would it not work?
As for getting the files from the app server to the db server. There are a number of options.
You can look at it from an o/s replication level. The command rdist is common on most (if not all) Unix/Linux flavours and used for remote distribution and sync'ing of files and directories. It also supports scp as the underlying protocol (instead of the older rcp protocol).
You can use file sharing - the typical Unix approach would be to use NFS. Samba is also an option if NTLM (Windows) is already used in the organisation and you want to hook this into your existing security infrastructure (e.g. using Microsoft's Active Directory).
You can use a cluster file system - a file system that resides on shared storage and can be used by by both app and db servers as a mounted/cooked file system. Cluster file systems like ACFS, OCFS2 and GFS exist for Linux.
You can go for a pull method - where the db server on client instruction (that provides the file details), connects to the app server (using scp/sftp/ftp), copy that file from the app server, and then proceed to load it. You can even add a compression feature to this - so that the db server copies a zipped file from the app server and then unzip it for loading.
Security issues. Well, if the internals is not exposed then security will not be a problem. For example, defining a trusted connection between app server ad db server - so the client instruction does not have to contain any authentication data. Letting the client instruction only specify the filename and have the internal code use a standard and fixed directory structure. That way the client cannot instruct something like +/etc/shadow+ be copied from the app server and loaded into the db sever as a data file. Etc.

Similar Messages

How to use External Tables

Hi, I want to know how to use external tables? The procedure i follow for SQL*Loader in my shell script is:
Call the sqlldr
Check return code
If return code = 2. get count of records from bad file and exit.
If return code != 0 and != 2 then exit as fatal error
If return code = 0 then continue.
But i don't understand how to use external table as it is not invoked and does not have a return code.

Hello,
What do you mean alogrithim, do you mean to invoke external table as you invoke sqlldr? You can generate external table script from sqlldr and use it to load data but you might have to modify the table name and datafile name . You can see if any bad file generated from sqlldr and if yes then you can generate external table (using sqlldr) or based on your bad file. May be if you can list what exactly you want in bullet points, then it will be easier to suggest you.
Regards

Report question: how to use external table to bring data from excel file?

i need to import excel data into oracle report, how to do that? thanks.

Hi... What is the error message you get when using external tables?
You must create a control center connection in the Connection Explorer.
I recommend you read this:
http://download-east.oracle.com/docs/cd/B31080_01/doc/owb.102/b28223/toc.htm
Look for the following sections:
- Using External Tables (and the "External Tables versus Flat File Operators" sub section)
- About Flat File Modules
- Chapter 16, Deploying Target Systems
Regards,
Marcos

How to use external libraries jar files in the composite application?

Hi,
I have some external jars files (which consists of java classes) that I would like to use in my composite application.
I have created a DC with type as external library for the jars. However, I don't know how to specify the DC to be used in my caf application.
Can anyone help?
Thanks.
Regards,
Fui Voon

Hi,
My jar files consists only java classes and a manifest.mf. The files are compiled using jdk 1.4.2-b28. Will this be any issue?
It seems that if I ignore the error and continue step 7 which is to do deployment of the ear, the error given is
Cannot update it.
Caused by: com.sap.engine.library.bytecode.cf.CFException: Invalid constant pool tag, 60 at index 41.
Does anyone know what that mean?
Is there any sample that teach how to deploy external libraries as bundled library?
Thanks.
Regards,
Fui Voon

How to write data to text file using external tables

can anybody tell how to write data to text file using external tables concept?

Hi,
Using external table u can load the data in your local table in database,
then using your local db table and UTL_FILE pacakge u can wrrite data to text file
external table
~~~~~~~~~~~
http://download-east.oracle.com/docs/cd/B19306_01/server.102/b14200/statements_7002.htm#i2153251
UTL_FILE
~~~~~~~~~
http://download-east.oracle.com/docs/cd/B19306_01/appdev.102/b14258/u_file.htm#sthref14093
Message was edited by:
Nicloei W
Message was edited by:
Nicloei W

Need to merge a csv file using external tables into a main table

Hi,
I have a csv file which contains the date(with time stamp), column1(number),column2(number), column3 (number). I am using external tables concept to load the data froom csv to this external table and then merging into the main table. Problem here is : the csv file is a system generated file and nothing can be edited under it. our aim is to automate this process of loading data from csv to the table. In this csv the date time stamp is not in the proper format.I mean the date is not visible and only minutes and seconds are visible.By changing the format in csv manually this can be overcome.but we donot need any manual intervention.
how can i overcome this problem ?? please help mee...
Excels data looks like:
(PDH-TSV 4.0) (India Standard Time)(-330)     \\DISAPPSER01\Processor(_Total)\% Privileged Time     \\DISAPPSER01\Processor(_Total)\% Processor Time     \\DISAPPSER01\Web Service(_Total)\Current Connections
56:59.0               47
57:09.0     0.72379582     4.204561281     46
57:19.0     0.916548537     4.006179927     44
57:29.0     0.663034771     3.674662541     43
57:39.0     0.750789844     4.093933999     42
57:49.0     0.721538487     2.650858026     40
57:59.0     0.594781604     3.333393703     40

please format your sample data giving header to the column so that we can make sense out of the values, also since the minutes and seconds are only given, what is the date to be considered for records to be moved to the master table, sysdate or will the date be passed as a parameter?

Need info on using external tables load/write data

Hi All,
We are planning to load conversion/interface data using external tables feature available in the Oracle database.
Also for outbound interfaces, we are planning to use the same feature to write data in the file system.
If you have done similar exercise in any of your projects, please share sample code units. Also, let me know if there
are any cons/limitations in this approach.
Thanks,
Balaji

Please see old threads for similar discussion -- http://forums.oracle.com/forums/search.jspa?threadID=&q=external+AND+tables&objID=c3&dateRange=all&userID=&numResults=15&rankBy=10001
Thanks,
Hussein

How to read any file using external tables.

Hi folks,
I have written an application that reads a series of csv files using external tables which works fine as long as I specify each file name in the directory i.e.......
CREATE TABLE gb_test
(file_name varchar2(10),
rec_date date
rec_name VARCHAR2(20),
rec_age number,
ORGANIZATION EXTERNAL
TYPE ORACLE_LOADER
DEFAULT DIRECTORY GB_TEST
ACCESS PARAMETERS
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
LOCATION ('data1.csv','data2.csv','data3.csv','data4.csv')
PARALLEL 5
REJECT LIMIT 20000;
However I have discovered that I may not know the name of the files to be processed prior to the program being run so just want to read any file regardless of it's name (although it will always be a .csv file).
Is there a way to ensure that you don't need to specify the files to be read in the LOCATION part of the syntax.
Thanks in advance.
Graham.

Right, I have now completed this, however it's currently only working as SYS as opposed to any user, however here is a detail of the scenario and the steps required in case any of you guys need in the future ......
The problem was I needed to search for csv files on my hard-drive. These files would be stored in a series of directories (a through to z), so I needed a way to read all 26 directories and process all files in these directories.
The problem was, prior to running the program, the user would remove all the files in the directories and insert new ones, but it was never known how many he would decide to do each time.
Solution: I created a table called stock_data_directories as follows ...
create table stock_data_directories(sdd_rec_no number,
sdd_table_name varchar2(50),
sdd_directory_name varchar2(50),
sdd_directory_path varchar2(100));
Then inserted 26 records like ...
insert into stock_data_directories(sdd_rec_no,sdd_table_name,sdd_directory_name,sdd_directory_path)
values(1,'rawdata_a','KPOLLOCKA','C:\KPOLLOCK\A')
insert into stock_data_directories(sdd_rec_no,sdd_table_name,sdd_directory_name,sdd_directory_path)
values(2,'rawdata_b','KPOLLOCKB','C:\KPOLLOCK\B');
etc...etc...
Then created 26 DIRECTORIES E.G.
CREATE OR REPLACE DIRECTORY KPOLLOCKA AS 'C:\KPOLLOCK\A';
CREATE OR REPLACE DIRECTORY KPOLLOCKB AS 'C:\KPOLLOCK\B';
Then created 26 external tables like the following ...
CREATE TABLE rawdata_a
(stock varchar2(1000),
stock_date varchar2(10),
stock_open VARCHAR2(20),
stock_high varchar2(20),
stock_low varchar2(20),
stock_close VARCHAR2(30),
stock_qty varchar2(20) )
ORGANIZATION EXTERNAL
TYPE ORACLE_LOADER
DEFAULT DIRECTORY KPOLLOCKA
ACCESS PARAMETERS
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
LOCATION ('AA.csv')
PARALLEL 5
REJECT LIMIT 20000
This basically says in directory rawdata_a it currently has 1 file called AA.csv.
Then wrote a procedure as follows ...
procedure p_process_files(pv_return_message OUT varchar2)is
cursor c_get_stock_data_directories is
select distinct sdd_directory_path,
sdd_table_name
from stock_data_directories
order by sdd_table_name;
vv_return_message varchar2(1000);
begin
-- here get the files for each directory
for r_get_stock_directories in c_get_stock_data_directories loop
p_build_external_table(r_get_stock_directories.sdd_directory_path,
     r_get_stock_directories.sdd_table_name,
     vv_return_message);
end loop;
end;
then wrote a procedure called p_build_external_table as follows ...
procedure p_build_external_table(pv_directory_path IN stock_data_directories.sdd_directory_path%type, -- e.g. 'C:\kpollock\A\
pv_table_name IN stock_data_directories.sdd_table_name%type, -- e.g. rawdata_a
pv_return_message OUT varchar2) is
vv_pattern VARCHAR2(1024);
ns VARCHAR2(1024);
vv_file_name varchar2(4000);
vv_start_string varchar2(1) := '''';
vv_end_string varchar2(3) := ''',';
vn_counter number := 0;
vv_err varchar2(2000);
BEGIN
vv_pattern := pv_directory_path||'*';
SYS.DBMS_BACKUP_RESTORE.searchFiles(vv_pattern, ns);
FOR each_file IN (SELECT FNAME_KRBMSFT AS name FROM X$KRBMSFT) LOOP
if each_file.name like '%.CSV' then
vv_file_name := vv_file_name||vv_start_string||substr(each_file.name,instr(each_file.name,'\',1,3)+1)||vv_end_string;
     vn_counter := vn_counter + 1;
end if;
END LOOP;
vv_file_name := substr(vv_file_name,1,length(vv_file_name)-1); -- remove final , from string
execute immediate 'alter table '||pv_table_name||' location('||vv_file_name||')';
pv_return_message := 'Successfully changed '||pv_table_name||' at '||pv_directory_path||' to now have '||to_char(vn_counter)||' directories';
exception
when others then
vv_err := sqlerrm;
pv_return_message := ' Error found updating directories. Error = '||vv_err;
END;
This reads every file in the directory and appends it to a list, so if it finds A.csv and ABC.csv, then using the dynamic sql, it alters the location to now read 'a.csv','abc.csv',
It ignores all other file extentions.

How to find rejected rows using External Table

Hi all,
I had written a stored procedure to read comma(,) separated flat file using External Tables in oracle 9i with Reject Limit "Unlimited" Option. Sometimes all rows are successfully loaded into external table and sometimes some rows are failed, I can find out those rows from logfile created by oracle. I want to inserted those rows key values into some other table. Can any body suggest/links how to write this.
Thanking you in advance

"Is there a way to have the system truncate the log files besides some O/S utility that will scour the directory every night, week, etc.?"
You can use UTL_FILE.FREMOVE to delete the file, if you have sufficient privileges. You can schedule it using DBMS_JOB if you like or run it before you recreate the file, as demonstrated below. The pl/sql block to check whether the file exists is used for demonstration purposes only and is not necessary. This is just one method. There are various ways to delete or truncate a file. This just seems like the simplest.
scott@ORA92> CREATE OR REPLACE DIRECTORY mydir AS 'c:\oracle'
2 /
Directory created.
scott@ORA92> DECLARE
2    v_bfile BFILE := BFILENAME ('MY_DIR', 'test_tab.log');
3 BEGIN
4    IF DBMS_LOB.FILEEXISTS (v_bfile) = 1
5    THEN DBMS_OUTPUT.PUT_LINE ('test_tab.log exists');
6    ELSE DBMS_OUTPUT.PUT_line ('test_tab.log does not exist');
7    END IF;
8 END;
9 /
test_tab.log exists
PL/SQL procedure successfully completed.
scott@ORA92> EXECUTE UTL_FILE.FREMOVE ('MY_DIR', 'test_tab.log')
PL/SQL procedure successfully completed.
scott@ORA92> DECLARE
2    v_bfile BFILE := BFILENAME ('MY_DIR', 'test_tab.log');
3 BEGIN
4    IF DBMS_LOB.FILEEXISTS (v_bfile) = 1
5    THEN DBMS_OUTPUT.PUT_LINE ('test_tab.log exists');
6    ELSE DBMS_OUTPUT.PUT_line ('test_tab.log does not exist');
7    END IF;
8 END;
9 /
test_tab.log does not exist
PL/SQL procedure successfully completed.
scott@ORA92> CREATE TABLE test_tab
2    (col1 NUMBER,
3      col2 VARCHAR2(4))
4 ORGANIZATION external
5    (TYPE ORACLE_LOADER
6      DEFAULT DIRECTORY mydir
7      ACCESS PARAMETERS
8        (RECORDS DELIMITED BY NEWLINE
9         BADFILE 'MYDIR':'test_bad.bad'
10         LOGFILE 'MYDIR':'test_tab.log'
11         FIELDS TERMINATED BY ","
12           (col1,
13            col2))
14      LOCATION ('test.dat'))
15 REJECT LIMIT UNLIMITED
16 /
Table created.
scott@ORA92> SELECT * FROM test_tab
2 /
      COL1 COL2
         1 a
         2 b
scott@ORA92> "Or is there a way to prevent the log from being written to every time it's accessed?"
You can use NOLOGFILE as an access parameter to prevent it from being written to, as shown below.
scott@ORA92> CREATE OR REPLACE DIRECTORY mydir AS 'c:\oracle'
2 /
Directory created.
scott@ORA92> CREATE TABLE test_tab
2    (col1 NUMBER,
3      col2 VARCHAR2(4))
4 ORGANIZATION external
5    (TYPE ORACLE_LOADER
6      DEFAULT DIRECTORY mydir
7      ACCESS PARAMETERS
8        (RECORDS DELIMITED BY NEWLINE
9         BADFILE 'MYDIR':'test_bad.bad'
10         NOLOGFILE
11         FIELDS TERMINATED BY ","
12           (col1,
13            col2))
14      LOCATION ('test.dat'))
15 REJECT LIMIT UNLIMITED
16 /
Table created.
scott@ORA92> SELECT * FROM test_tab
2 /
      COL1 COL2
         1 a
         2 b
scott@ORA92> "Is there a way to have the log written only when an error occurs?"
Not that I know of, but that does not mean that somebody else doesn't know how or isn't able to figure out a way. If you do find a way, please post it for the benefit of the rest of us.

How to implement row level security using external tables

Hi All Gurus/ Masters,
I want to implement row level security using external tables, as I'm not sure how to implement that. and I'm aware of using it by RPD level authentication.
I can use a filter condition in my user level so that he can access his data only.
But when i have 4 tables in external tables
users
groups
usergroups
webgrups
Then in which table I need to give the filter conditions..
Pl let me know this ...

You pull the Group into a repository variable using a session variable init block, then reference that variable in the data filters either in the LTS directly or in the security management as Filters. You reference it with the syntax VALUEOF("NQ_SESSION.Variable Name")
Hope this helps

How can I use external tables with directories not on Oracle host?

Oracle 11.2.0.2
We have developers using C# to populate global temporary tables (two one header and detail with 1 to many relationbship between these two tables). The process of using C# was blowing up memory on Windows.
As an alternative I requested them to create two CSV files and I can use sqlldr to load the detail CSV file of over 1 million rows under 15 seconds. It came to my mind that I could use external tables for this purpose with getting read of header and trailer from the CSV files.
The issue I have is that I am not a DBA so I do not have access to OS host that Oracle instance is running. sqlldr is a client side tool, whereas external tables are server side. I was wondering if there is an alternative in 11g to use external tables without creating directories on the OS host that Oracle runs on? Are there other alternatives?
Thanks

905989 wrote:
Oracle 11.2.0.2
We have developers using C# to populate global temporary tables (two one header and detail with 1 to many relationbship between these two tables). The process of using C# was blowing up memory on Windows.
As an alternative I requested them to create two CSV files and I can use sqlldr to load the detail CSV file of over 1 million rows under 15 seconds. It came to my mind that I could use external tables for this purpose with getting read of header and trailer from the CSV files.
The issue I have is that I am not a DBA so I do not have access to OS host that Oracle instance is running. sqlldr is a client side tool, whereas external tables are server side. I was wondering if there is an alternative in 11g to use external tables without creating directories on the OS host that Oracle runs on? Are there other alternatives?
Thanks
no other alternative

Is there a way to know how many rows were rejected using external tables?

Hi,
I'm developing a package that needs to load different kinds of text files into the DB.
I was wondering whether there's a way of knowing in SQL or PL/SQL how many rows were rejected if my external table is defined with "reject limit unlimited".
I'm using the external tables in "insert into ... (select ... from external_table where...)"
sql%rowcount returns the number of inserted rows
I was hoping for something similar/easy to get the number of rejected rows as well.
I know I can try to analyze the bad file for the number of records if there's no other way...
Thanks in advance.

Pyrocks wrote:
I was afraid of this answer...
this is a certain "no there's no other way" or should i wait for other responses (no disrespect - just want to make sure :) )Well the difference would be that this external table will have no complex logic. You would simple read in all the lines from the bad file. The chances that this is broken are therefore very slim compared with your original external table.
And tehre are many other ways possible. For example you can first copy the file into a real internal (temp) table without using any processing logic but with using an external table. Then do the processing / logic checking of the data and move the data from this temp table into your final data structure. Since you move the data two times the whole process will be slower then the original version.

How to import tab delimited using external table?

Hi all,
I'm using external table to import data from a tab delimited file. However, I keep getting an error message. I used both TERMINATED BY OX'09' and X'09' but still could not get it work.
Does anyone know how to solve this problem?
Thank you very much.

Hi
Try this:
import datafile="/myfiles/mydata" out=mydata dbms=tab replace delimiter='&' getnames=yes
run;
Thanks

How to load external data without using external tables

Hi,
I'm asked to develop an ETL for loading external data into a database but, unfortunately, the vers. is 8i so I can't use external tables.
I know that it's possible by using SQL-Loader. What I want to know is whether there's another way in 8i to load external data from a text file or a CSV file directly with a stored procedure without having to write CTL of SQL-Loader on the server. I mean, is there a way to write everything on the database the same manner I'd do whether I wrote data from a table to a file using for example UTL_FILE or the unique way is to write control files of SQL-Loader directly on the server?
Thanks!

Mark1970 wrote:
Thank you very much Karthick
I didn't know I could use UTL_FILE also for reading files. I've always use it only for writing data into external files.Yes you can use UTL_FILE to read. The version of oracle you are using i last used in 2004 :) Still remember giving the OS path of the file to open the file. That is long gone now. Its surprising that you are still in 8i.

How to Authorize external table users in OBIEE 11g

Hi All,
I have created Session System Variables and i am Using External table Data level Authentication and successfully external table Authentication is working.
My question is i want to Authorize this Extrenal table users in presentation services.e.g. I want to assign some dashboards or Reports to users.
In 10g when u login with the external table users automatically users will be created in Answers and used to assign this to webcat group.
In 11g how to achieve this???
Reply ASAP...
Thanks and Regards
Kiran Kumar
07795980891.

Hi Kiran,
Check this link.
http://www.rittmanmead.com/2010/11/oracle-bi-11g-active-directory-security-using-init-blocks-variables-10g-style/

How to use external table - creating NFS mount -the details involved

Similar Messages

Maybe you are looking for