SQL*Loader and pipe on unix

On unix is it possible to pipe the data to be loaded in to SQL*Loader instead of reading it from file? I have a program that generates data that is to be loaded, and it is a bit waste of time to first save it to file, just to read it back in.

Yes, it's possible :
TEST@db102 SQL> desc test1
Name                                                  Null?    Type
A                                                              VARCHAR2(20)
B                                                              VARCHAR2(20)
TEST@db102 SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
[ora102 work db102]$ cat test1.dat
aaaaa,bbbbb
ccccc,ddddd
eeeee,fffff
[ora102 work db102]$ cat test1.ctl
load data
replace
INTO TABLE test1
fields terminated by ','
trailing nullcols
a, b
[ora102 work db102]$ cat test1.dat | sqlldr test/test control=test1.ctl
SQL*Loader: Release 10.2.0.1.0 - Production on Mon Sep 25 14:29:50 2006
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
Commit point reached - logical record count 3
[ora102 work db102]$ sqlplus test/test
SQL*Plus: Release 10.2.0.1.0 - Production on Mon Sep 25 14:29:58 2006
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
TEST@db102 SQL> select * from test1;
A                    B
aaaaa                bbbbb
ccccc                ddddd
eeeee                fffff
TEST@db102 SQL>                                                                                        

Similar Messages

  • SQL-Loader and Pipes

    I have to insert 300.000.000 datasets (generated dynamically by a seperate process) into an Oracle 8.1.6 database. I'd like to do this via SQL-Loader but since Linux (kernel 2.2 at least) supports only files up to 2Gigs and I don't have the necessary space on my machine to generate a large amount of ASCII files I'd like to ask whether there is a possibility to use an ordinary pipe in combination with SQL-Loader. I tried to use a named pipe but all my attempts failed.
    I don't like to use Perl and the DBI module since this would be far so slow (would take about one week I guess).
    Hans

    External tables are accessible from SQL, which generally simplifies life if the data files are physically located on the database server since you don't have to coordinate a call to an external SQL*Loader script with other PL/SQL processing. Under the covers, external tables are normally just invoking SQL*Loader.
    SQL*Loader is more appropriate if the data files are on a different server or if it is easier to call an executable rather than calling PL/SQL (i.e. if you have a batch file that runs on a server other than the database server that wants to FTP a data file from a FTP server and then load the data into Oracle).
    Justin

  • Help in calling sql loader and an oracle procedure in a script

    Hi Guru's
    please help me in writing an unix script which will call sql loader and also an oracle procedure..
    i wrote an script which is as follows.
    !/bin/sh
    clear
    #export ORACLE_SID='HOBS2'
    sqlldr USERID=load/ps94mfo16 CONTROL=test_nica.ctl LOG=test_nica.log
    retcode=`echo $?`
    case "$retcode" in
    0) echo "SQL*Loader execution successful" ;;
    1) echo "SQL*Loader execution exited with EX_FAIL, see logfile" ;;
    2) echo "SQL*Loader execution exited with EX_WARN, see logfile" ;;
    3) echo "SQL*Loader execution encountered a fatal error" ;;
    *) echo "unknown return code";;
    esac
    sqlplus USERID=load/ps94mfo16 << EOF
    EXEC DO_TEST_SHELL_SCRIPT
    it is loading the data in to an oracle table
    but the procedure is not executed..
    any valuable suggestion is highly appriciated..
    Cheers

    multiple duplicate threads:
    to call an oracle procedure and sql loader in an unix script
    Re: Can some one help he sql loader issue.

  • HELP: SQL*LOADER AND Ref Column

    Hallo,
    I have already posted and I really need help and don't come further with this
    I have the following problem. I have 2 tables which I created the following way:
    CREATE TYPE gemark_schluessel_t AS OBJECT(
    gemark_id NUMBER(8),
    gemark_schl NUMBER(4),
    gemark_name VARCHAR2(45)
    CREATE TABLE gemark_schluessel_tab OF gemark_schluessel_t(
    constraint pk_gemark PRIMARY KEY(gemark_id)
    CREATE TYPE flurstueck_t AS OBJECT(
    flst_id NUMBER(8),
    flst_nr_zaehler NUMBER(4),
    flst_nr_nenner NUMBER(4),
    zusatz VARCHAR2(2),
    flur_nr NUMBER(2),
    gemark_schluessel REF gemark_schluessel_t,
    flaeche SDO_GEOMETRY
    CREATE TABLE flurstuecke_tab OF flurstueck_t(
    constraint pk_flst PRIMARY KEY(flst_id),
    constraint uq_flst UNIQUE(flst_nr_zaehler,flst_nr_nenner,zusatz,flur_nr),
    flst_nr_zaehler NOT NULL,
    flur_nr NOT NULL,
    gemark_schluessel REFERENCES gemark_schluessel_tab
    Now I have data in the gemark_schluessel_tab which looks like this (a sample):
    1 101 Borna
    2 102 Draisdorf
    Now I wanna load data in my flurstuecke_tab with SQL*Loader and there I have problems with my ref column gemark_schluessel.
    One data record looks like this in my file (it is without geometry)
    1|97|7||1|1|
    If I wanna load my data record, it does not work. The reference (the system generated OID) should be taken from gemark_schluessel_tab.
    LOAD DATA
    INFILE *
    TRUNCATE
    CONTINUEIF NEXT(1:1) = '#'
    INTO TABLE FLURSTUECKE_TAB
    FIELDS TERMINATED BY '|'
    TRAILING NULLCOLS (
    flst_id,
    flst_nr_zaehler,
    flst_nr_nenner,
    zusatz,
    flur_nr,
    gemark_schluessel REF(CONSTANT 'GEMARK_SCHLUESSEL_TAB',GEMARK_ID),
    gemark_id FILLER
    BEGINDATA
    1|97|7||1|1|
    Is there a error I made?
    Thanks in advance
    Tig

    multiple duplicate threads:
    to call an oracle procedure and sql loader in an unix script
    Re: Can some one help he sql loader issue.

  • SQL Loader and Insert Into Performance Difference

    Hello All,
    Im in a situation to measure performance difference between SQL Loader and Insert into. Say there 10000 records in a flat file and I want to load it into a staging table.
    I know that if I use PL/SQL UTL_FILE to do this job performance will degrade(dont ask me why im going for UTL_FILE instead of SQL Loader). But I dont know how much. Can anybody tell me the performance difference in % (like 20% will decrease) in case of 10000 records.
    Thanks,
    Kannan.

    Kannan B wrote:
    Do not confuse the topic, as I told im not going to use External tables. This post is to speak the performance difference between SQL Loader and Simple Insert Statement.I don't think people are confusing the topic.
    External tables are a superior means of reading a file as it doesn't require any command line calls or external control files to be set up. All that is needed is a single external table definition created in a similar way to creating any other table (just with the additional external table information obviously). It also eliminates the need to have a 'staging' table on the database to load the data into as the data can just be queried as needed directly from the file, and if the file changes, so does the data seen through the external table automatically without the need to re-run any SQL*Loader process again.
    Who told you not to use External Tables? Do they know what they are talking about? Can they give a valid reason why external tables are not to be used?
    IMO, if you're considering SQL*Loader, you should be considering External tables as a better alternative.

  • SQL *Loader and External Table

    Hi,
    Can anyone tell me the difference between SQL* Loader and External table?
    What are the conditions under we can use SQL * Loader and External Table.
    Thanx

    External tables are accessible from SQL, which generally simplifies life if the data files are physically located on the database server since you don't have to coordinate a call to an external SQL*Loader script with other PL/SQL processing. Under the covers, external tables are normally just invoking SQL*Loader.
    SQL*Loader is more appropriate if the data files are on a different server or if it is easier to call an executable rather than calling PL/SQL (i.e. if you have a batch file that runs on a server other than the database server that wants to FTP a data file from a FTP server and then load the data into Oracle).
    Justin

  • Using SQL*Loader and UTL_FILE to load and unload large files(i.e PDF,DOCs)

    Problem : Load PDF or similiar files( stored at operating system) into an oracle table using SQl*Loader .
    and than Unload the files back from oracle tables to prevoius format.
    I 've used SQL*LOADER .... " sqlldr " command as :
    " sqlldr scott/[email protected] control=c:\sqlldr\control.ctl log=c:\any.txt "
    Control file is written as :
    LOAD DATA
    INFILE 'c:\sqlldr\r_sqlldr.txt'
    REPLACE
    INTO table r_sqlldr
    Fields terminated by ','
    id sequence (max,1) ,
    fname char(20),
    data LOBFILE(fname) terminated by EOF )
    It loads files ( Pdf, Image and more...) that are mentioned in file r_sqlldr.txt into oracle table r_sqlldr
    Text file ( used as source ) is written as :
    c:\kalam.pdf,
    c:\CTSlogo1.bmp
    c:\any1.txt
    after this load ....i used UTL_FILE to unload data and write procedure like ...
    CREATE OR REPLACE PROCEDURE R_UTL AS
    l_file UTL_FILE.FILE_TYPE;
    l_buffer RAW(32767);
    l_amount BINARY_INTEGER ;
    l_pos INTEGER := 1;
    l_blob BLOB;
    l_blob_len INTEGER;
    BEGIN
    SELECT data
    INTO l_blob
    FROM r_sqlldr
    where id= 1;
    l_blob_len := DBMS_LOB.GETLENGTH(l_blob);
    DBMS_OUTPUT.PUT_LINE('blob length : ' || l_blob_len);
    IF (l_blob_len < 32767) THEN
    l_amount :=l_blob_len;
    ELSE
    l_amount := 32767;
    END IF;
    DBMS_LOB.OPEN(l_blob, DBMS_LOB.LOB_READONLY);
    l_file := UTL_FILE.FOPEN('DBDIR1','Kalam_out.pdf','w', 32767);
    DBMS_OUTPUT.PUT_LINE('File opened');
    WHILE l_pos < l_blob_len LOOP
    DBMS_LOB.READ (l_blob, l_amount, l_pos, l_buffer);
    DBMS_OUTPUT.PUT_LINE('Blob read');
    l_pos := l_pos + l_amount;
    UTL_FILE.PUT_RAW(l_file, l_buffer, TRUE);
    DBMS_OUTPUT.PUT_LINE('writing to file');
    UTL_FILE.FFLUSH(l_file);
    UTL_FILE.NEW_LINE(l_file);
    END LOOP;
    UTL_FILE.FFLUSH(l_file);
    UTL_FILE.FCLOSE(l_file);
    DBMS_OUTPUT.PUT_LINE('File closed');
    DBMS_LOB.CLOSE(l_blob);
    EXCEPTION
    WHEN OTHERS THEN
    IF UTL_FILE.IS_OPEN(l_file) THEN
    UTL_FILE.FCLOSE(l_file);
    END IF;
    DBMS_OUTPUT.PUT_LINE('Its working at last');
    END R_UTL;
    This loads data from r_sqlldr table (BOLBS) to files on operating system ,,,
    -> Same procedure with minor changes is used to unload other similar files like Images and text files.
    In above example : Loading : 3 files 1) Kalam.pdf 2) CTSlogo1.bmp 3) any1.txt are loaded into oracle table r_sqlldr 's 3 rows respectively.
    file names into fname column and corresponding data into data ( BLOB) column.
    Unload : And than these files are loaded back into their previous format to operating system using UTL_FILE feature of oracle.
    so PROBLEM IS : Actual capacity (size ) of these files is getting unloaded back but with quality decreased. And PDF file doesnt even view its data. means size is almot equal to source file but data are lost when i open it.....
    and for images .... imgaes are getting loaded an unloaded but with colors changed ....
    Also features ( like FFLUSH ) of Oracle 've been used but it never worked
    ANY SUGGESTIONS OR aLTERNATE SOLUTION TO LOAD AND UNLOAD PDFs through Oracle ARE REQUESTED.
    ------------------------------------------------------------------------------------------------------------------------

    Thanks Justin ...for a quick response ...
    well ... i am loading data into BLOB only and using SQL*Loader ...
    I've never used dbms_lob.loadFromFile to do the loads ...
    i 've opend a file on network and than used dbms_lob.read and
    UTL_FILE.PUT_RAW to read and write data into target file.
    actually ...my process is working fine with text files but not with PDF and IMAGES ...
    and your doubt of ..."Is the data the proper length after reading it in?" ..m not getting wat r you asking ...but ... i think regarding data length ..there is no problem... except ... source PDF length is 90.4 kb ..and Target is 90.8 kb..
    thats it...
    So Request u to add some more help ......or should i provide some more details ??

  • Is there any difference in Oracle 9i SQL Loader and Oracle 10g SQL Loader

    Hi
    Can anyone tell me whether is there any difference in Oracle 9i SQL Loader and Oracle 10g SQL Loader?
    I am upgrading the 9i db to 10g and wanted to run the 9i SQL Loader control files on upgraded 10g db. So please let me know is there any difference which I need to consider any modifications in the control files..
    Thank you in advance
    Adi

    answered

  • Import and process larger data with SQL*Loader and Java resource

    Hello,
    I have a project to import data from a text file in a schedule. A lager data, with nearly 20,000 record/1 hours.
    After that, we have to analysis the data, and export the results into a another database.
    I research about SQL*Loader and Java resource to do these task. But I have no experiment about that.
    I'm afraid of the huge data, Oracle could be slowdown or the session in Java Resource application could be timeout.
    Please tell me some advice about the solution.
    Thank you very much.

    With '?' mark i mean " How i can link this COL1 with column in csv file ? "
    Attilio

  • SQL*Loader Named Pipe Load Thread goes to 0

    Hi,
    Running SQL*Loader 10.2.0.1.0 on XP pro against 10g EE 10.2.0.2.0 Production, RAC.
    Control file:
    OPTIONS ( MULTITHREADING=TRUE, DIRECT=TRUE, ROWS=5000000, COLUMNARRAYROWS = 200000 )
    UNRECOVERABLE
    LOAD DATA
    --INFILE 'C:\Program Files\Delta Data Software, Inc\DDS Twenty Two C\SQL_LOADER\LDR.DAT' "fix 427"
    INFILE '\\.\pipe\testpipe' "fix 427"
    BADFILE 'C:\Program Files\Delta Data Software, Inc\DDS Twenty Two C\DataGen\S-INTERSP-DL.bad'
    DISCARDFILE 'C:\Program Files\Delta Data Software, Inc\DDS Twenty Two C\DataGen\S-INTERSP-DL.dsc'
    INSERT
    INTO TABLE "TDW999999"
    Data File is 1 million row text file, 427 byte per line
    Problem:
    When calling SqlLdr directly against the data file, the import runs in 1:47(min\secs).
    Total stream buffers loaded by SQL*Loader main thread: 408
    Total stream buffers loaded by SQL*Loader load thread: 2036
    However, when I pipe the same data to SqlLdr via a simple c# application with a named pipe, it takes 5:56(mins\secs).
    Pipe Base Load:
    Total stream buffers loaded by SQL*Loader main thread: 104249
    Total stream buffers loaded by SQL*Loader load thread: 0
    It looks like SqlLdr can not go parallel when input is from a named pipe.
    Any Ideas on how I can get my load thread count back up when using pipes ?
    Thanks

    Hi,
    According to http://www.mcse.ms/archive27-2006-1-2068094.html you can do the following (substitute your c# program for 'type load.dat'):
    >
    type load.dat|sqlldr control=load.ctl userid=user/password data=\"-\"
    Alternative (for both , unix and windows)- infile syntax:
    load data
    infile "-"
    into table...
    Metalink note 191043.1 may be interesting for you as well.
    Does the performance then improve?
    Cheers,
    Colin

  • SQL loader and stream files with new line '

    I have been trying unsuccessfully to load EDI files using SQL loader. The problem
    is that the lines are terminated by ' and when I use the stream file option it does
    not recognise the line terminator given. As I understand it from the documentation
    this should work - but it does not. I have also used the Hex option with no better
    result. Does anyone have any ideas ?
    I can and have used tr "[']" "[\n]" in Unix to convert the ' to newlines - I just
    wonder am I missing something in SQL loader which will allow me to do this ?
    This is the sql loader control file
    LOAD DATA
    INFILE 'WS860685.MFD' "Str ''' "
    BADFILE 'WS860685.bad'
    DISCARDFILE 'WS860685.dsc'
    INTO TABLE "DUND1"."EDI_LOADED_TEMP"
    REPLACE
    FIELDS TERMINATED BY '+'
    TRAILING NULLCOLS
    (L1,
    L2,
    L3,
    L4,
    L5,
    L6,
    L7,
    L8,
    L9,
    L10,
    L11,
    L12,
    L13,
    L14,
    L15,
    L16,
    L17,
    L18,
    L19,
    L20,
    L21,
    L22,
    L23,
    L24,
    L25,
    L26,
    L27,
    L28,
    L29,
    L30,
    L31,
    L32,
    L33,
    L34,
    L35,
    L36,
    L37,
    L38,
    L39,
    L40,
    LNO)
    Heres a sample of the data
    UNB+UNOA:2+5398888501357+5398888501838+080306:0737+395+ DESADV+++1'UNH+0001+DESADV:D:93A:UN:EAN004'

    http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:6020061915147
    answers this question perfectly

  • SQL*Loader and integrity constraints

    I am running into trouble with integrity constraints in my SQL*Loader script runs. Should I be going up to unix and removing the constraints from sqlplus, then asking the wizard to reinstall them after the data load? Is this documented somewhere? I can't find it. Sorry for asking such a basic question.
    Thanks,
    Ann Cantelow

    Hi,
    Your approach is correct.
    Create tables first
    Move data via the scripts
    Create contraints, indexes, primary keys
    See the User Guide for the ORacle Migration Workbench http://otn.oracle.com/tech/migration/workbench
    Regards
    John

  • Sql Loader and carriage returns

    I am currently trying to use sql loader to load data from flat files that was extracted from sybase using bcp and delimited with pipes. There are text and varchar columns that contain carriage return line feeds. I want to preserve these, but I can not load them into Oracle 8.05 using sql loader as it interprets them as end of record indicators. Does anyone have a way to solve this problem? Any assistance would be appreciated.
    Thanks in advance,
    Jignesh

    External tables are accessible from SQL, which generally simplifies life if the data files are physically located on the database server since you don't have to coordinate a call to an external SQL*Loader script with other PL/SQL processing. Under the covers, external tables are normally just invoking SQL*Loader.
    SQL*Loader is more appropriate if the data files are on a different server or if it is easier to call an executable rather than calling PL/SQL (i.e. if you have a batch file that runs on a server other than the database server that wants to FTP a data file from a FTP server and then load the data into Oracle).
    Justin

  • SQL*Loader and DECODE function

    Hi All,
    I am loading data from data files into oracle tables and while loading the data using SQL*Loader, the following requirement needs to be fulfilled.
    1) If OQPR < 300, RB = $ 0-299, SC = "SC1"
    2) If 300 < OQPR < 1200, RB = $ 300-1199, SC = "SC2"
    3) If 1200 < OQPR < 3000, RB = $ 1200-2999, SC = "SC3"
    4) If OQPR > 3000 USD, RB = > $3000, SC = "SC4"
    Here OPQR is a field in the data file.
    Can anyone suggest how do we handle this using DECODE function? Triggers and PL/SQL functions are not to be used.
    TIA.
    Regards,
    Ravi.

    The following expression gives you different values for your different intervals and boundaries :
    SIGN(:OQPR - 300) + SIGN(:OQPR - 1200) + SIGN(:OQPR - 3000)

  • Problem with SQL*Loader and different date formats in the same file

    DB: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bi
    System: AIX 5.3.0.0
    Hello,
    I'm using SQL*Loader to import semi-colon separated values into a table. The files are delivered to us by a data provider who concatenates data from different sources and this results in us having different date formats within the same file. For example:
    ...;2010-12-31;22/11/1932;...
    I load this data using the following lines in the control file:
    EXECUTIONDATE1     TIMESTAMP     NULLIF EXECUTIONDATE1=BLANKS     "TO_DATE(:EXECUTIONDATE1, 'YYYY-MM-DD')",
    DELDOB          TIMESTAMP     NULLIF DELDOB=BLANKS          "TO_DATE(:DELDOB, 'DD/MM/YYYY')",
    The relevant NLS parameters:
    NLS_LANGUAGE=FRENCH
    NLS_DATE_FORMAT=DD/MM/RR
    NLS_DATE_LANGUAGE=FRENCH
    If I load this file as is the values loaded into the table are 31 dec 2010 and 22 nov *2032*, aven though the years are on 4 digits. If I change the NLS_DATE_FORMAT to DD/MM/YYYY then the second date value will be loaded correctly, but the first value will be loaded as 31 dec *2020* !!
    How can I get both date values to load correctly?
    Thanks!
    Sylvain

    This is very strange, after running a few tests I realized that if the year is 19XX then it will get loaded as 2019, and if it is 20XX then it will be 2020. I'm guessing it may have something to do with certain env variables that aren't set up properly because I'm fairly sure my SQL*Loader control file is correct... I'll run more tests :-(

Maybe you are looking for

  • Wireless Modem and Airport Express USB Printer

    I am already connected to the internet via a wireless modem provided by my ISP. The connection uses Airport. I just purchased an Airport Express to connect my computer to a printer through the USB port. However, I can't get the airport utility to con

  • Exchange rate difference amount need to be posted to venmdor

    Hi We have created import PO with exchange rate 76.35 then we have did MIRO with exchange rate 78 Liability created was with 78 value while making the payment with f-53 we are getting the value which is created with 76.35. While making the payment cl

  • Drop partition without disabling foreign key

    Hi All, I have parent and child table. Parent table create table parent_1 (id number, create_date date, constraint parent_1_pk001 PRIMARY KEY (id)) PARTITION BY RANGE (create_date) INTERVAL (NUMTODSINTERVAL(1,'DAY')) (PARTITION parent_1_part VALUES L

  • Jtable - Set visible columns

    Hi, I'm trying to build an applet to edit mysql tables. I used a Jtable inside a JscrollPane but I would like to display only 3 or 4 columns at a time (I can have up to 40 !) and then let the user scrolls to the others with the horizontal scrollbar.

  • IWeb 09 links not working correctly using IE8

    Hello all, I built a website within using iweb 09 and hosted it on Godaddy. The problem that i have has to do with my hyper links directing me to the Home page. this seems to only happen when I use IE8. If I use Safari all of the hyperlinks seem to w