Fastest way to load data

Option 1:
Insert statement with:
table mode: NOLOGGING    
insert mode: APPEND        
archivelog mode: noarchive log mode  
Option 2:
CTAS with NOLOGGING mode
Both options above would generate no redo log. Which one is better for performance? I'm loading large volume or rows (a few million) on a daily basis and this is a staging table so there is no problem to reprocess in case of failure.

Jonathan,
> Insert /*+ append */ can optimise for indexes by capturing the necessary column values as the data is loaded and then creating the indexes without needing to re-read the table
How did you do to came to this conclusion?
I did a simple test (t2 has a single column index) and got the following trace files
1- Direct path load
insert /*+ append */ into t2
select * from t1
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.00       0.01          0          1          0           0
Execute      1      0.05       0.08          3        140         87        1000
Fetch        0      0.00       0.00          0          0          0           0
total        2      0.05       0.09          3        141         87        1000
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 39
Rows     Row Source Operation
      1  LOAD AS SELECT  (cr=140 pr=3 pw=3 time=84813 us)
   1000   TABLE ACCESS FULL T1 (cr=5 pr=0 pw=0 time=92 us)
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  control file sequential read                    8        0.00          0.00
  db file sequential read                         2        0.00          0.00
  direct path write                               1        0.02          0.02
  SQL*Net message to client                       1        0.00          0.00
  SQL*Net message from client                     1        3.49          3.49
2- Conventional load
insert
into t2
select * from t1
call     count       cpu    elapsed       disk      query    current        rows
Parse        1      0.01       0.00          0          1          0           0
Execute      1      0.02       0.00          1         22        275        1000
Fetch        0      0.00       0.00          0          0          0           0
total        2      0.03       0.01          1         23        275        1000
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 39
Rows     Row Source Operation
   1000  TABLE ACCESS FULL T1 (cr=5 pr=0 pw=0 time=31 us)
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                         1        0.00          0.00
  SQL*Net message to client                       1        0.00          0.00
  SQL*Net message from client                     1        1.21          1.21
The trace file of the append hint contains a direct path write and a control file sequential read wait events confirming the direct path insert. But I still have not found in the trace file an indication on how the index is maintained separately from the table during a direct path load. Additionally I see that in both trace files there are two TABLE ACCESS FULL T1.
What I used to know is that during a direct path insert indexes are maintained differently from their table. Mini indexes are built on the incoming data and are finally merged with the physical index. But I don't see this also in the trace file.
However, In the append trace file there is the following select (occurring before the insert statement) that does not exist in the normal insert
select pos#,intcol#,col#,spare1,bo#,spare2
from
icol$ where obj#=:1
call     count       cpu    elapsed       disk      query    current        rows
Parse        2      0.00       0.00          0          0          0           0
Execute      2      0.00       0.01          0          0          0           0
Fetch        4      0.00       0.00          0          7          0           2
total        8      0.00       0.01          0          7          0           2
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)
Rows     Row Source Operation
      1  TABLE ACCESS BY INDEX ROWID ICOL$ (cr=3 pr=0 pw=0 time=25 us)
      1   INDEX RANGE SCAN I_ICOL1 (cr=2 pr=0 pw=0 time=21 us)(object id 40)
I am not sure if this has a relation with the mini-index pre-built data to be merged with the physical index.
That is my question : where to see this in the trace file?
Thanks
Mohamed Houri

Similar Messages

  • Fastest way to load data From MS Access to Oracle DB

    Hi All,
    We get an Access DB every week which we need to load to an Oracle DB, currently we have been using SQL loader but the process is slightly painful and horribly boring, so m trying to do some automation kind of stuff. I was thinking of doing the whole thing using a java application and then schedule it to run it at some pre-decided time. The approach I took was to read the access file and then load it using PreparedStatements to the oracle DB. But going through various threads in the forum, i found that its going to b a bit too slow (and the record count in my tables is around 600,000). Can there be a better way to do this? anyone done something similar b4.

    Well the only reason I want to use Java (i may go for C#) is that i dont want to spend time manually creating those CSV files from Access. Can that be done using something else?So use java to create the CSV files.
    And have you actually tried going straight to Oracle? What exactly is your time constraint?
    And another issue is that I sometimes have to make some adjustments (rounding off) to the data which is usually through query, but that is usually done after the data has been loaded in the DB.Which would make it irrelevant to actually moving the data then. If you are cleaning the data with SQL already then it is simple to wrap that in a proc and do it that way. Presumably you are loading to temp (non-production in-lined) tables first, then cleaning and moving.

  • Best Way to Load Data in Hash Partition

    Hi,
    I have partitioning by Hash on a Large Table of 5 TB. We have to load Data say more than 500GB daily on that table from ETL.
    What is the best way to Load data into that Big Table which has hash Partition .
    Regards
    Sahil Soni

    Do you have any specific requirements to match records to lookup tables or it just a straight load - that is an insert?
    Do you have any specific performance requirements?
    The easiest and fastest way to load data into Oracle is via external file and parallel query/parallel insert. Remember that parallel DML is not enabled by default and you have to do so via alter session command. You can leverage multiple CPU cores and direct path operation to perform the load.
    Assuming your database is on a linux/unix server - you could NFS load the file if it is on a remote system, but then you will be most likely limited by network transfer speed.

  • Best way to load data in table from combination of Table and flat file?

    Hi All,
    Could you please share your thoughts on best way of achieving this objective -
    Flat File - 15 Million records (Field A,B,C)
    Table A - 15 Million records ( Field A,D)
    Objective -
    Load Field A,B,C,D in Table B from Flat file and Table A.
    Data can be loaded from flat file in Table B then updated from Table A but this update operation is taking lot of time. (Intermediate Commit, Bulk operations already tried)
    Regards,
    Dark Knight

    Environment -
    Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    Tables are analyzed.
    Indexes are their.
    Update statement using the index.
    Data is close to 200 MB.
    I am interested in knowing if there are alternate ways of doing this, other than conventional way of loading data then updating it.

  • Fastest way to transfert data from cRIO

    Is anybody know the fastest way to transfert data from cRIO? I tested shared variable, but it's not fast enough. What is the fastest speed could we achieve with shared variable in cRIO? how can I transfert 50000 32 bit word per second from cRIO to my PC? This should run 24h/day.
    Thanks
    B.
    Benoit Séguin
    Software Designer

    Hi Benoit,
    Thanks for your post and I hope your well. I noticed you've not received a reply - and I would like offer my advice.
    Shared variables are one way to communicate over the Ethernet. You can use UDP, TCP and VI Server also. 
    UDP is the fastest network protocol but it includes little error handling and can lose data. TCP is commonly used for large systems. TCP takes more programming effort than shared variables, however, it provides a good mix of speed and reliability. You can use the VI server to run Vis and/or access and set the value of controls and indictors, but it is slower and not recommended for larger data sets. 
    Please let me know what you think,
    Kind Regards,
    James.
    Kind Regards
    James Hillman
    Applications Engineer 2008 to 2009 National Instruments UK & Ireland
    Loughborough University UK - 2006 to 2011
    Remember Kudos those who help!

  • Looking for the best way to load data to the appended ODS

    Hi all,
    I appended ODS with some new infoobjects.
    if there is a way to load data only to these datafields without all data deleting and then all data reloading? 
    thnx

    Hi Andrius,
    If your ODS is set to be overwritten AND the newly appended IOs are not key fields then a full upload without data deletion will works. Because old entries will be overwritten by new ones which have values for the appended IOs.
    On the other hand, if your ODS additive then from my point of view you need to remove all data and then reload again. Or else key figures would be doubled after a full upload.
    Qing

  • What is the fastest way of getting data?

    With a scanning electron microscope, I need to scan a 512*512 pixel area with a pixel repetition of 15000 (two channels), meaning averaging over 15000 measurements. Simultaneously I have to adjust the voltage output for every pixel.
    I am using a 6111E Multifunction I/O board in a 800MHz P3. The whole task has do be done as fast as possible (not more than 20 minutes altogether).
    What is the fastest way to get this huge amount of data with averaging and output in between? (E.g. do I use buffered read with hardware triggering or is there a faster way?)

    Using the NI-DAQ API (not LabView) will give you a significant amount of more control over what happens and when to the data stream; which translates to a more efficient program. But you need to program in C/C++ or Delphi then. The Measurement Studio provides ActiveX controls that are like the LabView ones for C&C++ (they�re slow like the LabView ones though � not a lot you can do about the Windows GDI).
    What are you trying to sample 15000 times? The 512*512 pixel field?
    That�s almost 15Gigs of data! And it means you need to process data at 12.8MB/s to finish it in 20 minutes. I hope you know C, x86 assembly and MMX.
    I would setup a huge circular buffer (NI-DAQ calls them �double buffers�), about 30 seconds worth or so, to use with SCAN_Start. Then I would proces
    s the actual buffer the card is DMA�ing the data into with a high priority thread. Progressively sum the scan values from the 16bit buffer (the samples are only 12 bit, but the buffer should still be 16bits wide) into a secondary buffer of DWORDs the size of the screen (512*512), and you�ll need two of those, one for each channel. Once the 15000 scans are complete, convert each entry into a float divide by 15000.0f, and store it in a third buffer of floats.
    If you wish to contract this out, send me an email at [email protected]

  • Fastest way to load csv into oracle BE table

    I have csv file which is having 10 million records in it . what is the fastest approach to load this data to oracle BE table .
    I am using Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production With the Partitioning, OLAP and Oracle Data Mining options .
    csv format
    first_name,last_name,occupation,address
    above all the fields are of varchar data type.
    I have tried to use external table while inserting its taking too much time .
    Thanks

    hi,
    You can use sql loader.
    options (skip=1)
    LOAD DATA
    INFILE 'csv file path'
    truncate /append
    INTO table table_name
    FIELDS TERMINATED BY ','
    OPTIONALLY ENCLOSED BY '"'
    trailing nullcols
    column name ,
    column name,
    column name
    {code}
    Create this file as ctl(control file.) on the lcoation where your csv file is.
    Execute it by goint to the path where your csv file resides.
    {code}
    cmd
    cd path where your csv file resides
    sqllder userid=username/password@database name control=your_control_file_name
    {code}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

  • Best way to load data from External data to ADC

    Hi Guy,
    I am new in BAM. I currently working for a existing system that have all it data in existing system, which i only have the view access to the database/table.
    so i just wonder, what is the best way to load the data so that it can populate to the active data cache?
    thanks.

    Hello,
    Use the EDS.For reference http://docs.oracle.com/cd/E14571_01/integration.1111/e10224/bam_extl_data_sources.htm
    Regards
    Siva Sankar

  • Fastest way to load a BufferedImage on a JPanel

    I am looking for the fastest way to display a BufferedImage on a JPanel.
    I am using JAI to take in photo files (JPG, BMP, GIF, TIFF, PNG) and create thumbnails (BufferedImage).
    I was reading through the forums and saw you can either
    1)overwrite the Graphics method or
    2)imageicon->JLabel->JPanel
    Currently, I am doing number 2, but I was wondering what the best way truly is.

    as you arn't doing any animation or that kind of thing, using Swing Components will work just fine.

  • Fastest way to access data between String and char[]

    Hi all,
    I've been programming a small String generator and was curious about the best and fastest way to do so.
    I've done it in two ways and now hope you can tell me if there is a "more java version" or fastest way between those two or if I'm totally wrong as some classe in the API already does that.
    Here are the codes:
    version 1:
            //describe the alphabet
         String alphabet = "abcdefghijklmnopqrstuvwxyz";
         //The "modifiable String"
         StringBuffer stringBuffer = new StringBuffer();
         //Just put the temporary int declaration outside the loop for performance reasons
         int generatedNumber;
         //Just do a "for" loop to get one letter and add it the String
         //Let's say we need a 8 letters word to be generated
         for (int i=0; i < 8; i++)
             generatedNumber = (int)(Math.random() * 26);
             stringBuffer.append(alphabet.charAt(generatedNumber));
         System.out.println(stringBuffer.toString());
         stringBuffer =null;version 2:
            //describe the alphabet
         char[] alphabetArray = {'a', 'b', 'c', 'd', 'e', 'f',
                        'g', 'h', 'i', 'j', 'k', 'l',
                        'm', 'n', 'o', 'p', 'q', 'r',
                        's', 't', 'u', 'v', 'w', 'x',
                        'y', 'z'}
         //The "modifiable String"
         StringBuffer stringBuffer = new StringBuffer();
         //Just put the temporary int declaration outside the loop for performance reasons
         int generatedNumber;
         //Just do a "for" loop to get one letter and add it the String
         //Let's say we need a 8 letters word to be generated
         for (int i=0; i < 8; i++)
             generatedNumber = (int)(Math.random() * 26);
             stringBuffer.append(alphabetArray[generatedNumber]);
         System.out.println(stringBuffer.toString());
         stringBuffer =null;Basically, the question is, what is the safest, fastest and more "to the rules" way to access a char in a sequence?
    Thanks in advance.
    Edited by: airchtit on Jan 22, 2008 6:02 AM

    paul.miner wrote:
    Better, use a char[] instead of a StringBuffer/StringBuilder, since you seem to know the size of the array in advance.
    Although I imagine making "alphabet" a char[] has slightly less overhead than making it a String
    1. It's a lot clearer to write it as a String.
    2. You can just call toCharArray() on the String to get a char[], instead of writing out each char individually.
    3. Or if you're going to be using a plain alphabet, just use (randomNumber + 'a')
    And use Random.nextInt()Hello and thx for the answers,
    I know I shouldn't put constants in my code, it was just a piece of code done in 1 minute to help a colleague.
    Even if it was just a one minute piece of code, I was wondering about the performance problem on large scale calculating, I mean something like a 25 characters word for billions of customers but anyway, once again, the impact should be minimal.
    By the way, I didn't know the Random Class (shame on me, I still don't know the whole API) and, I don't understand why I should be using this one more than the Random.Math which is static and thus take a few less memory and is easier to call.
    Is it because of the repartition factor?
    According to the API, the Random.Math gives (almost) a uniform distribution whether the Random.nextInt() gives a "more random" int.
    I

  • Fastest way to load an image into memory

    hi, ive posted before but i was kinda vague and didnt get much of a response so im going into detail this time. im making a java program that is going to contol 2 robots in a soccer competition. ive decided that i want to go all out and use a webcam instead of the usual array of sensors so the first step is to load an image into the memory. (ill work on the webcam once ive got something substanical, lol) since these robots have to be rather small (21cm in diameter) i can only use some pretty crappy processors. the robots are going to be both running gentoo linux on a 600 mhz processor, therefore it is absoleutely vital i have a fast method of loading and analyzing images. i need to be able to both load the image quickly, and more importainly analyze it quickly. ive looked at pixelgrabber which looks good, but ive read several things about javas image handling beging crap. these articles are a few years old, and im therefore wonding if there anything from the JAI that will do this for me. thx in advance.

    well i found out why i was having so much trouble
    installing JAI. im now back on windows and i cant
    stand it, so hopefully the bug will be fixed soon. oIt's not so bad. I mean, that's why we love Java! Once your linux problem is fixed you can just transfer your code there as is.
    well. i like the looks of JAI.create() but im not so
    sure how to use it. at this stage im tying to load an
    image from a file. i no to do this with pixelgrabber
    you use getcodebase(), but i dont know how to do it
    with JAI.create. any advice is appreciated. thx.Here are some example statements for handling a JAI image. There are other ways, I'm sure, but I've no idea which are faster, sorry. But this is quite fast on my machine.
    PlanarImage pi = JAI.create("fileload", imgFilename);
    WritableRaster wr = Raster.createWritableRaster(pi.getSampleModel(), null);
    int width = pi.getWidth(); // in case you need to loop through the image
    int height = pi.getHeight();
    wr = pi.copyData(); // copy data from the planar image to the writable one
    wr.getPixel(w,h,pixel); //  pixel is an int array with three elements.
    int red = pixel[0];     // to find out what's the red component
    int[] otherPixel = {0,0,0}
    wr.setPixel(w,h,otherPixel); // make pixel at location (w,h) black.                And here's a link with sample code using JAI. I've tried several of the programs there and they work.
    https://jaistuff.dev.java.net/

  • What is the fastest way to pass data between parallel threads?

    I have a top level vi running with 6 parallel threads. I need to pass some data from digital I/O to several of the threads. What is the fastest responding way to do this. I am controlling a machine that has quite a few sensed events happening at very close intervals, some as close together as 1 to 2 milliseconds, and I seem to be randomly missing the signal from these sensors. How can I distribute the I/O to the different threads and not miss any inputs?

    I usually use a Queue to pass data from one loop to another. Other
    choices are Functional Globals or Notifiers. It kind of depends on what
    you need to do as to which one is best, so it's a bit hard to recommend
    one over the others without knowing more about your application.
    Both Queues and the Functional Globals (if written correctly) can
    buffer data so you're less likely to lose data if one loop gets behind
    the others.
    Ed
    Ed Dickens - Certified LabVIEW Architect - DISTek Integration, Inc. - NI Certified Alliance Partner
    Using the Abort button to stop your VI is like using a tree to stop your car. It works, but there may be consequences.

  • Fastest Way to Load Text Files Containing Thousands of Words

    Hi, can anyone give me suggestions for fast loading of files containing many bytes in the form of a single word on a different line. The average file size is 45 kb. I'm currently splitting the words into separate files each containing 4000 words a piece. I use a different thread to read each file and then add all of the words into a single vector, formed by joining the vectors of each reader thread. Any suggestions for speeding up the process? (Currently, it takes about 30 seconds to load 52,000 words). My OS is Windows2000

    Something seems wrong. There is no way that what you describe should take anywhere close to that amount of time, unless the computer is a constraint. I created a file of 56,000 lines, each line consisted of the byte string "abcdefgh" plus a \r\n,
    for a total of 10 bytes per line - a total file size of 560,000 bytes (547 kb)
    The following program read the file, created an ArrayList, and added the 56,000 lines to the list in less than 1 second, about what I expected. You might want to run it and see what you get.
    import java.io.*;
    import java.util.*;
    class Zxin
        public static void main(String[] args)
            throws IOException
            FileInputStream fis = new FileInputStream(
                "C:/Documents and Settings/Chuck/My Documents/Java/junk/word.txt");
            InputStreamReader isr = new InputStreamReader(fis);
            BufferedReader br = new BufferedReader(isr);
            List<String> list = new ArrayList<String>();
            String data;
            while ((data = br.readLine()) != null)
                list.add(data);
            System.out.println("The list contains " + list.size() + " entries");
            System.out.println("The first entry in the list is: " + list.get(0));
            br.close();
    }It printed these 2 lines:
    The list contains 56000 entries
    The first entry in the list is: abcdefgh

  • Fastest way to get data from Multiple lists across multiple site collections

    HI
    I need to get data from multiple lists which spread across 20 site collections and need to show it as list view.
    I have searched on internet about this and got some info like options would be to use search core APIs or BCS . I can't use search because I want real time data. Not sure of any other ways.
    if anybody can provide ideas it would be help.

    Might LINQ be an option for you?  Using
    LINQPad and the
    SharePoint Connector, you should be able to write a query that'll retrieve this data, from which you can tabulate it.  I'm not sure how you'd be able to automate this any further so that it's then imported in as list.
    For something more specific, I used a third party tool called the
    Lightning Tools Lightning Conductor, which is essence a powerful content roll-up tool.  In one of my solutions, I created a calculated column that gave an order / ranking on each item, so that when lists were combined, they'd still have some form of
    order.  The web part is also fairly customisable and has always proven a useful tool.
    Hope that helps.
    Steven Andrews
    SharePoint Business Analyst: LiveNation Entertainment
    Blog: baron72.wordpress.com
    Twitter: Follow @backpackerd00d
    My Wiki Articles:
    CodePlex Corner Series
    Please remember to mark your question as "answered" if this solves (or helps) your problem.

Maybe you are looking for

  • DVD drive failure on some DVD brands

    Hello, I've been using Taiou Yuden DVD-R's for a while on my late 2009 Mac mini (Pioneer DVD-RW DVRTS08). Now suddenly they fail to burn (Toast Titanium 11.1) with errors like: The drive reported an error: Sense Key = HARDWARE ERROR (or MEDIUM ERROR)

  • How do I get back to my basic start page from any ...

    How do I get back to my basic start page from any other page? There should be a one click link but I can't see how to do it. Apologies for illiteracy. Like say from this help page, how can I get back on skype and make a call?

  • Problem with wireless nenetwork

    Can someone please help me set up this wireless network on my 6500 all in one.  It worked before and suddenly lost its mind.  I am unable to sync the ip addresses and/or do that MAC thing.  It is being used on a stupid verizon "D" router thing, and I

  • PE8: Distorted Video Preview

    Clips I captured from my DV Camcorder render properly but are distorted while playing in preview window.  Any ideas on how to fix?  Thanks.

  • N73 Music Edition A2DP Problem Inspite of FW: v4.0...

    I am using a Motorola "Snoop" S9 A2DP headset with m N73 with the latest FW :v4.0736.3.2.1 (04/09/07). The Problem I face is that whenever I listen to music when I am on the move (on my Motorcycle or Walking)the music Skips as if the Bluetooth connec