How to make dataload fast into a table having CONTEXT index on CLOB column?

Question

Hi all,
I have a table with the follwing structure :
   CREATE TABLE file_table (
       file_id    NUMBER(10) PRIMARY KEY,
       file_name  VARCHAR2(3000),
       file_data CLOB);I am loading around 50,000 files into the table using SQL*LOADER.
LOAD DATA
INFILE * BADFILE  'd:\mydata.bad' DISCARDFILE 'd:\mydata.dis'
INTO TABLE file_table
APPEND
FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY '"'
file_name  CHAR(1000),
file_data  LOBFILE (file_name) TERMINATED BY EOF
BEGINDATA
D:\TEXT\9701.1\00\00\01.txt
D:\TEXT\9701.1\01\01\03.txt
D:\TEXT\9701.2\02\02\04.txt
D:\TEXT\9701.3\03\03\05.txt
..after this I am creating Text index :
CREATE INDEX file_table_idx ON file_table (file_data)
INDEXTYPE IS CTXSYS.CONTEXT online 
PARAMETERS('filter ctxsys.null_filter LEXER mylex NOPOPULATE')
PARALLEL 2;       Here my problem is:
For the first time without creating index when I am trying to load 50000 files through SQL*LOADER conventional loading it is loading in 1 min 38 seconds.
After that for creating index around : 11 min 12 seconds.
But if I am loading another set of 10000 files after creating index, SQL*LOADER is taking considerable amount of time.
For example :
Index created with SYNC(ON COMMIT) : 28 min 45 sec
Index created with SYNC(MANUAL) : 8 min 34 sec
Index created with NOPOPULATE : 2 min 36 sec + 8 min 47 sec to rebuild the index.
Please suggest me the best approach so that further load of files to the table also should take comparatively same time as the first load of 50000 files.
Please suggest me how to create index to reduce load time.

Answer 1

Hi all,
I tried to load files using the below procedure also:
CREATE OR REPLACE PROCEDURE load_file_to_my_docs (p_file_name  IN  my_files.name%TYPE) AS
  v_bfile      BFILE;
  v_clob       CLOB;
BEGIN
  INSERT INTO my_files (id, name, doc)
  VALUES (my_files_seq.NEXTVAL, p_file_name, empty_clob())
  RETURN doc INTO v_clob;
  v_bfile := BFILENAME('SAMPLEDATA', p_file_name);
  Dbms_Lob.Fileopen(v_bfile, Dbms_Lob.File_Readonly);
  Dbms_Lob.Loadfromfile(v_clob, v_bfile, Dbms_Lob.Getlength(v_bfile));
  Dbms_Lob.Fileclose(v_bfile);
  COMMIT;
END;
  But I found SQL*LOADER is loading fast compartive to this procedure.
So please suggest me to reduce the time taking to load data once after creating TEXT index.