Reading from .CSV and storing it into a collection

Question

Hi folks,
Is there a way to make a dynamic procedure to work with .CSV documents and store it into a collection? For example you have to make a procedure to read from .CSV but users upload 10 different version that have different number of columns.
Normally I would define a record type to match those columns and store it into a collection. However if I don't know the number of columns I would need to define 10 records in advance which I am trying to avoid.
Problem is I cant define SQL elements on the fly. Meaning on production I don't have the rights to dynamically create a table to match my columns and then drop the table after I no longer need it so I need to store data into a collection.
And the last option where I would loop through the document and then do the operations I need is not good since the document is a part of other procedures that write and read from it. The idea is to pick the data, store it into a collection, close the file and then work with it.
This is what I got so far:
declare
  -- Variables
  l_file      utl_file.file_type;
  l_line      varchar2(10000);
  l_string    varchar2(32000);
  l_delimiter varchar2(10);
  -- Types
  type r_kolona is record(
    column_1 varchar2(500)
   ,column_2 varchar2(500)
   ,column_3 varchar2(500)
   ,column_4 varchar2(500)
   ,column_5 varchar2(500));
  type t_column_table is table of r_kolona;
  t_column    t_column_table := t_column_table();
begin
  /*Define the delimiter*/
  l_delimiter := ';';
  /*Open file*/
  l_file      := utl_file.fopen( 'some dir', 'some.csv', 'R');
  /*Takes first row of document as header*/
  utl_file.get_line( l_file, l_line);
  loop
    begin
      utl_file.get_line( l_file, l_line);
      /*Delete newline operator*/
      l_string                         := rtrim( l_line, chr(13)) || l_delimiter;
      /*Extend array and insert parsed values */
      t_column.extend;
      t_column(t_column.last).column_1 := substr( l_string, 1, instr( l_string, l_delimiter, 1, 1) - 1);
      t_column(t_column.last).column_2 := substr( l_string, instr( l_string, l_delimiter, 1, 1) + 1, instr( l_string, l_delimiter, 1, 2) - instr( l_string, l_delimiter, 1, 1) - 1);
      t_column(t_column.last).column_3 := substr( l_string, instr( l_string, l_delimiter, 1, 2) + 1, instr( l_string, l_delimiter, 1, 3) - instr( l_string, l_delimiter, 1, 2) - 1);
      t_column(t_column.last).column_4 := substr( l_string, instr( l_string, l_delimiter, 1, 3) + 1, instr( l_string, l_delimiter, 1, 4) - instr( l_string, l_delimiter, 1, 3) - 1);
      t_column(t_column.last).column_5 := substr( l_string, instr( l_string, l_delimiter, 1, 4) + 1, instr( l_string, l_delimiter, 1, 5) - instr( l_string, l_delimiter, 1, 4) - 1);
    exception
      when no_data_found then
        exit;
    end;
  end loop;
  /*Close file*/
  utl_file.fclose(l_file);
  /*Loop through collection elements*/
  for i in t_column.first .. t_column.last
  loop
    dbms_output.put_line(
         t_column(i).column_1
      || ' '
      || t_column(i).column_2
      || ' '
      || t_column(i).column_3
      || ' '
      || t_column(i).column_4
      || ' '
      || t_column(i).column_5);
  end loop;
exception
  when others then
    utl_file.fclose(l_file);
end; Stupid version would be to define a record with 50 elements and hope they dont nuke the excel with more columns :)
Best regards,
Igor

Answer 1

Igor S. wrote:
Use some to query data and then fix wrong entries on prod (insert, update, delete). Manipulate with some and then make new reports. The first that come to mind but basicly is to write a procedure that can be used for ANY .csv so I dont have to rewrite the code.This is logically wrong and smacks of poor design.
You're wanting to take CSV files with various unknown formats of data, read that data into some generic structure, and then somehow magically be able to process the unknown data to be able to "fix wrong entries". If everything is unknown... how will you know what needs fixing?
Good design of any system stipulates the structures that are acceptable, and if that means you know there are just 20 possible CSV formats and you can implement a mechanism to determine which format a particular CSV is in (perhaps something in the filename?) then you will create 20 known targets (record structures/tables or whatever) to receive that data into, using 20 external tables, or procedure or whatever is necessary.
Doing anything other than that is poor design, leaves the code open to breaking, is non-scalable, hard to debug, and just wrong on so many levels. This isn't how software is engineered.
For example you have 20 developers that have to work with .CSV files. So when someone has to work with a .CSV he would call a procedure with parameters directory and file name. And as a out parameter would get a collection with .CSV stored inside.As others have mentioned, give the developers an Apex application for their data entry/manipulation, working directly on the database with known structures and validation so they can't create "wrong" data in the first place. They can then export that as .CSV data for other purposes if really required.