How to handle line break embeded inside CSV column

Question

Hi there,
I am under the pressure to make it work. I already put this question on APEX forum, but on second thought, I think it relates more to PL/SQL rather than APEX since APEX 4.1 already have utility to handle CSV Upload.
If you read it already in APEX forum, please ignore.
I am sorry for that. Thanks for reading.
I need to develop an app that allows user to upload CSV file to a interface table.
The APEX version at my workplace is 4.0.2.
I used the code from
http://dbswh.webhop.net/htmldb/f?p=BLOG:READ:0::::ARTICLE:11000346061523
It all works well till recently I find out
If a column in a CSV file cotain a line break (or new line) e.g. (The tester copy and paste this text which has line break into a column in a spreadsheet)
This is the first sentence.
This is the second sentence.
It will break the “This is the second sentence”. To a new column.
The contents of the CSV viewed in Notepad look as below
Assessment Date,Scheduled Date,Assessment Provider,Assessor Name,Court,First Name,Middle Name,Last Name,PRN Person Record Number,NHI Number,Defendant Attended Y/N,Is Dependent Y/N,Notes,Primary Ethnicity,"Ethnicity Other, please specify",Gender,Currently in Treatment Y/N,Substance of Concern 5,Other Substance Specified
22/09/2012,,Provider Co Name,Warren Edgley,Wellington,,,Salty,2545554,dgsdf,ergerg,,"This is the first sentence.
This is the second sentence.",Japanese,,Female,b,,
Here is the code from the CSV UTIL, please help me to find out how can I replace the line break to a space so that the uploading process is correct.
  CREATE OR REPLACE PACKAGE BODY "CSV_UTIL"
AS
     --strip the beginning and the end quotes, then replace double quotation with single
   FUNCTION de_quote (p_str IN VARCHAR2, p_enc_by IN VARCHAR2)
      RETURN VARCHAR2
   IS
   v_str VARCHAR2(32767) := p_str;
   BEGIN
      IF (p_enc_by IS NULL)
      THEN
         RETURN p_str;
      ELSE
        IF SUBSTR(p_str,-1) = p_enc_by THEN
           v_str := SUBSTR(p_str,1,LENGTH(p_str)-1);
        END IF;
        IF SUBSTR(p_str,1,1) = p_enc_by THEN
           v_str := SUBSTR(v_str,2);
        END IF; 
        RETURN REPLACE (v_str,
                         p_enc_by || p_enc_by,
                         p_enc_by
      END IF;
   END de_quote;
   PROCEDURE parse (p_str IN VARCHAR2, p_enc_by IN VARCHAR2, p_sep IN VARCHAR2)
   IS
      l_n          NUMBER   DEFAULT 1;
      l_in_quote   BOOLEAN  DEFAULT FALSE;
      l_ch         NCHAR (1);
      l_len        NUMBER   DEFAULT NVL (LENGTH (p_str), 0);
   BEGIN
      IF (l_len = 0)
      THEN
         RETURN;
      END IF;
      g_words := g_empty;
      g_words (1) := NULL;
      FOR i IN 1 .. l_len
      LOOP
         l_ch := SUBSTR (p_str, i, 1);
         IF (l_ch = p_enc_by)
         THEN
            l_in_quote := NOT l_in_quote;
         END IF;
         IF (l_ch = p_sep AND NOT l_in_quote)
         THEN
            l_n := l_n + 1;
            g_words (l_n) := NULL;
         ELSE
            g_words (l_n) := g_words (l_n) || l_ch;
         END IF;
      END LOOP;
      g_words (l_n) := de_quote (g_words (l_n), CHR(10));
      g_words (l_n) := de_quote (g_words (l_n), CHR(13));
      FOR i IN 1 .. l_n
      LOOP
         g_words (i) := de_quote (g_words (i), p_enc_by);
      END LOOP;
   END parse;
Author: Oleg Lihvoinen
Company: DbSWH
Changes:
10.02.2011, There was a miscalculation of the file line last position in case it is the end of file
   PROCEDURE upload (p_file_name VARCHAR2, p_collection_name VARCHAR2, p_enc_by IN VARCHAR2, p_sep_by IN VARCHAR2, p_rows NUMBER)
   IS
      v_blob_data    BLOB;
      v_clob_data    CLOB;
      v_clob_len     NUMBER;
      v_position     NUMBER;
      v_char         NCHAR (1);
      c_chunk_len    NUMBER           := 1;
      v_line         VARCHAR2 (32767) := NULL;
      v_data_array   vcarray;
      v_rows         NUMBER           := 0;
      n_seq          NUMBER           := 1;
      dest_offset    NUMBER           := 1;
      src_offset     NUMBER           := 1;
      amount         INTEGER          := DBMS_LOB.lobmaxsize;
      blob_csid      NUMBER           := DBMS_LOB.default_csid;
      lang_ctx       INTEGER          := DBMS_LOB.default_lang_ctx;
      warning        INTEGER;
      l_sep          VARCHAR2(100)    := CASE WHEN p_sep_by = '\t' THEN chr(9) ELSE p_sep_by END;
   BEGIN
      htmldb_collection.create_or_truncate_collection
                                      (p_collection_name      => p_collection_name);
      -- Read blob from wwv_flow_files
      SELECT blob_content
        INTO v_blob_data
        FROM wwv_flow_files
       WHERE NAME = p_file_name;
      v_position := 1;
      DBMS_LOB.createtemporary (lob_loc      => v_clob_data,
                                CACHE        => TRUE,
                                dur          => DBMS_LOB.SESSION
      DBMS_LOB.converttoclob (v_clob_data,
                              v_blob_data,
                              amount,
                              dest_offset,
                              src_offset,
                              blob_csid,
                              lang_ctx,
                              warning
      v_clob_len := DBMS_LOB.getlength (v_clob_data);
      IF v_clob_len = 0 THEN
         RETURN;
      END IF;
      WHILE (v_position <= v_clob_len + 1)
      LOOP
         v_char := DBMS_LOB.SUBSTR (v_clob_data, c_chunk_len, v_position);
         v_line := v_line || v_char;
         v_position := v_position + c_chunk_len;
         -- When the whole line is retrieved and not end of file or end of file
         IF v_char = CHR (10) AND v_position < v_clob_len OR v_position = v_clob_len + 1
         THEN
            parse (p_str => v_line, p_enc_by => p_enc_by, p_sep => l_sep);
            v_data_array := g_words;
            FOR i IN 1..g_words.count LOOP
               IF i <= 50 THEN
                  v_data_array(i) := g_words(i);
               ELSE
                  exit;
               END IF;
            END LOOP;
            FOR i IN g_words.count + 1..50 LOOP
               v_data_array(i) := null;
            END LOOP;           
            v_rows := v_rows + 1;
            -- exit if uploaded specified number of rows
            IF p_rows IS NOT NULL AND v_rows > p_rows THEN
               EXIT;
            END IF;
            -- Store data to collection
            n_seq :=
               htmldb_collection.add_member
                                     (p_collection_name      => p_collection_name,
                                      p_c001                 => v_data_array
                                                                           (1),
                                      p_c002                 => v_data_array
                                                                           (2),
                                      p_c003                 => v_data_array
                                                                           (3),
                                      p_c004                 => v_data_array
                                                                           (4),
                                      p_c005                 => v_data_array
                                                                           (5),
                                      p_c006                 => v_data_array
                                                                           (6),
                                      p_c007                 => v_data_array
                                                                           (7),
                                      p_c008                 => v_data_array
                                                                           (8),
                                      p_c009                 => v_data_array
                                                                           (9),
                                      p_c010                 => v_data_array
                                                                           (10),
                                      p_c011                 => v_data_array
                                                                           (11),
                                      p_c012                 => v_data_array
                                                                           (12),
                                      p_c013                 => v_data_array
                                                                           (13),
                                      p_c014                 => v_data_array
                                                                           (14),
                                      p_c015                 => v_data_array
                                                                           (15),
                                      p_c016                 => v_data_array
                                                                           (16),
                                      p_c017                 => v_data_array
                                                                           (17),
                                      p_c018                 => v_data_array
                                                                           (18),
                                      p_c019                 => v_data_array
                                                                           (19),
                                      p_c020                 => v_data_array
                                                                           (20),
                                      p_c021                 => v_data_array
                                                                           (21),
                                      p_c022                 => v_data_array
                                                                           (22),
                                      p_c023                 => v_data_array
                                                                           (23),
                                      p_c024                 => v_data_array
                                                                           (24),
                                      p_c025                 => v_data_array
                                                                           (25),
                                      p_c026                 => v_data_array
                                                                           (26),
                                      p_c027                 => v_data_array
                                                                           (27),
                                      p_c028                 => v_data_array
                                                                           (28),
                                      p_c029                 => v_data_array
                                                                           (29),
                                      p_c030                 => v_data_array
                                                                           (30),
                                      p_c031                 => v_data_array
                                                                           (31),
                                      p_c032                 => v_data_array
                                                                           (32),
                                      p_c033                 => v_data_array
                                                                           (33),
                                      p_c034                 => v_data_array
                                                                           (34),
                                      p_c035                 => v_data_array
                                                                           (35),
                                      p_c036                 => v_data_array
                                                                           (36),
                                      p_c037                 => v_data_array
                                                                           (37),
                                      p_c038                 => v_data_array
                                                                           (38),
                                      p_c039                 => v_data_array
                                                                           (39),
                                      p_c040                 => v_data_array
                                                                           (40),
                                      p_c041                 => v_data_array
                                                                           (41),
                                      p_c042                 => v_data_array
                                                                           (42),
                                      p_c043                 => v_data_array
                                                                           (43),
                                      p_c044                 => v_data_array
                                                                           (44),
                                      p_c045                 => v_data_array
                                                                           (45),
                                      p_c046                 => v_data_array
                                                                           (46),
                                      p_c047                 => v_data_array
                                                                           (47),
                                      p_c048                 => v_data_array
                                                                           (48),
                                      p_c049                 => v_data_array
                                                                           (49),
                                      p_c050                 => v_data_array
                                                                           (50)                                                                          
            -- Clear the line
            v_line := NULL;
         END IF;
      END LOOP;
   END;
END;In my apps, I save these straight into a table rather than an APEX collection because the number of columns can be longer than 50.
I want to find out how can replace these line break inside a column to a space.
If any one has any ideas, please let me know.
Thanks a lot in advance.
Ann

Answer 1

Ann586341 wrote:
I think the code split the whole thing by this line
-- When the whole line is retrieved and not end of file or end of file
IF v_char = CHR (10) AND v_position < v_clob_len OR v_position = v_clob_len + 1
THEN
Yes, exactly. That piece of code believes all CHR(10) occurences are record delimiters.
It is not smart enough to recognize that a CHR(10) within quotation marks are part of the data.
Optimally a solution should keep the CHR(10) rather than replacing with spaces, but that will be a bigger rewrite of the UTL_CSV code ;-)
If you are happy with replacing with spaces, a "simple" solution could be something like:
Declare a boolean variable in upload procedure:
v_within_text_column   boolean := false;And use it like this:
      WHILE (v_position <= v_clob_len + 1)
      LOOP
         v_char := DBMS_LOB.SUBSTR (v_clob_data, c_chunk_len, v_position);
         IF v_char = '"' THEN
           v_within_text_column := NOT v_within_text_column;
         ELSIF v_char = CHR(10) AND v_within_text_column THEN
           v_char := ' ';
         END IF;
         v_line := v_line || v_char;
         v_position := v_position + c_chunk_len;
         -- When the whole line is retrieved and not end of file or end of file
         IF v_char = CHR (10) AND v_position < v_clob_len OR v_position = v_clob_len + 1
         THEN
           v_within_text_column := false; -- To be safe always set this on "true" linebreaks
{code}
+(This is untested code just written here in the text editor.)+
It should work by toggling a flag whether you are "within" the quotes or not and then replacing CHR(10) with a space if you are within a text column.
This way we avoid having to go through the clob more than once (it is enough that this code walks the clob one character at a time...)
It will not handle if the clob contains situations like:
{code}
abc,123,"This is a text with a quote from a man who said \"To Be,
or Not To Be\" some hundred years ago",123,xyz
{code}
Escaped quotes would need separate attention ;-)