Solution to execute sql from mysql application logs

Question

We have some custom mysql application, which generate logs that are stored on common location on open linux box which has below format
$ cat minor20100809.log
20100809001^Aselect count(prod_id) from product_logs;
20100809002^Aselect count(order_id) from order_logs;
ID and the SQL statement with control A as seperater and daily 1000 logs files are generated
Now we have one oracle DWH, where we import all the mysql data and we need to test these counts on daily basis.
Could you please suggest a solution in oracle for this?
at high level we need to follow below steps
a) Import these logs in a table --> which is better way sqlldr, impdp, or plsql read file ?
b) execute these sql statement one by one
c) store the counts in log table on daily basis

Answer 1

Hi,
It is a simple two step process.
Step 1. Convert your log files statements to meaningful oracle statements. like removing prefixes 20100809001^A from each line.
This can be done in two ways.
(a). Using OS editors OR
(b). Following PL/SQL code...
-- Connect as sysdba
create or replace directory dir_tmp as '/tmp';                        -- If oracle server is on Unix
create or replace directory dir_tmp as 'C:\your_folder';            -- If oracle server is on Windows
-- Grant permission to your_schema
grant read, write on directory dir_tmp to your_schema;    Removing prefixes using:
       SELECT regexp_replace (vNewLine, '([[:alnum:]]{1,})\^A', '\2')
         INTO vStmt
         FROM DUAL;
CREATE OR REPLACE PROCEDURE convert_to_sql_stmts
( input_file     in varchar2,
  output_file   in varchar2
AUTHID CURRENT_USER
IS
    InFile   utl_file.file_type;
    OutFile  utl_file.file_type;
    vNewLine VARCHAR2(4000);
    vStmt    VARCHAR2(4000);
    i        PLS_INTEGER;
    j        PLS_INTEGER := 0;
    SeekFlag BOOLEAN := TRUE;
BEGIN
  -- open a file to read
  InFile := utl_file.fopen(dir_tmp, input_file,'r');
  -- open a file to write
  OutFile := utl_file.fopen(dir_tmp, output_file, 'w');
  -- if the file to read was successfully opened
  IF utl_file.is_open(InFile) THEN
    -- loop through each line in the file
    LOOP
      BEGIN
        utl_file.get_line(InFile, vNewLine);
        i := utl_file.fgetpos(InFile);
        dbms_output.put_line(TO_CHAR(i));
        SELECT regexp_replace (vNewLine, '([[:alnum:]]{1,})\^A', '\2')
          INTO vStmt
          FROM DUAL;
        utl_file.put_line(OutFile, vStmt, FALSE);
        utl_file.fflush(OutFile);
        IF SeekFlag = TRUE THEN
          utl_file.fseek(InFile, NULL, -30);
          SeekFlag := FALSE;
        END IF;
      EXCEPTION
        WHEN NO_DATA_FOUND THEN
          EXIT;
      END;
    END LOOP;
    COMMIT;
  END IF;
  utl_file.fclose(InFile);
  utl_file.fclose(OutFile);
EXCEPTION
  WHEN OTHERS THEN
    RAISE_APPLICATION_ERROR (-20099, 'Unknown UTL_FILE Error');
END convert_to_sql_stmts;
Step 2. Directly executing above output log file
Use spool to generate the count....
    set termout off
    set feedback off
    set trimspool on
    set pagesize 0
    set timing OFF
    spool   count.log
    @new_file.log
    spool off;Hope this helps...