Creating Packages from BLOB field contents, line wrap problems

Question

Good afternoon,
we use an in-house developed database-driven update system to update both our databases and filesystems, the system itself performs really well but there is still one problem I can't fix without some expert help:
the code of to-be-updated Oracle packages is stored inside of a BLOB field, the BLOB field will contain both the package specification and package body and needs to be split into two parts to first execute the spec and then to execute the package body (I tried to execute both in a single step but this didn't work). This works for packages with less than 32767 characters and also works in some other cases but I found one case where the executed code contains an extra line wrap right in the middle of a word.
To make it more clear (I hope it's comprehensible), imagine the following database content:
CREATE OR REPLACE Package
MyPack
AS
[... a lot procedure headers ...]
END MyPack;
CREATE OR REPLACE
Package Body MyPack AS
[... a lot more procedures ...]
PROCEDURE myTest (intID OUT integer)
AS
BEGIN
  SELECT count (*) into intID FROM MyTa[--this is where the dbms_lob.substr() ends --]ble;
END;
END MyPack;My code searches for the second occurrence of the "Create or replace package", splits the code into spec and body, executes the specification and keeps on adding the rest of the code to a VARCHAR2A variable called "storedCode" from the BLOB. Now in the above example, after the specification has been removed from the string, the remaining characters (ending with the "MyTa" string) are added to the varchar2a variable, the next line is fetched from the BLOB via "dbms_lob.substr()" and added as long as dbms_lob.substr() does not return a NULL value (end of BLOB). When the code is executed after all has been fetched, the generated Package Body will contain an extra line wrap right in the middle of the "MyTable" word compiling the package invalid.
This is the procedure code I use (definitely improvable, I'm better in MSSQL and MySQL dialects ...) to load, parse and execute the BLOB content:
   -- to run package code
  procedure runPackageCode (stepRef integer)
  AS
    numLines integer default 1;
    pos     integer default 1;
    storedCode    LOG_CHANGEDOBJECT.STOREDOBJECT%type;
    objectCursor  integer;
    lSqlOut     integer;
    sqlCommand  dbms_sql.varchar2a;
    emptyCommand dbms_sql.varchar2a;
    pIsError integer := 0;
    pErrorMsg varchar2(200) := '';
    updateRef integer := 0;
    currentUpdate integer := 0;
    schemaReference varchar2(20);
    -- required to do string cutting
    strLine varchar2(32767);
    strLeftFromSlash varchar2(32767);
    strRemaining varchar2(32767);
    intDelimiterPos integer := 0;
  begin
    -- retrieve update ID
    SELECT log_update_ref INTO currentUpdate FROM link_update_with_taskstep WHERE log_taskstep_ref = stepRef;
     begin
        select storedobject, change_area
        into storedCode, schemaReference
        from vw_storedobjects
        where step_id = stepRef;
     exception
      when no_data_found then
        pIsError := 1;
        pErrorMsg := 'Invalid SQL ID ' || stepRef;
        pkg_generic.LogError(updateRef, 'LocalUpdater', stepRef, 'Run package code failed: ' || pErrorMsg);
     end;
      if pIsError = 0 then     
        begin
          -- change schema
          execute immediate 'alter session set current_schema = ' || schemaReference;         
          objectCursor := dbms_sql.open_cursor;   
          loop
            strLine := UTL_RAW.CAST_TO_VARCHAR2(dbms_lob.substr(storedCode, 32767, pos));
            intDelimiterPos := regexp_instr(strLine, '\s*Create\s*or\s*Replace\s*Package', 2, 1, 0, 'i');
            while intDelimiterPos > 0
            loop
              -- '/' found, execute currently stored statement
              strLeftFromSlash := substr(strLine, 1, intDelimiterPos-1);
              strLine := substr(strLine, intDelimiterPos);
              -- execute the extracted part without any '/' in it
              sqlCommand(numLines) := regexp_replace(strLeftFromSlash, '(^|\s+)/(\s+|$)', '', 1, 0, 'm');
              if (sqlCommand(numLines) is not null) then
                objectCursor := dbms_sql.open_cursor;   
                dbms_sql.parse(objectCursor, sqlCommand, 1, numLines, true, dbms_sql.native);
                lSqlOut := dbms_sql.execute(objectCursor);
                dbms_sql.close_cursor(objectCursor);
              end if;
              -- reset sqlCommand
              sqlCommand := emptyCommand;
              -- reset line counter and store remaining string
              numLines := 1;
              -- check for further '/'s           
              intDelimiterPos := regexp_instr(strLine, '\s*Create\s*or\s*Replace\s*Package', 2, 1, 0, 'i');
            end loop;
            -- add the remaining strLine to the sqlCommand
            strLine := regexp_replace(strLine, '(^|\s+)/(\s+|$)', '', 1, 0, 'm');
   --> I assume this line breaks the code, lpad()'ing the content to move it to the end of a varchar2a line didn't help
            sqlCommand(numLines) := strLine;
            exit when sqlCommand(numLines) is null;
            pos := pos+32767;
            numLines := numLines+1;
          end loop;
          objectCursor := dbms_sql.open_cursor;   
          dbms_sql.parse(objectCursor, sqlCommand, 1, numLines, true, dbms_sql.native);   
          lSqlOut := dbms_sql.execute(objectCursor);
          dbms_sql.close_cursor(objectCursor);
          commit;
          -- reset schema
          execute immediate 'alter session set current_schema = UPDATE_DB';
          -- set state to installed
          pkg_update.setstepstate(stepRef, 'Installed');
    exception
    when others then
          -- reset schema
          execute immediate 'alter session set current_schema = UPDATE_DB';
          -- set state to installFailed
          pkg_update.setstepstate(stepRef, 'InstallFailed');
          pkg_generic.LogError(updateRef, 'Database', stepRef, 'Run package code failed: ' || sqlerrm);
    end;
    end if;
  END;    Thanks if you kept on reading so far, I would really appreciate any feedback!
Regards, Sascha

Answer 1

Welcome to the forum!
Whenever you post provide your 4 digit Oracle version (result of SELECT * FROM V$VERSION).
Thanks for providing an easy-to-understand problem statement and for using code tags.
>
the code of to-be-updated Oracle packages is stored inside of a BLOB field
>
This should be stored in a CLOB since it is character data. Why are you using BLOB?
>
the BLOB field will contain both the package specification and package body and needs to be split into two parts to first execute the spec and then to execute the package body
>
Good, clear problem statement. So why doesn't your code do just what you said it should do: 1) split the code into two parts, 2) execute the spec and 3) execute the body.
Instead of writing code that does these three relatively simple steps your code tries to combine splitting and executing and mushes/mashes it all together. The result, as you found, is code that is hard to understand, hard to debug, doesn't work and doesn't report on what it is doing.
Code like this doesn't have a performance issue so the code should implement the simple step-by-step process that you so elegantly stated in your problem description:
1. split the code into two parts
2. execute the spec
3. execute the body
My advice is to refactor your code to perform the above steps in the proper order and to add proper exception handling and reporting for each step. Then when a step isn't working you will know exactly where and what the problem is.
Here are my recommendations.
1. Add two CLOB variables - one will hold the spec, the second will hold the body
2. Add a comment (you have some good ones in the code now) for every step no matter how trivial it may be
3. Add exception/error handling to EVERY STEP
Your code for the first step has a comment but no exception handling. What should happen if you don't get any data? Why aren't you validating the data you get? Dynamic SQL using table-driven data is great, I love it, but you MUST validate that the data you get is what you expect to get.
    -- retrieve update ID
    SELECT log_update_ref INTO currentUpdate FROM link_update_with_taskstep WHERE log_taskstep_ref = stepRef;Recommended
    -- step 1 - retrieve update ID - This is the id that determines BLAH, BLAH, BLAH - add appropriate to tell a new developer what this ID is and what it means.
    BEGIN
        SELECT log_update_ref INTO currentUpdate FROM link_update_with_taskstep WHERE log_taskstep_ref = stepRef;
    EXCEPTION
        WHEN ??? THEN -- what should happen if step 1 fails? Do it here - don't default to an exception handler that is several pages away.
    END;Your code
     begin
        select storedobject, change_area
        into storedCode, schemaReference
        from vw_storedobjects
        where step_id = stepRef;
     exception
      when no_data_found then
        pIsError := 1;
        pErrorMsg := 'Invalid SQL ID ' || stepRef;
        pkg_generic.LogError(updateRef, 'LocalUpdater', stepRef, 'Run package code failed: ' || pErrorMsg);
     end;
Good - except there is no comment that says what this does - I assume that the query above and this block are the 'retrieve update ID ' step?
You log an error message and set the error flag to '1'. But since you don't want to continue why aren't you exiting the procedure and returning right here?
      if pIsError = 0 then     
        beginSo now you check the error flag and do several pages of code if there were no errors.
I don't like that 'inverted' logic.
If you don't want to continue then STOP right now! Don't make a developer scan through pages and pages of code to find out you really aren't doing anything else if there was an error.
Either put a RETURN statement in the exception handler above or change your code to
      if pIsError = 1 then     
        RETURN;
      end if;Now the rest of the code doesn' t have to be indented; you will never get there if there is an error. Check for errors after every step and exit right then as appropriate.
          -- change schema
          execute immediate 'alter session set current_schema = ' || schemaReference;         
          objectCursor := dbms_sql.open_cursor;   
          loop
            strLine := UTL_RAW.CAST_TO_VARCHAR2(dbms_lob.substr(storedCode, 32767, pos));
            intDelimiterPos := regexp_instr(strLine, '\s*Create\s*or\s*Replace\s*Package', 2, 1, 0, 'i');
            while intDelimiterPos > 0
            loopThis code mixes all of the steps together into one giant mess. You open a cursor, try to split the BLOB into spec and body and try to parse and execute both all within a double nested loop.
Even if that works correctly another developer will have a hard time understanding what the code is doing and fixing it if it goes wrong. And it will go wrong if you let me test if for you because I will put garbage into the BLOB for the spec, body or both to make sure it breaks and to see how your code handles it.
I suggest you rewrite this double nested loop to perform the three steps separately.
1. split the code into two parts
a. put the spec into one new CLOB variable and the body into the other.
b. use DBMS_OUTPUT (or a work table) to output the spec and body so you can see what the code is and make sure the 'split' process worked properly. You probably created that BLOB by manually concatenating the SPEC and BODY to begin with so now create a SPLIT process to split them again and give them back to you. This is such a fundamental component that I suggest creating a new SPLIT_MY_BLOB procedure. This procedure would take a BLOB and return two CLOBS as OUT parameters: one CLOB is the spec and one is the body. Then you can reuse this 'split' procedure in other code or more complex versions of code. Modular programming is the key.
2. execute the spec - Now have a step that executes the spec and does something appropriate if the step succeeds or if it fails. I mean, do you really want to do execute the body if the spec execution fails? What do you want to do? Should you delete the body and spec? If you don't you might wind up with an INVALID body based on old code and an INVALID spec based on the new code you were trying to use. How will anyone, including you, know that the spec and body in the ALL_SOURCE view is really two different versions of things?
This suggests that for your use case you may want to consider DROPPING the entire package, spec and body, before trying to recreate them both. At least if the package is totally missing anyone will know that the entire thing needs to be put back. Ahhhh - but to do that you need to know the package name so you can drop it. Than may require adding another step to get the package name from your data-driven table and adding a DROP PACKAGE step.
3. execute the body - This step executes the body. Hmmmm - we have two nearly identical steps. So why don't you create a new function/procedure that takes a CLOB as input, uses dynamic sql to execute it and returns a result code. That would be useful. Then you could execute ANY sql stored in a CLOB and have a generic function that you can use for other things.
Right now you are using the VARCHAR2 table version of DBMS_SQL.PARSE but you would change this to use the CLOB version.
The end result of this refactoring is a main function/procedure that acts as the CONTROL - it decides what to do and what data (BLOB) to do it with. But it doesn't actually do ANY of the work. It just controls what is done.
And you have a couple of generic, reuseable functions that actually do the work. One knows how to split a CLOB into spec and body. The other knows how to use dynamic SQL to execute a CLOB.
Now you have a modular architecture that is easy to understand, easy to code and easy to test.