How to come up with a magic number for any table that returns more than 32KB?

Question

I am in a unique situation where in I am trying to retrieve values from multiple tables and publish them as XML output. The problem is based on the condition a few tables could retrieve data more than 32KB and a few less than 32KB. Less than 32KB is not an issue as XML generation is smooth. The minute it reaches greater than 32KB it generates a run time error. Just wondering if there is any way to ensure that the minute the query's results is greater than 32 kb, it should break say - if the results is 35KB, then I should break that result into 32 KB and 3kb; once done then pass this data to be published as an XML output. This is again not just for one table, but all the tables that are called in the function.
Is there any way?? I am unable to get any ideas nor have I done anything so complex from production support stand point. Would appreciate if someone can guide me on this.
The way it is, is as follows:
I have a table called ctn_pub_cntl
CREATE TABLE CTNAPP.ctn_pub_cntl
(ctn_pub_cntl_id          NUMBER(18)
,table_name                  VARCHAR2(50)
,last_pub_tms              DATE
,queue_name               VARCHAR2(50)
,dest_system              VARCHAR2(50)
,frequency                  NUMBER(6)
,status                      VARCHAR2(8)
,record_create_tms          DATE
,create_user_id                VARCHAR2(8)
,record_update_tms          DATE
,update_user_id             VARCHAR2(8)
,CONSTRAINT ctn_pub_cntl_id_pk PRIMARY KEY(ctn_pub_cntl_id)
Data for this is:
INSERT INTO CTNAPP.ctn_pub_cntl
(ctn_pub_cntl_id   
,table_name        
,last_pub_tms 
,queue_name 
,dest_system       
,frequency         
VALUES
(CTNAPP_SQNC.nextval
,'TRKFCG_SBDVSN'
,TO_DATE('10/2/2004 10:17:44PM','MM/DD/YYYY HH12:MI:SSPM')
,'UT.TSD.TSZ601.UNP'
,'SAP'
,15
INSERT INTO CTNAPP.ctn_pub_cntl
(ctn_pub_cntl_id   
,table_name        
,last_pub_tms 
,queue_name 
,dest_system       
,frequency         
VALUES
(CTNAPP_SQNC.nextval
,'TRKFCG_TRACK_SGMNT_DN'
,TO_DATE('02/06/2015 9:50:00AM','MM/DD/YYYY HH12:MI:SSPM')
,'UT.TSD.WRKORD.UNP'
,'SAP'
,30
INSERT INTO CTNAPP.ctn_pub_cntl
(ctn_pub_cntl_id   
,table_name        
,last_pub_tms 
,queue_name 
,dest_system       
,frequency         
VALUES
(CTNAPP_SQNC.nextval
,'TRKFCG_FXPLA_TRACK_LCTN_DN'
,TO_DATE('10/2/2004 10:17:44PM','MM/DD/YYYY HH12:MI:SSPM')
,'UT.TSD.YRDPLN.INPUT'
,'SAP'
,30
INSERT INTO CTNAPP.ctn_pub_cntl
(ctn_pub_cntl_id   
,table_name        
,last_pub_tms 
,queue_name 
,dest_system       
,frequency         
VALUES
(CTNAPP_SQNC.nextval
,'TRKFCG_FXPLA_TRACK_LCTN2_DN'
,TO_DATE('02/06/2015 9:50:00AM','MM/DD/YYYY HH12:MI:SSPM')
,'UT.TSD.TSZ601.UNP'
,'SAP'
,120
INSERT INTO CTNAPP.ctn_pub_cntl
(ctn_pub_cntl_id   
,table_name        
,last_pub_tms
,queue_name 
,dest_system       
,frequency         
VALUES
(CTNAPP_SQNC.nextval
,'TRKFCG_FXPLA_TRACK_LCTN2_DN'
,TO_DATE('04/23/2015 11:50:00PM','MM/DD/YYYY HH12:MI:SSPM')
,'UT.TSD.YRDPLN.INPUT'
,'SAP'
,10
INSERT INTO CTNAPP.ctn_pub_cntl
(ctn_pub_cntl_id   
,table_name        
,last_pub_tms
,queue_name 
,dest_system       
,frequency         
VALUES
(CTNAPP_SQNC.nextval
,'TRKFCG_FIXED_PLANT_ASSET'
,TO_DATE('04/23/2015 11:50:00AM','MM/DD/YYYY HH12:MI:SSPM')
,'UT.TSD.WRKORD.UNP'
,'SAP'
,10
INSERT INTO CTNAPP.ctn_pub_cntl
(ctn_pub_cntl_id   
,table_name        
,last_pub_tms
,queue_name 
,dest_system       
,frequency         
VALUES
(CTNAPP_SQNC.nextval
,'TRKFCG_OPRLMT'
,TO_DATE('03/26/2015 7:50:00AM','MM/DD/YYYY HH12:MI:SSPM')
,'UT.TSD.WRKORD.UNP'
,'SAP'
,30
INSERT INTO CTNAPP.ctn_pub_cntl
(ctn_pub_cntl_id   
,table_name        
,last_pub_tms
,queue_name 
,dest_system       
,frequency         
VALUES
(CTNAPP_SQNC.nextval
,'TRKFCG_OPRLMT_SGMNT_DN'
,TO_DATE('03/28/2015 12:50:00AM','MM/DD/YYYY HH12:MI:SSPM')
,'UT.TSD.WRKORD.UNP'
,'SAP'
,30
COMMIT;
Once the above data is inserted and committed, then I created a function in a package:
CREATE OR REPLACE PACKAGE CTNAPP.CTN_PUB_CNTL_EXTRACT_PUBLISH
IS
TYPE tNameTyp IS TABLE OF ctn_pub_cntl.table_name%TYPE INDEX BY BINARY_INTEGER;
g_tName tNameTyp;
TYPE tClobTyp IS TABLE OF CLOB INDEX BY BINARY_INTEGER;
g_tClob tClobTyp;
FUNCTION GetCtnData(p_nInCtnPubCntlID IN CTN_PUB_CNTL.ctn_pub_cntl_id%TYPE,p_count OUT NUMBER ) RETURN tClobTyp;
END CTNAPP.CTN_PUB_CNTL_EXTRACT_PUBLISH;
--Package body
CREATE OR REPLACE PACKAGE BODY CTNAPP.CTN_PUB_CNTL_EXTRACT_PUBLISH
IS
     doc           xmldom.DOMDocument;
     main_node     xmldom.DOMNode;
     root_node     xmldom.DOMNode;
     root_elmt     xmldom.DOMElement;
     child_node    xmldom.DOMNode;
     child_elmt    xmldom.DOMElement;
     leaf_node     xmldom.DOMNode;
     elmt_value    xmldom.DOMText;
     tbl_node      xmldom.DOMNode;
     table_data    XMLDOM.DOMDOCUMENTFRAGMENT;
     l_ctx         DBMS_XMLGEN.CTXHANDLE;
     vStrSqlQuery  VARCHAR2(32767);
     l_clob        tClobTyp;
     l_xmltype     XMLTYPE;
--Local Procedure to build XML header    
PROCEDURE BuildCPRHeader IS
  BEGIN
    child_elmt := xmldom.createElement(doc, 'PUBLISH_HEADER');
    child_node  := xmldom.appendChild (root_node, xmldom.makeNode (child_elmt));
    child_elmt := xmldom.createElement (doc, 'SOURCE_APLCTN_ID');
    elmt_value := xmldom.createTextNode (doc, 'CTN');
    leaf_node  := xmldom.appendChild (child_node, xmldom.makeNode (child_elmt));
    leaf_node  := xmldom.appendChild (leaf_node, xmldom.makeNode (elmt_value));
    child_elmt := xmldom.createElement (doc, 'SOURCE_PRGRM_ID');
    elmt_value := xmldom.createTextNode (doc, 'VALUE');
    leaf_node  := xmldom.appendChild (child_node, xmldom.makeNode (child_elmt));
    leaf_node  := xmldom.appendChild (leaf_node, xmldom.makeNode (elmt_value));
    child_elmt := xmldom.createElement (doc, 'SOURCE_CMPNT_ID');
    elmt_value := xmldom.createTextNode (doc, 'VALUE');
    leaf_node  := xmldom.appendChild (child_node, xmldom.makeNode (child_elmt));
    leaf_node  := xmldom.appendChild (leaf_node, xmldom.makeNode (elmt_value));
    child_elmt := xmldom.createElement (doc, 'PUBLISH_TMS');
    elmt_value := xmldom.createTextNode (doc, TO_CHAR(SYSDATE, 'YYYY-MM-DD HH24:MI:SS'));
    leaf_node  := xmldom.appendChild (child_node, xmldom.makeNode (child_elmt));
    leaf_node  := xmldom.appendChild (leaf_node, xmldom.makeNode (elmt_value));
END BuildCPRHeader;
--Get table data based on table name
FUNCTION GetCtnData(p_nInCtnPubCntlID IN CTN_PUB_CNTL.ctn_pub_cntl_id%TYPE,p_Count OUT NUMBER) RETURN tClobTyp IS
    vTblName      ctn_pub_cntl.table_name%TYPE;
    vLastPubTms   ctn_pub_cntl.last_pub_tms%TYPE;
BEGIN
            g_vProcedureName:='GetCtnData';   
            g_vTableName:='CTN_PUB_CNTL';
        SELECT table_name,last_pub_tms
        INTO   vTblName, vLastPubTms
        FROM   CTN_PUB_CNTL
        WHERE  ctn_pub_cntl_id=p_nInCtnPubCntlID;
    -- Start the XML Message generation
        doc := xmldom.newDOMDocument;
        main_node := xmldom.makeNode(doc);
        root_elmt := xmldom.createElement(doc, 'PUBLISH');
        root_node := xmldom.appendChild(main_node, xmldom.makeNode(root_elmt));
      --Append Table Data as Publish Header
        BuildCPRHeader;
      --Append Table Data as Publish Body
       child_elmt := xmldom.createElement(doc, 'PUBLISH_BODY');
       leaf_node  := xmldom.appendChild (root_node, xmldom.makeNode(child_elmt));
       DBMS_SESSION.SET_NLS('NLS_DATE_FORMAT','''YYYY:MM:DD HH24:MI:SS''');
       vStrSqlQuery := 'SELECT * FROM ' || vTblName
                      || ' WHERE record_update_tms <= TO_DATE(''' || TO_CHAR(vLastPubTms, 'MM/DD/YYYY HH24:MI:SS') || ''', ''MM/DD/YYYY HH24:MI:SS'') ' ;
                    --  ||  ' AND rownum < 16'
      DBMS_OUTPUT.PUT_LINE(vStrSqlQuery);
       l_ctx  := DBMS_XMLGEN.NEWCONTEXT(vStrSqlQuery);
      DBMS_XMLGEN.SETNULLHANDLING(l_ctx, 0);
      DBMS_XMLGEN.SETROWSETTAG(l_ctx, vTblName);
      -- Append Table Data as XML Fragment
      l_clob(1):=DBMS_XMLGEN.GETXML(l_ctx); 
      elmt_value := xmldom.createTextNode (doc, l_clob(1));
     leaf_node  := xmldom.appendChild (leaf_node, xmldom.makeNode (elmt_value));
     xmldom.writeToBuffer (doc, l_clob(1));
     l_clob(1):=REPLACE(l_clob(1),'&lt;?xml version=&quot;1.0&quot;?&gt;', NULL);
     l_clob(1):=REPLACE(l_clob(1),'&lt;', '<');
     l_clob(1):=REPLACE(l_clob(1),'&gt;', '>');
     RETURN l_clob;
     DBMS_OUTPUT.put_line('Answer is' ||l_clob(1));
     EXCEPTION
        WHEN NO_DATA_FOUND THEN
        DBMS_OUTPUT.put_line('There is no data with' || SQLERRM);
        g_vProcedureName:='GetCtnData';
        g_vTableName:='CTN_PUB_CNTL';
        g_vErrorMessage:=SQLERRM|| g_vErrorMessage;
        g_nSqlCd:=SQLCODE;
        ctn_log_error('ERROR',g_vErrorMessage,'SELECT',g_nSqlCd,p_nInCtnPubCntlID,g_vPackageName,g_vProcedureName,g_vTableName);
        WHEN OTHERS THEN
       DBMS_OUTPUT.PUT_LINE('ERROR : ' || SQLERRM);
       ctn_log_error('ERROR',g_vErrorMessage,'OTHERS',g_nSqlCd,p_nInCtnPubCntlID,g_vPackageName,g_vProcedureName,g_vTableName);
END GetCtnData;
PROCEDURE printClob (result IN OUT NOCOPY CLOB) IS
    xmlstr   VARCHAR2 (32767);
    line     VARCHAR2 (2000);
BEGIN
    xmlstr := DBMS_LOB.SUBSTR (result, 32767);
    LOOP
       EXIT WHEN xmlstr IS NULL;
       line := SUBSTR (xmlstr, 1, INSTR (xmlstr, CHR (10)) - 1);
       DBMS_OUTPUT.put_line (line);
       xmlstr := SUBSTR (xmlstr, INSTR (xmlstr, CHR (10)) + 1);
    END LOOP;
END printClob;
END CTN_PUB_CNTL_EXTRACT_PUBLISH;
If you notice my query:
vStrSqlQuery := 'SELECT * FROM ' || vTblName
                      || ' WHERE record_update_tms <= TO_DATE(''' || TO_CHAR(vLastPubTms, 'MM/DD/YYYY HH24:MI:SS') || ''', ''MM/DD/YYYY HH24:MI:SS'') ' ;
                     ||  ' AND rownum < 16'
The minute I comment
||  ' AND rownum < 16' ;
, it throws an error because this query returns around 600 rows and all of those rows need to be published as XML and the tragedy is that there is a C program in between as well i.e. C will call my packged functions and then will do all the processing. Once this is done will pass the results back to C program. So obviously C does not recognise CLOB and somewhere in the process I have to convert the CLOB to VARCHAR or instead of CLOB I have to use VARCHAR array as a return type. This is my challenge.
Anyone that can help me to find out the required magic number and also a brief know how, I would appreciate that. Many thanks in advance.

Answer 1

Not sure I understand which part is failing.
Is it the C program calling your packaged function? Or does the error occur in the PL/SQL code, in which case you should be able to pinpoint where it's wrong?
A few comments :
1) Using DOM to build XML out of relational data? What for? Use SQL/XML functions.
2) Giving sample data is usually great, but it's not useful here since we can't run your code. We're missing the base tables.
3) This is wrong :
vStrSqlQuery := 'SELECT * FROM ' || vTblName                     || ' WHERE record_update_tms <= TO_DATE(''' || TO_CHAR(vLastPubTms, 'MM/DD/YYYY HH24:MI:SS') || ''', ''MM/DD/YYYY HH24:MI:SS'') ' ;
A bind variable should be used here for the date.
4) This is wrong :
elmt_value := xmldom.createTextNode (doc, l_clob(1));
createTextNode does not support CLOB so it will fail as soon as the CLOB you're trying to pass exceeds 32k.
Maybe that's the problem you're referring to?
5) This is most wrong :
     l_clob(1):=REPLACE(l_clob(1),'&lt;?xml version=&quot;1.0&quot;?&gt;', NULL); 
     l_clob(1):=REPLACE(l_clob(1),'&lt;', '<'); 
     l_clob(1):=REPLACE(l_clob(1),'&gt;', '>'); 
I understand what you're trying to do but it's not the correct way.
You're trying to convert a text() node representing XML in escaped form back to XML content.
The problem is that there are other things to take care of besides just '&lt;' and '&gt;'.
If you want to insert an XML node into an existing document, treat that as an XML node, not as a string.
Anyway,
Anyone that can help me to find out the required magic number
That would be a bad idea. Fix what needs to be fixed.
And please clearly state which part is failing : the C program or the PL/SQL code?
I'd vote for PL/SQL, as pointed out in [4].