Validating XML stored in CLOB without exceptions for malformed XML

Question

Hello,
I have a CLOB column that should contain an XML which conforms to a registered XSD schema. However, there might also be cases, in which the CLOB content violates the XSD or is not even wellformed XML at all.  Now I am trying to find a way to either
identify all records, for which the CLOB contains well-formed XML that validates against my schema (find all "good" records) , or
find all the other records (those with either no XML at all, malformed XML, or XML that does not validate against the XSD) (find all "exceptions")
The problem is that all XML-validation methods I know of (e.g. isXmlValid or XmlType.isSchemaValid)  require an XmlType instance, and that records, for which no proper XmlType can be constructed from the CLOB, because the CLOB does not contain well-formed XML, will cause an exception, rather than just returning false or NULL or something else I could filter out.
Is there a way to do something like
SELECT * FROM MYTABLE where <MY_XML_COL is wellformed XML> and <MY_XML_COL validates against 'blabla.xsd'>
without getting an ORA-31011 or whatever other kind of exception as soon as there is a row that does not contain proper XML?
Thank you...

Answer 1

So here is my example - this will be quiet a long post now...
First, I create the table with the CLOBs
CREATE TABLE ZZZ_MINITEST_CLOB (
  ID NUMBER(10,0) NOT NULL ENABLE,
  XML_DOC CLOB,
  REMARKS VARCHAR2(1000),
  CONSTRAINT PK_ZZZ_MINITEST_CLOB PRIMARY KEY (ID)
) NOLOGGING;
Then I insert some examples
INSERT INTO ZZZ_MINITEST_CLOB VALUES (
10,
'<minitest_root>
  <l2Node>
  <l2Num>1</l2Num>
  <l2Name>one.one</l2Name>
  </l2Node>
  <minitestId>1</minitestId>
  <minitestName>One</minitestName>
</minitest_root>',
'Basic valid example');
INSERT INTO ZZZ_MINITEST_CLOB VALUES (
20,
'<minitest_root>
  <l2Node>
  <l2Num>1</l2Num>
  <l2Name>two.one</l2Name>
  </l2Node>
  <minitestId>2</minitestId>
  <!-- minitestName element is missing -->
</minitest_root>',
'Invalid - minitestName element is missing');
INSERT INTO ZZZ_MINITEST_CLOB VALUES (
30,
'<minitest_root>
  <l2Node>
  <l2Num>1</l2Num>
  <l2Name>three.one</l2Name>
  </l2Node>
  <!-- minitestName and minitestId are switched -->
  <minitestName>Three</minitestName>
  <minitestId>3</minitestId>
</minitest_root>',
'Invalid - minitestName and minitestId are switched');
INSERT INTO ZZZ_MINITEST_CLOB VALUES (
40,
'<minitest_root>
  <l2Node>
  <l2Num>1</l2Num>
  <l2Name>four.one</l2Name>
  </l2Node>
  <l2Node>
  <l2Num>2</l2Num>
  <l2Name>four.two</l2Name>
  </l2Node>
  <l2Node>
  <l2Num>3</l2Num>
  <l2Name>four.three</l2Name>
  </l2Node>
  <minitestId>4</minitestId>
  <minitestName>Four</minitestName>
</minitest_root>',
'Valid - multiple l2Node elements');
INSERT INTO ZZZ_MINITEST_CLOB VALUES (
50,
'<minitest_root>
  <l2Node>
  <l2Num>1</l2Num>
  <l2Name>five.one</l2Name>
  </l2Node>
  <l2Node>
  <l2Num>2</l2Num>
  <l2Name>five.two</l2Name>
  </l2Node>
  <minitestId>4</minitestId>
  <minitestName>Five</minitestName>
  <!-- another l2Node node, but too far down -->
  <l2Node>
  <l2Num>3</l2Num>
  <l2Name>five.three</l2Name>
  </l2Node>
</minitest_root>',
'Invalid - another l2Node node, but too far down');
INSERT INTO ZZZ_MINITEST_CLOB VALUES (
60,
'something that is not even xml',
'Invalid - something that is not even xml');
INSERT INTO ZZZ_MINITEST_CLOB VALUES (
70,
NULL,
'Invalid - good old NULL');
INSERT INTO ZZZ_MINITEST_CLOB VALUES (
80,
'<minitest_root>
  <l2Node>
  <l2Num>1</l2Num>
  <l2Name>
  <unexpected_node>
  this one should not be here!
  </unexpected_node>
  </l2Name>
  </l2Node>
  <minitestId>8</minitestId>
  <minitestName>Eight</minitestName>
</minitest_root>',
'Invalid - unexpected addl node');
INSERT INTO ZZZ_MINITEST_CLOB VALUES (
90,
'<something> that has tags but is no xml </either>',
'Invalid - something that has tags but is no xml either');
COMMIT;
Next I register the XSD...
BEGIN
DBMS_XMLSCHEMA.REGISTERSCHEMA(
  'http://localhost/minitest.xsd',
  '<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" xmlns:oraxdb="http://xmlns.oracle.com/xdb" oraxdb:storeVarrayAsTable="true" oraxdb:schemaURL="http://localhost/minitest.xsd">
  <xs:element name="minitest_root" oraxdb:SQLName="MINITEST_ROOT" oraxdb:SQLType="MINITEST_TYPE" oraxdb:defaultTable="" oraxdb:tableProps="NOLOGGING" oraxdb:maintainDOM="false">
    <xs:complexType oraxdb:SQLType="MINITEST_TYPE" oraxdb:maintainDOM="false">
      <xs:sequence>
        <xs:element maxOccurs="unbounded" ref="l2Node" oraxdb:SQLName="L2_NODE" oraxdb:SQLType="L2_NODE_TYPE" oraxdb:SQLCollType="L2_NODE_COLL" oraxdb:maintainDOM="false"/>
        <xs:element ref="minitestId" oraxdb:SQLName="MINITEST_ID" oraxdb:SQLType="NUMBER" oraxdb:maintainDOM="false"/>
        <xs:element ref="minitestName" oraxdb:SQLName="MINITEST_NAME" oraxdb:SQLType="VARCHAR2" oraxdb:maintainDOM="false"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="l2Node" oraxdb:SQLName="L2_NODE" oraxdb:SQLType="L2_NODE_TYPE" oraxdb:defaultTable="" oraxdb:tableProps="TABLESPACE CATALOG NOLOGGING" oraxdb:maintainDOM="false">
    <xs:complexType oraxdb:SQLType="L2_NODE_TYPE" oraxdb:maintainDOM="false">
      <xs:sequence>
        <xs:element ref="l2Num" oraxdb:SQLName="L2_NUM" oraxdb:SQLType="NUMBER" oraxdb:maintainDOM="false"/>
        <xs:element ref="l2Name" oraxdb:SQLName="L2_NAME" oraxdb:SQLType="VARCHAR2" oraxdb:maintainDOM="false"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="l2Num" type="number10_0Type" oraxdb:SQLName="L2_NUM" oraxdb:SQLType="NUMBER" oraxdb:maintainDOM="false" />
  <xs:element name="l2Name" type="varchar100Type" oraxdb:SQLName="L2_NAME" oraxdb:SQLType="VARCHAR2" oraxdb:maintainDOM="false" />
  <xs:element name="minitestId" type="number10_0Type" oraxdb:SQLName="MINITEST_ID" oraxdb:SQLType="NUMBER" oraxdb:maintainDOM="false" />
  <xs:element name="minitestName" type="varchar100Type" oraxdb:SQLName="MINITEST_NAME" oraxdb:SQLType="VARCHAR2" oraxdb:maintainDOM="false" />
  <xs:simpleType name="varchar100Type">
    <xs:restriction base="xs:string">
      <xs:maxLength value="100"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:simpleType name="number10_0Type">
    <xs:restriction base="xs:integer">
      <xs:totalDigits value="10"/>
    </xs:restriction>
  </xs:simpleType>
</xs:schema>',
  GENTABLES=>FALSE
END;
And then I create my XML tables - four different ones for four different test cases. I am trying unstructured binary XML and strucured OR-based XML, each with and without a trigger.
One with structured OR storage and no trigger
CREATE TABLE ZZZ_MINITEST_ORXML (
  ID NUMBER(10,0) NOT NULL ENABLE,
  XML_CONTENT XMLTYPE,
  CONSTRAINT PK_ZZZ_MINITEST_ORXML PRIMARY KEY (ID)
) NOLOGGING
VARRAY "XML_CONTENT"."XMLDATA"."L2_NODE" STORE AS TABLE "ZZZ_OR_L2_NODE" (NOLOGGING) RETURN AS LOCATOR
XMLTYPE XML_CONTENT STORE AS OBJECT RELATIONAL XMLSCHEMA "http://localhost/minitest.xsd" ELEMENT "minitest_root";
One with structured OR storage which will also have a trigger added further down
CREATE TABLE ZZZ_MINITEST_ORXML_TRGR (
  ID NUMBER(10,0) NOT NULL ENABLE,
  XML_CONTENT XMLTYPE,
  CONSTRAINT PK_ZZZ_MINITEST_ORXML_TRGR PRIMARY KEY (ID)
) NOLOGGING
VARRAY "XML_CONTENT"."XMLDATA"."L2_NODE" STORE AS TABLE "ZZZ_OR_L2_NODE_TRGR" (NOLOGGING) RETURN AS LOCATOR
XMLTYPE XML_CONTENT STORE AS OBJECT RELATIONAL XMLSCHEMA "http://localhost/minitest.xsd" ELEMENT "minitest_root";
One with unstructured binary XML in a SECUREFILE, no trigger
CREATE TABLE ZZZ_MINITEST_BINXML_TRGR (
  ID NUMBER(10,0) NOT NULL ENABLE,
  XML_CONTENT XMLTYPE,
  CONSTRAINT PK_ZZZ_MINITEST_BINXML_TRGR PRIMARY KEY (ID)
) NOLOGGING
XMLTYPE COLUMN XML_CONTENT STORE AS SECUREFILE BINARY XML
  (NOCOMPRESS NOCACHE NOLOGGING KEEP_DUPLICATES);
One with unstructured binary XML in a SECUREFILE, which will also have a trigger
CREATE TABLE ZZZ_MINITEST_BINXML (
  ID NUMBER(10,0) NOT NULL ENABLE,
  XML_CONTENT XMLTYPE,
  CONSTRAINT PK_ZZZ_MINITEST_BINXML PRIMARY KEY (ID)
) NOLOGGING
XMLTYPE COLUMN XML_CONTENT STORE AS SECUREFILE BINARY XML
  (NOCOMPRESS NOCACHE NOLOGGING KEEP_DUPLICATES);
Then I create the error logging tables
begin
DBMS_ERRLOG.CREATE_ERROR_LOG ('ZZZ_MINITEST_BINXML', 'ZZZ_MINITEST_BINXML_E', NULL, NULL, TRUE);
DBMS_ERRLOG.CREATE_ERROR_LOG ('ZZZ_MINITEST_BINXML_TRGR', 'ZZZ_MINITEST_BINXML_TRGR_E', NULL, NULL, TRUE);
DBMS_ERRLOG.CREATE_ERROR_LOG ('ZZZ_MINITEST_ORXML', 'ZZZ_MINITEST_ORXML_E', NULL, NULL, TRUE);
DBMS_ERRLOG.CREATE_ERROR_LOG ('ZZZ_MINITEST_ORXML_TRGR', 'ZZZ_MINITEST_ORXML_TRGR_E', NULL, NULL, TRUE);
END;
Now the two triggers
create or replace trigger TRG_ZZZ_MINITEST_BINXML
BEFORE UPDATE OR INSERT ON ZZZ_MINITEST_BINXML_TRGR
REFERENCING NEW AS NEW
for each row
BEGIN
  :NEW.XML_CONTENT := :NEW.XML_CONTENT.createSchemaBasedXML('http://localhost/minitest.xsd');
  :NEW.XML_CONTENT.SCHEMAVALIDATE();
END TRG_ZZZ_MINITEST_BINXML;
CREATE OR REPLACE TRIGGER TRG_ZZZ_MINITEST_ORXML
BEFORE UPDATE OR INSERT ON ZZZ_MINITEST_ORXML_TRGR
REFERENCING NEW AS NEW
for each row
BEGIN
  :NEW.XML_CONTENT := :NEW.XML_CONTENT.createSchemaBasedXML('http://localhost/minitest.xsd');
  :NEW.XML_CONTENT.SCHEMAVALIDATE();
END TRG_ZZZ_MINITEST_ORXML;
I also tried to just validate the XML using queries such as the below, since validating all without the WHERE also caused exception and abortion:
SELECT ID, DECODE(XMLISVALID(XMLTYPE(XML_DOC), 'http://localhost/minitest.xsd'), 1, 'Y', 0, 'N', '?') VALID
FROM ZZZ_MINITEST_CLOB WHERE ID=90;
The results are in column "VALID" in the below tables
I tried inserting all records from ZZZ_MINITEST_CLOB into the four test tables at once using a query such as
INSERT INTO ZZZ_MINITEST_ORXML_TRGR
SELECT ID, XMLPARSE(DOCUMENT XML_DOC WELLFORMED) FROM ZZZ_MINITEST_CLOB
LOG ERRORS INTO ZZZ_MINITEST_ORXML_TRGR_E REJECT LIMIT UNLIMITED;
I also tried different versions of creating the XML in the SELECT portion:
XMLPARSE(DOCUMENT XML_DOC WELLFORMED)
XMLPARSE(DOCUMENT XML_DOC WELLFORMED)
XMLTYPE(XML_DOC)
XMLTYPE(XMLDOC, 'http://localhost/minitest.xsd', 0, 1)
Almost all combinations of the four test tables with the four ways to instantiate the XML caused exceptions and query abortion despite the LOG ERRORS INTO clause.
In order to find the exact problems, I started inserting records one by one using queries like
INSERT INTO ZZZ_MINITEST_ORXML
SELECT ID, XMLPARSE(DOCUMENT XML_DOC) FROM ZZZ_MINITEST_CLOB WHERE ID=10
LOG ERRORS INTO ZZZ_MINITEST_ORXML_E REJECT LIMIT UNLIMITED;
or
INSERT INTO ZZZ_MINITEST_BINXML_TRGR
SELECT ID, XMLTYPE(XMLDOC, 'http://localhost/minitest.xsd', 0, 1) FROM ZZZ_MINITEST_CLOB WHERE ID=20
LOG ERRORS INTO ZZZ_MINITEST_BINXML_TRGR_E REJECT LIMIT UNLIMITED;
I captured the results of each in the below four tables. "1" and "0" are number of rows inserted. 0 means there was no record inserted and instead, there was an error logged in the ERROR LOGGING table.
The ORA-????? exception numbers mean that this error actually caused the INSERT to fail and abort rather than just being logged in the LOGGING TABLE.These are the most critical cases for me. Why do these exceptions "bubble up" forcing the query to be aborted rather than being logged in the error log as well???
This table is for INSERT of XMLs using XMLTYPE(XML_DOC)
Test case
VALID
Binary XML,
no trigger
Binary XML,
trigger
OR XML,
no trigger
OR XML,
trigger
10
Good
Y
1
1
1
1
20
no name
N
1
0
1
0
30
switched tags
N
1
1
1
1
40
Good
Y
1
1
1
1
50
L2 down
N
1
1
1
1
60
no xml
EX
ORA-31011
ORA-31011
ORA-31011
ORA-31011
70
NULL
EX
ORA-06502
ORA-06502
ORA-06502
ORA-06502
80
addl. Node
N
1
0
ORA-31187
ORA-31187
90
crappy xml
EX
ORA-31011
ORA-31011
ORA-31011
ORA-31011
This table is for INSERT of XMLs using XMLTYPE(XML_DOC, 'http://localhost/minitest.xsd', 0, 1)
Test case
VALID
Binary XML,
no trigger
Binary XML,
trigger
OR XML,
no trigger
OR XML,
trigger
10
Good
Y
1
1
1
1
20
no name
N
1
0
1
0
30
switched tags
N
1
1
1
1
40
Good
Y
1
1
1
1
50
L2 down
N
1
1
1
1
60
no xml
EX
ORA-31043
ORA-31043
ORA-31043
ORA-31043
70
NULL
EX
ORA-06502
ORA-06502
ORA-06502
ORA-06502
80
addl. Node
N
1
0
ORA-31187
ORA-31187
90
crappy xml
EX
ORA-31043
ORA-31043
ORA-31043
ORA-31043
This table is for INSERT of XMLs using XMLPARSE(DOCUMENT XML_DOC WELLFORMED)
Test case
VALID
Binary XML,
no trigger
Binary XML,
trigger
OR XML,
no trigger
OR XML,
trigger
10
Good
Y
1
1
1
1
20
no name
N
1
0
1
0
30
switched tags
N
1
1
1
1
40
Good
Y
1
1
1
1
50
L2 down
N
1
1
1
1
60
no xml
ORA-31061
0
ORA-31011
ORA-31011
70
NULL
1
0
1
0
80
addl. Node
N
0
0
ORA-31187
ORA-31187
90
crappy xml
ORA-31061
0
ORA-30937
ORA-30937
This table is for INSERT of XMLs using XMLPARSE(DOCUMENT XML_DOC) (same as above, but without "WELLFORMED")
Test case
VALID
Binary XML,
no trigger
Binary XML,
trigger
OR XML,
no trigger
OR XML,
trigger
10
Good
Y
1
1
1
1
20
no name
N
1
0
1
0
30
switched tags
N
1
1
1
1
40
Good
Y
1
1
1
1
50
L2 down
N
1
1
1
1
60
no xml
ORA-31011
ORA-31011
ORA-31011
ORA-31011
70
NULL
1
0
1
0
80
addl. Node
N
1
0
ORA-31187
ORA-31187
90
crappy xml
ORA-31011
ORA-31011
ORA-31011
ORA-31011
As you can see, different ways to instantiate the XML which is being inserted will cause different results for the most critical cases (60, 80 and 90). One will go through with an error logged in the logging table and no record inserted, others cause exceptions. Also, using XMLPARSE with or without WELLFORMED causes different ORA numbers for record 90...
It seems like the only way to not get any exceptions for my test records is inserting XMLs that were created using XMLPARSE (DOCUMENT ... WELLFORMED) into the trigger-protected binary XML table. However, that causes me to miss record number 80, which is one that I really need, because it is well formed and I could use XSLTs to fix the XML by removing the extra tags, as long as I know which ones those are.