SQL to find out changes

Question

Hi,
I have a table with 50 columns, lets call it Table_A . There is a identical table called Table_B. The data in Table_B gets flushed every night and data from Table_A is moved to Table_B. A new set of data is inserted into Table_A. Now I need to find out which field value is changed and what is changed . A minus command can give the changed records, but how to get the what was prior value for each changed field . Here is what I am looking for
WITH TABLE_A AS  (  SELECT 1 ID, 'JOHN' NAME,  'SALES' DIV FROM DUAL
UNION
        SELECT 2 ID, 'MARRY' NAME,  'ACCOUNTS' DIV FROM DUAL
UNION
        SELECT 3 ID, 'KIM' NAME,  'SERVICE' DIV FROM DUAL),
TABLE_B AS  (  SELECT 1 ID, 'JOHN' NAME,  'SALES' DIV FROM DUAL
UNION
        SELECT 2 ID, 'MARRY' NAME,  'ACCOUNTS' DIV FROM DUAL
UNION
        SELECT 3 ID, 'KIM' NAME,  'SALES' DIV FROM DUAL) 
SELECT * FROM TABLE_A
MINUS
SELECT * FROM TABLE_B
what i want is something that will spit out this (only one record, because there is only one diff record found in the above query.)
ID,     OLD_ID,     NAME,   OLD_NAME ,  DIV ,       OLD_DIV
3           3       KIM       KIM       SERVICE     SALESThe above one is for sample purpose, The actual table has 50 fields, and except the ID field, all other field values might change.
All ideas and solutions are appreciated.
Thanks in advance.

Answer 1

>
I understand your frustration. Here is the scenario. There are set of 5 tables. 1 master and 4 child tables ( and an identical set of 5 tables called "prior" tables, to store the flushed data from the "current" 5 tables). Every night data comes in XML files, the data from the 5 "prior" tables get flushed out . The data from the 5"current" table set moves to 5 "prior" tables. Then XML data gets loaded into the 5 "current"tables. Now once XML gets loaded into the original table ,I have to find out the diffs in those 5 tables(between current and prior) and generate result sets listing only the columns that have changed value. So what is current value and what was the prior value(from the prior table) side by side .
>
I'm not frustrated. It's just that any suggestions we make are solely based on the information provided. Sometimes posters start focusing on the solution they want to use (which is ok for their viewpoint) and don't explain the actual problem they are trying to solve. That often limits our ability to understand the options.
Now that you posted the above there is another option you could use that might better deal with some of the issues people have mentioned.
That solution is to compare the XML versions of the data. You haven't mentioned HOW you load that XML data into relational tables (what tool and process) but there is one VERY USEFUL feature of data in XML format:
ALL data is already in TEXT format (excluding the obvious LOB possibilities, which you also haven't mentioned).
That means ALL datatypes can use simple string comparison to detect differences: numbers, dates, character data. The actual meaning doesn't matter. If even a single byte is different the value can be considered different.
Oracle uses XML and XSLT functionality as the base for their DBMS_METADATA_DIFF package. That package is used to compare schemas or databases to detect object (e.g. tables, views) differences and to automatically produce DDL for ALTER statements that can modify an object to have the same structure as the objects being compared.
I suggest that you consider doing your comparisons on the XML versions of the data. Each night you would compare the new XML file data against the previous night's XML data. You can use XSLT to do the comparison and also to produce an XML file that contains the differences. Then you can use that 'difference' file as the source of your process to ultimately display the data.
An additional advantage of doing the XML comparisons is that you can do them asynchronously as soon as the new file is available. And the, if you don't really need the 'change' info in the database you never need to load it. Just transform the change XML into an Excel file or flat file format for users to load into Excel for review.
Do you license the partitioning option? If so then another suggestion would be to stop doing your nightly 'flush and move'. Recreate your 'prior' tables as partitioned tables that have just ONE range partition; a MAXVALUE partition that covers ALL data. Then each night you would:
1. TRUNCATE the prior table
2. Do an EXCHANGE partition between the 'current' table and the associated 'prior' table. Very fast, metadata only - no data actually gets moved.
3. Load the 'current' table with new data.
The above process avoid any actual data movement between the 'current' and 'prior' tables.
Not picking on you but you also haven't said how that new daily 'current' data is being used. If the only reason you are loading it into a table is to compare it to yesterday's data you could just do that with XML like I mentioned above.
Or you could load the XML itself into a table (e.g. XMLTYPE) and uses Oracle's built-in XDB fuctionality and do the XML comparison 'inside' the database. Then load a standard heap table from the XML table when you need to.
Maybe the above helps explain why the more info we have about the actual end-to-end problem you have the more options we can suggest for dealing with it. You are the only one that can determine if an option is viable for your particular use case and constraints.
Updated to add links to Oracle's XML functionality
See the XMLDiff function that you can use IN the database if you load the XML files themselves. From the SQL Language doc
http://docs.oracle.com/cd/E11882_01/server.112/e26088/functions241.htm#SQLRF20025
>
Purpose
The XMLDiff function is the SQL interface for the XmlDiff C API. This function compares two XML documents and captures the differences in XML conforming to an Xdiff schema. The diff document is returned as an XMLType document.
For the first two arguments, specify the names of two XMLType documents.
For the integer, specify a number representing the hashLevel for a C function XmlDiff. If you do not want hashing, set this argument to 0 or omit it entirely. If you do not want hashing, but you want to specify flags, then you must set this argument to 0.
For string, specify the flags that control the behavior of the function. These flags are specified by one or more names separated by semicolon. The names are the same as the names of constants for XmlDiff function.
See Also:
Oracle XML Developer's Kit Programmer's Guide for more information on using this function, including examples, and Oracle Database XML C API Reference for information on the XML APIs for C
Examples
The following example compares two XML documents and returns the difference as an XMLType document:
>
See Oracle' own XML Developer's Kit for out-of-the-box functionality for 'diffing' XML documents:
http://docs.oracle.com/cd/E11882_01/appdev.112/e23582/adx_c_diff.htm
>
21 Determining XML Differences Using CThe Oracle XDK includes components that help you to determine the differences between the contents of two XML documents and then to apply the differences (patch) to one of the XML documents.
This chapter contains these topics:
Overview of XMLDiff in C
Using XmlDiff
Using XmlPatch
Using XmlHash
Overview of XMLDiff in C
You can use Oracle XmlDiff to determine the differences between two similar XML documents. XmlDiff generates an Xdiff instance document that indicates the differences. The Xdiff instance document is an XML document that conforms to an XML schema, the Xdiff schema.
You can then use XmlPatch, which takes the Xdiff instance document and applies the changes to other documents. This can be used to apply the same changes to a large number of XML documents.
XmlDiff only supports the DOM API for input and output.
XmlPatch also supports the DOM for the input and patch documents.
XmlDiff and XmlPatch can be used through a C API or a command line tool, and they are exposed by two SQL functions.
An XmlHash C API is provided to compute the hash value of an XML tree or subtree. If hash values of two trees or subtrees are equal, the trees are identical to a very high probability.
>
Did you notice that part about 'XmlDiff generates an Xdiff instance document that indicates the differences'? That is EXACTLY what your initial question was asking about, albeit from a DB perspective.
But XML is XML, DIFF is DIFF, a date/number/text are all strings. So XML 'database data' works the same as XML 'test or complex hierarchy' data.
Here is a pretty good writeup of how to do XML diff. See the XMLDIFF2 section for the best Oracle sample:
http://ellebaek.wordpress.com/2011/02/01/comparing-xml-in-oracle/
>
XMLDIFF2
The algorithm in our extended XMLDIFF function that we’re going to call XMLDIFF2 is:
1.Prepare CLOB variable for constructing the difference XML.
2.Find minimum number of leading elements that form primary key values (unique lookup).
3.Look for primary key values in XML1 not present in XML2: These primary key values have been deleted.
4.Look for primary key values in XML2 not present in XML1: These primary key values have been inserted.
5.Diff all common primary key values, ie in both XML1 and XML2.
6.Return NULL if no differences found.