Handling bad records in a file.

Question

Hi,
I am using external tables for loading data from text files. I am doing it well. I came across a case, where some records are not good, those records are diverted to bad file.
In that case, we are losing the valuable data. We can do some validation and we can use it.
Could anyone please suggest me the way we could perform this.
And I have one more doubt. Using OWB we can create single external table for multiple files. I would like to differentiate the records of each file, by adding filename as one extra field.
Can we do this using owb?
Any suggestions are welcome.
Thanks and regards
Gowtham Sen.

Answer 1

Indeed,
OWB is not a magic tool, but just a good ETL tool that will do what you tell it to :)
You must first figure out what cases you come a cross, and then figure out what you can do to solve each strange case.
First of all, as has been pointed out, get all the data into your external tables. Avoid doing any type checking or casting at this case. just stick to VARCHAR fields (and perhaps NUMBER but even here I tend to use VARCHAR). Then, in a second mapping, try to get your data in to a staging table and there you can isolate the bad records, and perhaps allow for human interaction to correct the data if all else fails.
Regarding the files, you have before mentioned that delta loads are what you have to do. Unless you are running your loads in a "live" system where data is flowing in on and off during daily operations, then going for a whole refresh would make your life so much better (20 mins of loading during the night is quite a good deal, if it makes your loading logic so much simpler).
But, as you have said before, you need delta loads.
Using UNIX shell scripts, you can use the 'diff' command to create your datafiles. That should be quite straight forward. Otherwise you can perhaps use staging tables and the MINUS operator. There isn't always a quick solution.
Borkur