Data Extract Design

Question

Hi All,
I have a requirement where I have to extract data from various different tables-->Only particular columns.
The requirements are same for different databases, hence I thought to have a single generic approach and reuse the same code to perform the extract and create an ascii file.
Below is the typical scenarion i want to achieve, hence need your expertise inputs to start off..
a) Define the required columns -- This should be configurable, i.e., add or remove columns in future.
b) Extract the column names from the database for those that are defined in the step a) above.
c) Extract the data from relevent tables/columns for various conditions based on step a and b above.
d) Create an ascii file for all the data extracted.
I'm unsure if there is anything wrong or please suggest the best approach.
Regs,
R

Answer 1

user10177353 wrote:
I'm unsure if there is anything wrong or please suggest the best approach.
The first thing to bear in mind is that developing a generic, dynamic solution is considerably more more complicated than writing a set of extract statements. So you need to be sure that the effort you're about to expend willl save you more time than writing a script and copying/editing it for subsequent re-use.
You'll probably need three tables:
1. Extracts - one record per extract definition (perhaps including info such as target file name)
2. Extract tables - tableges for each extract
3. Extract columns - columns for each extracted column.
I'm writing this as though you'll be extracting more than one table per run.
The writing to file is the trickiest bit. Choose a good target. Remember that although we called them CSV files, commas actually make a remarkably poor choice of separator, as way too much data contains them. Go for soemthing really unlikely, ideally a multi-character separator like ||¬.
Also remember text files only take strings, so you need to convert your data to text. Use the data dictionary ALL_TAB_COLUMNS view to get the metatdata for the extracted columns, and apply explicit masks to date and numeric columns. You may want to allow date columns to have masks which include or exclude the time element.
Consider what you want to do with complex data types (LOBs, UDTs, etc).
Finally, you need to address the problem of the extract file's location. PL/SQL has a lot of utilities to wrangle files but they only work on the server side. So if you want to write to a local drive you'll need to use SPOOL.
One last thought: how will you import the data? It would probably be a good idea to use this mechanism to generate DDL for a matching external table.
Cheers, APC
Edited by: APC on May 4, 2012 1:08 PM