Dynamic File Structure - Urgent

Question

Hi,
We have a requirement to use ODI to develop a POC that will do the following.
1. Create a Job to Load target from different file sources having different layouts or data structures.
2. The structure of the file needs to be fed to the job dynamically so the job can do a dynamic transformation and load to the target
3. An important requirement is to not change the job every time the input changes. The target will remain the same
The challenge is how to pass the schema of the file dynamically to the job.
This can be achieved in Datastage is our understanding but the POC is to showcase that Oracle Data Integrator also has this capability.
Any information docs or prior poc will be of great help
Appreciate your response!!!
-Basheer

Answer 1

I will leave the choice completely to Basir for going with implementation , but here i am sharing what i have thought about.
ODI Procedure -
Step 1 - To read the above File and creating the CTL file for SQLLDR
Technology- Jython
import re,os,string
# Reading the File
file_read=open('<%=odiRef.getOption("FILE_PATH")%>/Dynamic_struc.txt','r')
line=file_read.readline()
collect=[]
#Looping through each Line
while line:
    # Reading Delimiter
    if re.search("delim",line):
        delimiter= line[-5:-2]
    # reading Columns
    if ((string.find(line,';')> 0) & (string.find(line,')')> 0)):
        value=line[0:string.find(line,';')]
        collect.append(string.strip(value))   
    line=file_read.readline()
column=','.join(map(string.strip,collect))
output= ' The delimiter of this file is  '+delimiter +' The columns are '+column
file_read.close()
# Creating a CTL File for Loading into Temporary Table
file_write=open('<%=odiRef.getOption("FILE_PATH")%>/temp_file.ctl','w')
print >> file_write , 'OPTIONS (SKIP=1,ERRORS=0,DIRECT=FALSE )  LOAD DATA '
print >>  file_write , 'INFILE "<%=odiRef.getOption("FILE_PATH")%>/temp_file.txt" '
print >>  file_write , 'BADFILE "<%=odiRef.getOption("FILE_PATH")%>/temp_file.bad"'
print >>  file_write , 'DISCARDFILE "<%=odiRef.getOption("FILE_PATH")%>/temp_file.dsc" '
print >>  file_write , 'DISCARDMAX 1'
print >>  file_write , 'INTO TABLE ODI_TEMP_10G."file_temp"  '
print >>  file_write , 'FIELDS TERMINATED BY '+delimiter
print >>  file_write , 'TRAILING NULLCOLS'
print >>  file_write ,'('+column+')'
file_write.close() ODI Option - FILE_PATH i.e path of the directory where the file Dynamic_struc.txt resides
Second Step in Procedure - Truncating Temporary Table Technology- Oracle and Schema - Respective schema where the Temporary table resides
TRUNCATE TABLE TEMPORARY TABLE Third Step in ODI Procedure - Calling SQLLDRCommand on Target - Technology- Sunopsis API
Command on Source- Technology Oracle and Schema - Respective schema where the Temporary table resides [ Reason so that i can fetch the userid and password for SQLLDR from ODI itself ]
OdiOSCommand "-OUT_FILE=<%=odiRef.getOption("FILE_PATH")%>/TEMP_FILE.out" "-ERR_FILE=<%=odiRef.getOption("FILE_PATH")%>/TEMP_FILE.err"
sqlldr "control='<%=odiRef.getOption("FILE_PATH")%>/TEMP_FILE.ctl'" "log='<%=odiRef.getOption("FILE_PATH")%>/TEMP_FILE.log'" userid=<%=odiRef.getInfo("SRC_USER_NAME")%>/<%=odiRef.getInfo("SRC_PASS")%>@<%=odiRef.getInfo("SRC_DSERV_NAME")%>
Sample Output Process.
Dynamic_struc.txt - The Metadata File
record
{final_delim=end, delim=';'}
AGE_MIN;VARCHAR2(10),
AGE_MAX;VARCHAR2(10),
AGE_RANGE;VARCHAR2(10),
)Source Data
AGE_MIN;AGE_MAX;AGE_RANGE
0;19;Less than 20 years
20;29;20-29 years
30;39;30-39 years
40;49;40-49 years
50;59;50-59 years
60;110;60 years or moreCTL File Generated
OPTIONS (SKIP=1,ERRORS=0,DIRECT=FALSE )  LOAD DATA
INFILE "C:/temp_file.txt"
BADFILE "C:/temp_file.bad"
DISCARDFILE "C:/temp_file.dsc"
DISCARDMAX 1
INTO TABLE ODI_TEMP_10G."file_temp" 
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
(AGE_MIN,AGE_MAX,AGE_RANGE)I have tested the process by reducing columns too and random too and it seems to be working although not all condition :)
I have uploaded the procedure and here is the link where you can download.
http://cid-d7195c8badc40d0b.office.live.com/embedicon.aspx/odiexperts.com/TRT^_LOADING^_DYNAMIC^_DATA.xml
Please let me know if you need any help. You would need to tweak lots of things according to your environment .
Hope this helps