ODI target as flatfile -- split

Question

Hi can anyone let me know how to split the flatfile output in ODI. I do know that there is a tool called xmlsplit for xml files but i don't know how we can split the flatfiles. Please guide me on this. Thank you!

Answer 1

1. Create 2 ODI variables min_row_count, iteration_counter. Set all of them to 1.
2. Create another variable called max_row_count and set it to 50000.
3. Create another variable called total_row_count and put a query in refresh block: SELECT count(*) FROM your_source_table
3. Create a temporary interface (if you are not comfortable with that, create the target table and bring it to ODI model) that uses your table as source and map all the columns plus an extra column (let us call it rownbr) which should be mapped as row_number() OVER (ORDER BY <some other column>). Execute this if you are in 10g (no need of this if you are in 11g).
4. Create another interface that will write to a flat file. Use your temporary interface (or the table if you created it) as source. Use flat file model as target.
5. Create a filter like: source.rownbr between #min_row_count and #max_row_count
6. Find a suitable standard IKM that will load to a file. use it for the interface. Generate a scenario.
7. Create a package where you set the iteration_counter to 1. Invoke the scenario in the next step. If it succeeds, create a OS command where you will copy the output file to <some name>.#iteration_counter.dat file in the desired directory. Once the command succeeds, increment the iteration counter by 1. Increment the min_row_count and max_row_count by 50000. Test if min_row_count > total_row_count. If yes, exit the loop (and send mail or something). Else point to scenario run step again. What you have is a loop which will use iterate the scenario over your table 50k rows at a time.