Help Required with PL/SQL loading logic

Question

Hi All,
We have a simple pl/sql code which does loading(insertions) of records into the table. This pl/sql code actually refers to an translation website URL(microsoft based) and inserts the translated text into our table. What happens is sometimes this URL gets blocked out due to some issue. So we are forced to restart our entire operation of loading once again. Actually we have a commit at the end(outside the loop). on the other hand we cant keep commit inside the loop as it may pose an overhead to performance since its commiting once every insert happens.
What we are expecting is something like this. Say we have 10,000 records. the loading fails at 5500 record. next time when we start again we want to start from 5501 record and NOT from beginning. In informatica kind of ETL tools we have an option to do this. However not sure if we can do this through PL/SQL.
we thought of keeping counter variable and keep it incrementing for each and every insert happens. but before doing this we have to order by our table according to primary key. This is taking huge amount of time since the table is very big.
can you please tell any alternative to this?
Regards,
Paul

Answer 1

If I understand correctly, you are doing something as follows:
// process a defined data set
for c in (select .. some data... ) 
loop
    // translate the data (via a web service call)
    Translate( c, translatedOutput )
    // insert the translated data
    insert into some_table translatedOutput
end loop
commitIn principle, nothing wrong with this approach. And yes, committing inside the loop for every x number of rows will increase the complexity of code and could perhaps violate ANSI SQL standards of fetching across commits.
A better approach would be to consider parallelising this process. Run multiple copies of that code - but provide each copy of the code with a small and unique data set to process. So instead of doing 10,000 rows in serial, you schedule a 100 jobs and give each a unique set of a 100 rows to translate.
Let's say your job queue size is set to 20. This means up to 20 of these processes will run in parallel. Should one fail, the job manager will automatically reschedule it for execution (up to 16 times after which it will be marked as broken).
So not only does this type of approach enables you spread the processing load, it also enables you to scale your processing (you can handle much larger data volumes like this, and more powerful server h/w will enable you to use more parallel processes). In addition, restart is automated courtesy of the Oracle job manager.
Compare this type approach with committing inside a loop.. The latter buys you what ito flexibility and performance and scalability? Nothing really - it only makes the process a lot more complex as it needs to be able to correctly restart after a failure. You will not be able to process bigger volumes of data. You will not be able to increase existing performance.
One needs to carefully consider the approach used to processing data - serialisation is inherently limited. Restarting serialised processes is difficult and complex.