OO Batch Model and optimised Java for batch???

Question

Hi All,
I'm looking to see if there is any literature of OO models for batch processing and optimising of batch java.
Thoughts & comments welcome.........
I have an existing batch process running on a mainframe which is very successful. We would like to leverage this by building a similar batch process to run 'anywhere' so likely options are Java/Unix.
There are many patterns/models etc for OO based GUI / interactive processes but very few for (that I have found) for batch.
I have worked mainly with mainframe batch and online applications and come with the baggage that activity that can be processed in batch should be to avoid overloading the online container (CICS region, web server etc).
I believe that this continues to be true, as well as the particular data we are processing benefits from efficiencies of batching the data together to store eventually on tape.
In view of not finding any literature (which I doubt is the case) it seems that the problem is the same, so probably the solution is also similar.
In the procedural solution, a Jackson (or similar), would have been designed which would then reflect the procedures build into the code.
I expect that if instead of procedures classes where defined, certainly at a higher level then the design would still be ok.
(So a the higher level you have a 'main' class, which instansiates a 'read' io object, a processing object which handles the actual processing activity and a write io object).
The level to which would would combine procedures together, or further split them out, would then be the main point of discussion.
( However am open to the above suggestion being completly wrong).
Then there is efficient configuration when processing........
When running on the mainframe the code is loaded once, the memory for all the working storage strucures created. When actually processing there is no instantiating classes, or running the garbage collector etc. I re-use the same memory for each new record red in / processed / written out and all the code is normally loaded once when first called and the same code is re-used until all records processed.
Is there any way that I can replicate this within Java either in it's own JVM or running in a container such as websphere? When processing the volume of data that we do (20 million db entries + 40GB of document data avg) then anything not optimised is costing money and available processing time.

Answer 1

I suspect that batching is underused thoughrather
than overused.Can you elaborate on that? What kind ofconditions
would you advocate batching for? Running daily, monthly, etc reports. Or something
that feeds those.I don't disagree, I just don't even see this as 'batching'. Batching to me is when you take something that could be done incrementally and purposely doing it in large groups at set times or time-periods. If you have a daily report and you do it daily, you're just doing the most obvious approach. It might not even be the most efficient.
>>
I have some experience working with batch java
applications running in Unix. And I can tell you
that they did not improve anything. I suspect I would agree with that. I am not
advocating that the batching be done in java. Just
that idea that an 'incremental' process that requires
moving data versus a 'batch' process that doesn't
isn't something that I would normally consider a good
idea.I think we are thinking about different things. I'm really just talking about incremental or real-time vs. batching.
They were
actually the source of many of our issues. That
added abitrary time lags during times of lowvolumes,
sometimes adding 30 minutes or more to theprocessing
of a transaction as it waited for the nextscheduled
batch. They also made our backlogs worse in timesof
high volumes because the incoming data flow was
uneven, we would often get big batches of datafrom
partner systems (more batching, gotta love it)that
hit us when our batch process was sleeping. 5,10,
15 minutes would pass where the server ran at 10%
capacity while huge backlogs were piling up. It
didn't make anything better. It was just causing
idling.
What was the timeliness requirements for the
processing? Did it need to be completed by 2am in
the morning? Or could it have really just been
completed on demand?It was B2B transactions ASAP was the time requirement. I guess the upper limit was 6 hours or so. But batching didn't really decrease the processing time per transaction anyway and the server was never dedicated to the batch or anything so there were still context switches.
It just seems to me that in Java with all the nice
threading we have access to, the server shouldnever
be idle and if you cannot handle your volume youare
better off adding more servers, not attempting to
batch things.I have created applications that were intended to run
'batch' jobs which could be spread across servers.
Those particular processes had to finish within a
very narrow time span as well - about two hours as I
recall. There was an incremental as well as batch
functionality that needed to be run for this. The
batch functionality ran on the database. The
incremental took the batched results and handle the
incremental part.
Although management was never willing to dedicate
more than one server to the processing so I guess it
wasn't that important to them.
I have seen apps that claimed they were 'fast'
because they did all of the incremental processing
outside of the database. The design required moving,
literally, the entire database over the network to
other servers which would then process it.
Processing it in the database would have taken
orders of less time. And that was time sensitive
data. I can't remember if that app allowed for
multiple boxes to do the processing. I do know that
the people working on it could never figure out the
bottleneck (it was scaling to something like 12
hours a day which was not acceptable.)I guess I don't see doing it on the DB as implying batching. We use triggers to drive processes in Java, COBOL, whatever.