Batch processor design

Question

Hi my question is there a good resource to see how batch-processing design ( e.g. existing app, case-study...etc)
So how we can implemented a rule-based implementation in set-logic?
I work at this company that the main processing is batch-data-processing in nature.
It receives some file feed, and than some business logic processing and then pass it to another company's program to do the fulfillment.
When the main app is designed 8 years ago, the architect sees what kind of features it needs.
It is implemented in oracle. It uses a rule base engine to do this processing row-by-row.
It uses a set of dynamic-sql to implement every business rule. So if there are 10 rules, than every record will call a PL/SQL 10 times for that set of dynamic-sql
It is also heavy triggers for business logic so there is some-kind of locking package got implemented so when the batch processing is done. It will not got dead-locked.
So when I have the interview my boss says it is a OLTP machine and in a way it is true as it is designed work in a way row-by-row like OLTP even it is batch-processing in nature
My other co-worker and I both agree that it is a very sub-optimal design as it is not set-based operation even by nature it is.
I can see some good-things about have mulitiple-sql is similar to the object-design of extending. If one SQL-RULE is good than to build on top of that, we can just insert new dynamic-sql into the table and do the processing. The draw back is the performance is pretty sub-optimal.
Would you all share some experience in this topic?

Answer 1

>
Hi my question is there a good resource to see how batch-processing design ( e.g. existing app, case-study...etc)
>
Not that I am aware of.
>
So how we can implemented a rule-based implementation in set-logic?
>
Too general a question to be able to provide much of an answer. Especially given that you may have a very specialized custom implementation of the process being used now.
Custom implementations usually present several issues for conversion projects (such as replacement with new technologies or functionalities)
1. The original author/developer is no longer availlable.
2. The system lacks sufficient documentation.
3. The business rules being implemented are either not documented at all, poorly documented or the documentation is out of date and no longer matches the code that is actually being used.
Rule #1 is "if it ain't broke don't fix it'. If you so much at even touch the system you run the risk of breaking it completely, introducing new errors or making things worse.
{quote}
The draw back is the performance is pretty sub-optimal.
{quote}
So what? See rule #1. Does the current system meet the required service level agreements? Is there functionality needed that isn't currently available?
1. Determine if change is actually needed - In other words, what is the business reason for even thinking about changing things?
2. Determine the scope of the change that is needed - are we talking about minor changes or mods? Or major new modules and functionality? Are the current business rules getting the job done and we only need a couple more?
3. Determine if the required changes can be implemented by modifying the current system. You said 'It receives some file feed, and than some business logic processing and then pass it to another company's program to do the fulfillment'. So if you now need to add another feed for another company and you need pretty much what you have now why switch technologies? Well maybe the volume is 10 times what you have now and the current system can't handle it.
This would be a good case for doing the new feed as a separate project and use new functionality to do it. You can run it in parallel to get the kinks worked out and, ultimately, convert the existing feed to the new code later.
4. Determine the level of functionality that the business rules apply to. Data cleansing is often primarily done at the row level so this type of processing is very amenable to 'set' processing. Apply cleansing rule #1 to the entire table, then rule #2 and so on. A serialized 'batch' process rather than apply 10 rules to each row one at a time. Checks for dirty data (invalid nulls, data too long for column, invalid datatypes, etc) can be done in bulk.
Other functionality requires data from multiple tables: parent/child validation, lookup table validation, lookup code conversions from source values to target values.
In short you need to have a clear business objective and goals.
A. Project justification
B. Functional Requirements document
C. Technical Requirements document
B. And then define the project