Indefinite WIP record life

Question

We have a situation where a document published using IDS on Day 1 needs to be included with a new business policy or subsequent renewals that could happen on Day 2 or Day 5,000 - basically, we need to keep that transaction in its original state so it can be republished any time as long as the policy is active. We have an approach to combine the first document with the subsequent requests by using WIP and some bundling processes. My concern is with the "keeping original transactions indefinitely" part. In theory, we could end up with 200,000+ records in a WIP table, with even more corresponding rows in a WIPData table. I am a little leery of the performance of such a database, and am curious if anyone has gone down a similar path to handle this type of requirement?
I imagine we don't really use WIP like most customers, in that we've never maintained a system where we keep transactions for users to work on at a later time; typically, it is just a holding area so we can queue up transactions for batch print (emptying the database every day) or using it for an iDocumaker(ish) system where users manipulate transactions via WIPEdit and complete them in a single session. To that end, I don't think we've ever had more than several thousand rows in a WIP Database, and the performance has been great, but this process is considerably bigger.
We are using SQL Server if that makes any difference.
Thanks!
Tony

Answer 1

Tony,
First of all you are right to be concerned with this approach. I can tell you with certainty that it can be a problematic solution - I am familiar with environments with as many as 800,000 WIP transactions in process and it can be quite burdensome on the database and backend fileservers (if storing WIP data on the file system).
There are two approaches to solve this but it depends on whether or not you need to be able to change the WIP transaction. If you don't, then you can generate and archive the transaction (as PDF for example), and then use rules to pull back that image and incorporate it into a new transaction. Check out the AddMultiPageBitmap rule in the Documaker Rule book.
The other approach is a bit similar but gives you more flexibility if you need to modify the previous transaction data. There is a little-known function that you can take advantage of (in fact we ended up using this function in the aforementioned environment with 800,000 WIP transactions). This function allows you to store a transaction in the legacy Documaker Archive format (that is, NAPOL + Formset data) which is very compact, and then you can "rehydrate" the archive transaction and add to an existing transaction. The rule is called LoadFormsetFromArchive, also in the same book. I won't go into gory details here, I'll let you explore each option and come back if you have followup questions.
You didn't mention what Documaker version you're using but I'm guessing it's <12.0 so here are some links:
RUL_BOOK
(11.3) http://docs.oracle.com/cd/E15200_01/rul_book.pdf
(11.4) http://docs.oracle.com/cd/E14902_01/rul_book.pdf
(11.5) http://docs.oracle.com/cd/E16256_01/rul_book.pdf
Also, I'm sure this goes without saying but making sure your database is properly indexed and maintained goes a long way in keeping the system in good shape. Having a million rows in a database isn't problematic as long as you have the proper hardware and maintenance routines in place (Example: using 32bit database server software on 32-bit architecture hardware while trying to maintain an index in-memory to prevent swapping on searching a million rows in a database table is not going to work; if you want to hold that index in memory you need more addressable space so 64bit is the way to go). Similarly, if you're storing files on the file system, some operating systems don't function well with hundreds of thousands files in a single directory, so you need to use a leafed directory pattern.
-Andy