Speeding up Data Merge

Question

My company has really embraced the variable data craze, with one problem. It takes far too long. I've been primarily using InDesign's Data Merge function, but I routinely get system lockups and InD crashes even when merged the simplest of documents. I often find myself having to split the file into several small (less than 1000, or sometimes, less than 500) pieces in order to get the merge to complete. This is a huge ordeal when dealing with lists over 10,000.
These haven't been very complex documents, for the most part; usually some text and a graphic or two, like a signature or a PDF form.
However, we've done one very complex merge, a very nice peice with lots of color, and it took - and this is no exaggeration - over 30 hours to complete the PDF export.
There's got to be a better way, doesn't there? Will throwing RAM and/or new processors at the problem help much? I'm running CS3 on a G5 with 1GB RAM (yeah, I know, but I don't control the purse strings).

Answer 1

I've found a neat method to avoid this kind of InDesign bulkiness. Basically, you can use the Watermark feature of Acrobat 8 to take a PDF of only the variable data, on otherwise blank pages, and add the PDF of the static text as a watermark background on every page.
You can even nudge the watermark image around until the variable and static areas match up precisely, then apply it to all pages (or a subset, or odd/even pages only, etc.)
I use InDesign's Data Merge and Text Variables for some complex labels that are driven by a spreadsheet, and it's been sometimes unreliable. The Acrobat watermark feature can go a long way to making it more reliable I think.
Darryl