Cache as queue

Question

For my application, I need to periodically store a lot of persistent messages in a queue. Basically, the application takes periodic syncpoints in the form of persistent messages (representing all "dirty" objects that are not yet synced) pushed into queues. An offline (separate JVM) process periodically reads the queues and updates the database.
The requirement is to store quite a lot, say, 20,000 messages in multiple queues in a single JMS, local transaction. I started researching ActiveMQ but it hangs after a few large-batch transactions. My question is: could I use a cache as the queue? If I use an "overflow-to-disk" cache with a <high-units> value of 0, then every item written to the cache would go to disk (persistent "messages"). A separate JVM (with local.storage=false) that belongs to the same cluster could read from the same cache and update the database. The queue indirectly acts as a sorta/kinda transaction log of syncpoints and hence the messages must be persistent and all-or-nothing. (either the entire batch of messages is written or none)
I would appreciate your thoughts/inputs in this regard. Does the above approach make sense or is there a better approach?
Thanks
Ghanshyam

Answer 1

The ideal scenario, of course, is that the DB is always up AND writing to the DB is quick. However, both are false assumptions in my case.
The system is a transaction processing system. If the JVM crashes, I would need to go back to start-of-day to reprocess, which is not an option. Instead, if I take syncpoints to the DB, say, every 10 minutes, I would be able to go back to the last syncpoint taken and hence would only lose about 10 minutes worth of processing. Think of the syncpoint as the "last good system image" that Windows NT allows you to go to in case of a crash. The syncpoint involves writing dirty cache entries to the DB. Writing to the DB achieves two things: 1) there is a web-app that needs to read from the db, so the data needs to be written to the db in a timely manner 2) in case of a crash, or even as part of normal system startup/shutdown, the system just reads the last syncpoint from the db, initializes itself and resumes from there. (imagine that every table has a syncpoint_id column, whose value could be the time that the syncpoint was taken).
Now, since writing to the db is slow or the db may be down, to achieve purpose 2, I can atleast write to some form of local, persistent storage. That way, I can assure myself that I would just need to reprocess just 10 minutes in case of a crash and not have to go back to start-of-day. Effectively, it kinda works like a database transaction log that helps the database to do crash recovery. Of course, if nothing else woks, I can always fall back on writing to a local Derby database and then syncing it periodically with the main db.
Going back to the overflow-to-disk cache, if the high-units is set to zero, that should guarantee that every cache write goes to disk, so in case of a JVM crash, wouldn't the cache still retain the cache-items from disk when the JVM is restarted?