Algorithm for online backup and restore of the entire grid

Question

Hello!
I am trying to write a utility program that backs up the entire grid data to a binary file and restores it at a later point in time. Assuming that the class files are not updated, I am using the following simple logic
1. Use the cluster and CacheService APIs, get the names of all named caches of types replicated, distributed and optimistic
2. Iterate through each named cache and use ExternalizableHelper.writeMap method to extract the data as byte arrays
3. Create a custom datagrid object holding a map of byte arrays by names
4. Write the custom object to a persistent storage
Similar logic to restore the data grid from the custom object on the disk.
I have the following questions
1. I am sure the above logic is not going to recreate the entire datagrid in the exact state it was backed up. For e.g. if I used write-behind, I must either flush or backup the write-behind queue. What other objects should
I consider backing up other than the data from the named caches? If I used the incubator patterns, I am guessing I need to backup the command objects and functor objects.
2. I do not have a need for other cache services now. If I need to use the same utility across different grid configurations, what other services should I include for the back up (overflow, external, paged external) ?
3. Are there best practices for such use cases? Any insights you can provide to enhancing this utility would be much appreciated.
Thank you
Regards,
Sairam

Answer 1

Hi Sairam,
If you want a non-intrusive solution then you cannot really use cache stores, as these would certainly intrude into the cluster. Cache stores would also not really be an option if the cluster already used another type of cache store - unless you write some sort of multiplexing cache store.
If your backup/restore solution needs to be a "client" (optionally over Extend) then you have two choices:
1. Build something that is run whenever you want to do a backup - this would usually be an Extend client application. This application would then read data out of the caches and write it to disk. With this option a backup might take a while if the cluster is large but running the backup is pretty easy.
2. Build something that is permanently connected to the cluster being backed up and uses Map listeners to listen for inserts and updates and stores the data to disk. This application could be either an Extend client or a storage disabled cluster member. With this option your cluster is always backed up in real time but you have the overhead of another process to manage.
If all your caches use POF serialisation then either of the above options could be written to just store the POF binary without having to deserialize it and could then be "bolted on" to any cluster. Re-loading stored binaries would need a bit of thinking about although a couple of options come to mind. If you just worked with the binary you might not need to worry about you backup application needing the pof-config of the cluster.
You backup application would need to either be told which caches to back up or be able to dynamically figure out what caches had been created. You cannot always do this from the cache config file if the cache names in the mappings section use wild card names.
As you can see there is lots to think about if you want to build something truly non-intrusive.
JK