Error exporting NamedCache to file via PofWriter

Question

I'd like to dump some NamedCaches with a large number of entries (> 200,000) to files, and then be able to re-load the NamedCaches from those files. I realize that there may be a feature in an upcoming version of Coherence to do this sort of thing, but I'm using 3.5 for now and need to roll my own for the time being.
I already have working code that takes the entire NamedCache and writes it as one single Map to a file using the writeMap method of a PofBufferWriter (a variation of the code shown below).
The problem is that attempting to load a pof file containing one single Map with 200,000+ entries usually causes a out of memory error. This is because it is not possible to know ahead of time the number of elements in the Map in the pof file, which means it isn't possible to set the initial capacity of the Map that the contents of the file are being read into via the PofReader.readMap method. The consequence of this is that the Map's growth algorithm kicks in, doubling the size of the map over and over, and pretty soon the map is giant and there is insufficient heap. It's not that the objects in the file wouldn't fit in memory, the problem is that the Map growth algorithm just creates a Map too big for the heap.
So, I've been thinking about a variation on the theme that avoids the Map growth problem. A variety of potential solutions come to mind, and I've tried many of them.
First, I could write an int in position 0 of the PofWriter stream to say how many elements the Map in position 1 contains. That would allow me to compute an initial capacity for the Map that the file is loaded into (via PofReader) that avoids the Map growth algorithm problems.
Second, I can iterate the elements of the NamedCache and build a number of smaller sub-Maps of a known size, and then write those sub-Maps in consecutive positions of the PofWriter stream, probably with an int in position 0 that says how many Maps follow.
A third alternative is to iterate the elements of the NamedCache and write each key/value pair to the Nth posiition in the PofWriter stream (again, with an int in position zero saying how many key/value pairs follow).
The code shown below is basically strategy #3, but whether I try strategy #1 or #2 or #3, I always get the following exception upon the second invocation of a "write" method on the PofWriter:
<pre>
java.lang.IllegalArgumentException: not in a complex type
at com.tangosol.io.pof.PofBufferWriter.beginProperty(PofBufferWriter.java:1940)
at com.tangosol.io.pof.PofBufferWriter.writeObject(PofBufferWriter.java:1426)
at com.mycompany.util.coherence.NamedCacheExporter.exportCache(NamedCacheExporter.java:182)
</pre>
Just to be clear, if I only write one entry to the PofWriter (e.g., the entire NamedCache), it does not throw this exception.
I am confident that all of the objects being written implement PortableObject (correctly) and are registered in the pof config file. The NamedCacheEntry object used below is a simple key/value wrapper that implements PortableObject and has only two Object-valued instance variables named "key" and "value".
<pre>
public static void exportCacheNamedTo(String cacheName, File file)
throws Exception
WrapperBufferOutput wrappedBufferOutput = null;
try
NamedCache cache = CacheFactory.getCache( cacheName );
FileOutputStream fileOutputStream = new FileOutputStream( file );
BufferedOutputStream bufferedOutputStream =
new BufferedOutputStream( fileOutputStream, 1024 * 1024 );
DataOutputStream dataOutputStream = new DataOutputStream( bufferedOutputStream );
wrappedBufferOutput = new WrapperBufferOutput( dataOutputStream );
ConfigurablePofContext pofContext =
(ConfigurablePofContext) cache.getCacheService().getSerializer();
PofWriter pofWriter = new PofBufferWriter( wrappedBufferOutput, pofContext );
//this works fine
//pofWriter.writeMap( 0, cache.getAll( cache.keySet() ) );
//this fails with the above error
//pofWriter.writeInt( 0, cache.size() );
//pofWriter.writeMap( 1, cache.getAll( cache.keySet() ) );
//this fails with the above error
int pofIndex = 0;
//index 0 contains an int that says how many more POF indexes to expect
pofWriter.writeInt( pofIndex++, cache.size() );
//index N contains a NamedCacheEntry (a simple key/value wrapper)
for ( Object o : cache.entrySet() )
NamedCacheEntry nce = new NamedCacheEntry( (Entry) o );
pofWriter.writeObject( pofIndex++, nce );
finally
if ( wrappedBufferOutput != null )
wrappedBufferOutput.close();
</pre>
Anybody see what I'm doing wrong or have any idea what the cause of the exception above is?
Thanks in advance
Edited by: dm197 on Feb 20, 2010 3:07 PM
Edited by: dm197 on Feb 21, 2010 9:49 AM
Edited by: dm197 on Feb 21, 2010 9:51 AM

Answer 1

Hi DM
Can't you just use the ConfigurablePofContext you have to serialize the entries to the WrapperBufferOutput
public static void exportCacheNamedTo(String cacheName, File file)
throws Exception
    WrapperBufferOutput wrappedBufferOutput = null;
    try
        NamedCache cache = CacheFactory.getCache( cacheName );
        FileOutputStream fileOutputStream = new FileOutputStream( file );
        BufferedOutputStream bufferedOutputStream =
            new BufferedOutputStream( fileOutputStream, 1024 * 1024 );
        DataOutputStream dataOutputStream = new DataOutputStream( bufferedOutputStream );
        wrappedBufferOutput = new WrapperBufferOutput( dataOutputStream );
        ConfigurablePofContext pofContext =
            (ConfigurablePofContext) cache.getCacheService().getSerializer();
        for ( Object o : cache.entrySet() )
            NamedCacheEntry nce = new NamedCacheEntry( (Map.Entry) o );
            pofContext.serialize(wrappedBufferOutput, nce);
    finally
        if ( wrappedBufferOutput != null )
            wrappedBufferOutput.close();
}And then read them back like this:
public static void importCacheNamedTo(String cacheName, File file)
throws Exception
    WrapperBufferInput wrappedBufferInput = null;
    try
        NamedCache cache = CacheFactory.getCache( cacheName );
        FileInputStream fileInputStream = new FileInputStream( file );
        BufferedInputStream bufferedInputStream =
            new BufferedInputStream( fileInputStream, 1024 * 1024 );
        DataInputStream dataInputStream = new DataInputStream( bufferedInputStream );
        wrappedBufferInput = new WrapperBufferInput( dataInputStream );
        ConfigurablePofContext pofContext =
            (ConfigurablePofContext) cache.getCacheService().getSerializer();
        while (wrappedBufferInput.available() > 0)
            NamedCacheEntry nce = pofContext.deserialize(wrappedBufferInput);
            // Add the entry back into the cache
            cache.put(nce.????, nce.????);
    finally
        if ( wrappedBufferInput != null )
            wrappedBufferInput.close();
}I have not tested this so I cannot guarantee it works, especially the wrappedBufferInput.available() > 0 condition in the while loop. I am not sure if that is the best way to check for the end of the stream.
JK