How To Managed Serialized Data Over Multiple Application Versions

Question

Lately I've been struggling with deciding how to deal with serialized data between application versions. Assume you have some software that has to be able to save data files. As the application evolves, the data format changes. The application needs to be able to load data saved from prior application versions.
When I first started out I just used a Serializable class. It was quick and easy. However, as newer versions used slightly different data formats, I had to come up with a means to upgrade the data to the latest version. Initially I created a process where the latest data class knows how to upgrade from prior data class versions. Then, when loading data in the application, if the class doesn't match the latest data class version, it checks to see if it matches one of the versions that can be upgraded.
This sort of works, but still has one glaring problem. The old data classes must be etched in stone; never altered. For ever and ever. You can get away with certain simple updates to older data classes by defining a serialVersionUID and keeping it's value the same. But this doesn't always work. What if my overall package structure changes? I have to ensure that the old data classes remain in the same older package structures. Not ideal.
So I've started considering other ways to serialize data...a process where the data can be broken down into core Java classes. There are serializers for serializing to XML, but I don't feel that XML is an appropriate storage format for the data. I'm also considering using a custom built serializer that uses introspection in some fashion.
Before I go any further, however, I'd like to solicit some suggestions. Do you know of any serialization packages that might help me? Do you have any suggestions on what might be a good custom solution? Will Externalizable make this easier? (I've never used Externalizable before...looks like maybe it could help, but even that appears to be vulnerable to refactors that change package structure).
Thanks for your thoughts.

Answer 1

I've continued to work on my own custom serialization process as described in my previous post. Here is how it currently works:
In the example there is a class named ExampleSubClass identified by the String "xsub" of version "1.0". It is saved and then reloaded into a class named RevisedExampleSubClass identified by the String "xsub" of version "1.1". The String "xsub" identifies what the class represents, which allows the class and package to change as needed. Fields can be added, removed, or converted through the optional serialConvert method.
public class ExampleSubClass extends ExampleSuperClass {
     private static final String serialId = "xsub";
     private static final String serialVersion = "1.0";
     ExampleAggClass exNullAggClass = null;
     String exString;
     ExampleNonXCSerialClass nonXCSerialClass;
     String exNum;
     ExampleAggClass conAggClass;
public class RevisedExampleSubClass extends ExampleSuperClass {
     private static final String serialId = "xsub";
     private static final String serialVersion = "1.1";
     ExampleAggClass exNullAggClass = null;
     String newStringA;
     String newStringB;
     String exString;
     Integer exNum;
     ExampleNonXCSerialClass nonXCSerialClass;
     String conAggString;
     private void serialConvert(Converter converter, String fromVersion) {
          if (fromVersion.equals("1.0")) {
               converter.addField("newStringA", String.class, "new string!");
               // intentionally not adding newStringB to see if warning is thrown
               converter.convertField("exNum", new ConvertScript<String,Integer>(String.class, Integer.class) {
                    @Override
                    public Integer convert(String fromValue) {
                         return new Integer(fromValue);
               converter.convertField("conAggClass", "conAggString", new ConvertScript<ExampleAggClass, String>(ExampleAggClass.class, String.class) {
                    @Override
                    public String convert(ExampleAggClass fromValue) {
                         return fromValue.getAggString();
// and then in some other class...
     ExampleSubClass example = new ExampleSubClass();
     System.out.println("Serializing class...");
     XCObjectOutputStream xcoos = new XCObjectOutputStream(new FileOutputStream(file));
     xcoos.writeObject(example);
     xcoos.close();
     System.out.println("Deserializing class...");
     XCObjectInputStream xcois = new XCObjectInputStream(new FileInputStream(file), RevisedExampleSubClass.class);
     RevisedExampleSubClass resc = (RevisedExampleSubClass) xcois.readObject();There is still a little more work to do to fully support arrays and collections, and I need to develop a rigorous test suite for it. But so far it's looking good.
Edited by: Skotty on Jul 18, 2010 5:31 PM