Too Big to Serialize??

Question

I have run into the same problem someone else here mentioned before, namely that I get a StackOverflowError when I try to deserialize an object I successfully serialized. I wondered if the Java serialization routine (the default, since I didn't override any readObject or writeObject methods) had a bug and had worked itself into an infinite loop, but I think my object may simply be too large.
The data structure I'm trying to serialize contains hundreds of thousands of objects and has a complex pointer structure (used to sort the elements in more than one direction at once and also as a hash map) plus several thousand collection objects referencing small subsets of the set of objects in the large data structure. This means there are cycles in the object graph--serialization is supposed to be able to handle that, right?
Trying to serialize my object (using the writeObject method) initially resulted in a StackOverflowError, but I got results when I increased the stack size. Unfortunately, I still can't read them back in, even with a Java stack size of 10G and a native stack size of 8M.
Has anyone overcome a problem like this by writing your own serialization routines? I don't see how I'd get around having the same problem the default versions seem to be having--I think they're simply going into deep recursion on the pointers before coming back up.
Here is a snippet of the stack trace:
java.lang.StackOverflowError
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2163)
at java.io.ObjectInputStream$BlockDataInputStream.readInt(ObjectInputStream.java:2658)
at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1373)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1429)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1626)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1274)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1845)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1769)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1646)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1274)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1845)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1769)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1646)
[The last four lines are repeated over and over.]
By the way, I had the same problem I have seen reported (but not solved) here earlier with StackOverflowError not printing a stack trace, even when printStackTrace() is explicitly called on it, in Java 1.4.2. To get the stack trace above, I had to use Java 1.4.1_05.

Answer 1

I've discovered that serializing the object using writeObject to a FileOutputStream caused a StackOverflowError, even though it produced a file, so that could have to do with why it is getting a StackOverflowError trying to read in that file, or at least it is probably the same problem in both cases.
I get a very analogous error trying to write the object:
java.lang.StackOverflowError
at java.io.FileOutputStream.write(FileOutputStream.java:257)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1637)
at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1601)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1323)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1302)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1245)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052)
[The last four lines are repeated over and over.]
As I mentioned, I didn't override the writeObject method for any class, and I only make one call to writeObject on my main object--that one call results in this problem.
I wonder if the default serialization mechanism could be going about things in a dumb way that could cause problems for large, multiply-linked data structures like mine.
For example, suppose the pseudocode for writeObject looked like this:
writeObject(Object o){
writeObject(first field);
writeObject(second field);
writeObject(last field);
Then suppose I had a doubly-linked list of 100,000 listnodes like this:
class ListNode{
DataClass data;
ListNode next;
ListNode previous;
Then suppose I tried to serialize the first ListNode.
The serialization would behave like this:
write data;
serialize next:
---write data;
---serialize next:
------write data;
------serialize next:
---------write data;
---------serialize next:
There would be 100,000 function calls on the stack before it got to the end of the list, and it still wouldn't have finished serializing the first ListNode. Then when it got to the end of the list, it would continue serializing that object by proceeding through the previous pointers, so there would be 200,000 function calls on the stack before it finished processing the last ListNode. When you consider that my DataClass objects are themselves complex objects with pointers pointing, among other things, back to the ListNodes (I'm simplifying my structure for the purposes of discussion), it's easy for me to imagine that a serialization routine written in a simple way would simply fill up the stack before ever returning from its first function call. How many function calls would that take??
The only way I see around that problem would be to rewrite all the serialization routines for my objects in some way so they wouldn't be recursive, however I could do that. Am I missing something here?