Loading large documents

Question

I need a way to load large documents without major performance issues.
At present, the best I can find in Swing itself is the PlainDocument class, which has issues once it crosses the size of about 1MB. As it gets closer and closer to 10MB, it gets slower and slower, and even higher just gets unacceptable.
I started off writing a custom implementation of Document, but the need to implement the listener methods bogged me down and I can't progress any further without further help.
I find it hard to believe that nobody would have written a Swing editor that can handle over 10MB of text, but I can't find anything with Google, so...
Has someone out there already tackled this problem?

Answer 1

I think you should create a document which reads the text from the
file whenever the data is requested by getText-method in the document.
You could also use some caching of the previous read text in order to reach a good performance.
For instance you could reserve a buffer (byte-array) of about 500 kb,
which I would split up in 500 arrays of 1000 chars, since 1000 chars should be enough to fill at
least a screen and somewhat above.
I would propose to keep a map for holding these arrays, and coupling them to the
offset in the file from which they are read.
By alligning the offsets from which you read in the file with the size of such an array,
you will never read overlapping parts in your buffer.
You will also need some mechanism for remembering which array is least frequently used or least
recently used and may be replaced by the newly read data once the buffer is full.
You will also need some mechanism for adding or changing data from the file,
since you cannot change the data on the fly in the file. You should use some caching of this
data (perhaps in some other file) and mix this cached data with the cached buffer when the
getText-method is called.
While saving the new file this data should be mixed with the original file and written to a new file which
will then replace the old file. Afterwards all buffers should be cleaned (especially the change-buffers).
You could adapt some auto-saving mechanism for doing this save after a specified time,
or when the change-buffers are growing to large.
Finally you could also opt to split the read-buffer in a buffer using strong-references to the arrays and
an additional buffer which uses soft- or weak-references to the arrays.
The strong-references will make the first buffer a solid reliable buffer of fixed size (minimum),
while the soft- or weak-references will make the second buffer a more soft buffer which can lose
data when the memory-requirements of the application are high.
This way you get a maximum size buffer when memory-requirements for the application do not claim the
soft-references of the buffer and you will always keep a minimum buffer for the application.
Data will always flow from the solid to the soft buffer and never backwards.
kind regards,
to add to the