Append to a compressed file

Question

I have some test results that I want to save into a test result file. Since I will be accumulating many test results, I want to keep overhead down, so I would like to simply append the result to the end of the file. Here is what I have done...
1) Create an empty array, flatten to XML and write to a file.
2) When I want to add a record, I take the record control, flatten to XML, and then overwrite the </Array> at the end of the file with the XML record. I then append another </Array> onto the end of that to properly close it out.
3) I can then read this XML file back into an array of test record structures at a later time.
I chose to do it this way, because when I am generating the test records, I do not want the overhead of "file read->XML Parse entire file->append code->XML export appended structure->File write". I figured that by simply appending the XML record to the end of the file, I could cut down on a lot of this IO overhead.
The problem is that XML is a bit "chatty". I'd like to be able to read, write and append to compressed files (since XML compresses very well). The lvzlib routines only have a read and a write, but no append. My understanding of the compression schemes is that it makes a table of the common byte patterns and replaces them with shorter tokens. If I compress each record, I will have a table of tokens for each record that will basically be the same as all the other records. I can get better compression if I use one table for all records.
Has anybody implemented this, or do I need to dig into the guts of zlib?
Alternatively, can the normal File I/O routines be configured to work on a file in a compressed archive (like a zip file) and let the OS handle all the compression?
Thanks,
Brian Rose

Answer 1

Mister Rose wrote:
I have some test results that I want to save into a test result file. Since I will be accumulating many test results, I want to keep overhead down, so I would like to simply append the result to the end of the file. Here is what I have done...
1) Create an empty array, flatten to XML and write to a file.
2) When I want to add a record, I take the record control, flatten to XML, and then overwrite the </Array> at the end of the file with the XML record. I then append another </Array> onto the end of that to properly close it out.
3) I can then read this XML file back into an array of test record structures at a later time.
I chose to do it this way, because when I am generating the test records, I do not want the overhead of "file read->XML Parse entire file->append code->XML export appended structure->File write". I figured that by simply appending the XML record to the end of the file, I could cut down on a lot of this IO overhead.
The problem is that XML is a bit "chatty". I'd like to be able to read, write and append to compressed files (since XML compresses very well). The lvzlib routines only have a read and a write, but no append. My understanding of the compression schemes is that it makes a table of the common byte patterns and replaces them with shorter tokens. If I compress each record, I will have a table of tokens for each record that will basically be the same as all the other records. I can get better compression if I use one table for all records.
Has anybody implemented this, or do I need to dig into the guts of zlib?
Alternatively, can the normal File I/O routines be configured to work on a file in a compressed archive (like a zip file) and let the OS handle all the compression?
Thanks,
I'm not sure what you mean by appending. If you mean appending new data to an existing file in a ZIP archive this can't be done in any way. The ZIP file is not structured in a way that would allow appending data to an internal file stream inside the archive. I'm not even sure there is any compression scheme that would allow that. The only way for all standard compression schemes for this is to extract the file append the new data and then replace the existing file in the archive with the new one.
If you mean appending a new file to an existing archive then the newest released version 2.3 of lvzlib can do it. The ZLIB Open Zip Archive.vi function has an append mode input. Previously this was a boolean that indicated if a new file was to be created or if the archive should be tacked to the end of an existing file. In the new version this is an enum with an additional entry (adding to existing archive).
Rolf Kalbermatter
Message Edited by rolfk on 10-04-2007 09:21 AM
Rolf Kalbermatter
CIT Engineering Netherlands
a division of Test & Measurement Solutions