Coco Encoder Multiline Processing

Question

Hi Jun Xu
I first want to make clear that I appreciate the work you have done for the Coco Encoder Framework a lot! It seems to be a good base to start with copybook and Cobold data files. We are currently trying to integrate that in our new system on a direct non-OpenESB basis before we possibly migrate to OpenESB.
It seems that I have not understood some crucial part of the Encoder Framework:
1. The transformation of simple copybook files to XSD works like a charm. Although there is a limitation of having only one "01 item" per copybook: "CCCB4104: Copybook COBOL Copybook XSD Builder does not support processing of copybooks with multiple 01 items ". OK, we can get around this limitation by splitting the copybook file into N files containing each one "01 item", splitting the data files accordingly and remerging the final XML back into one at the end. Although this seems a bit clumpsy, it's a way. Or is there a simplier way to achive transformation of multi 01-item containing copybooks into a corresponding XSD and transform the data into XML?
2. Trying to achive that, I studied the "TestRuntime" file you deliver for testing the transformation to XML. In the test only one row is transformed. However, our data files contain more than one row, and we would like to translate the whole Cobold data file into an XML structure. For this, I don't see a concept in the Coco Encoder Framework and it appears to be a crucial limitation to it - at least as long as I don't know how it is intended to be done. Is there a simple way to achive the multi row transformation?
3. Trying to get around this limitation I extended the code you provide for TestRuntime by introducing a loop which peeks on a BufferedReader to process line after line. The extention is looking more-less like that:
final FileOutputStream fileOutputStream = new FileOutputStream(outputFileName, true);
final StreamResult streamResult = new StreamResult(fileOutputStream);
do {
    int readResult;
    do {
        input.mark(1);
        readResult = input.read(buffer);
    } while (buffer[0] == '\r' || buffer[0] == '\n');
    if (readResult == -1) {
        break;
    input.reset();
    final Source source = coder.decodeFromReader(input);
    printToFile(source, streamResult);
} while (true);
fileOutputStream.close();The problem I meet here is that the BufferedReader is not advancing properly to the end of the line after decodeFromReader has been executed. Instead it hangs around somewhere in the line. Certainly I could advance the Reader myself, but something is going wrong here.
I would appreciate some pointers, hints, help and roadmaps in order to be able to decide whether the framework is already mature to be integrated into a productive system.
Again, thanks for the work you have done. I appreciate it a lot!
Cheers
Daniel

Answer 1

And another one from Jun Xu from 10. Mai 2008
Hi Daniel,
I looked at the COCo Encoder source code and noticed that the decoding process is always based on byte stream. With this said, when you call decodeFromReader, the character stream will be converted to a byte stream on the fly. The mechanism of doing so is that an OutputStreamWriter is kept internally and whenever some bytes are requested, some characters will be read from the Reader and in turn be written to the OutputStreamWriter and the bytes resulted from this writing process will be collected and eventually returned to whoever is requesting the bytes when there are enough of them. You can take a look the implementation of ReaderInputStream in the encoder framework code, which is under encodersl/encoder-fw.
In the above process, the COCo encoder might over read from the Reader while it tries to get enough bytes to parse. For example, if the total content in your Reader is 201 characters and it represents two lines. The first line ends at 101st character. When first 100 characters get converted to bytes, it becomes 150 bytes. COCo encoder converts characters to bytes in a buffered fashion and on demand. Assuming the buffer size is 60 (in real implementation , it might be 1024 or something else), when the parsing process moves along, to provide 150 bytes, the Reader might have been read for 120 (2 times the buffer size) characters.
All I want to say is that there is nothing wrong here and what you have seen is the expected behavior. But it does make controlling Reader's current position harder and may cause a little bit confusion.
Thanks.
Jun