Plain-text parser

Question

Hi everybody
The project I've been working on involves parsing plain-text, structured messages. These messages contain records of fixed length, and of several different types. Some of these record types have data & behaviour in common, so that it feels quite natural AND good practice to use inheritance
to illustrate this, here's (roughly) the grammar of my messages :
Message => MessageHeader Invoice* MessageFooter
Invoice => InvoiceHeader Record* InvoiceFooter
Record => RecordX | RecordY | RecordZas I said, RecordX, RecordY and RecordZ have data & behaviour in common (same applies for MessageHeader/Footer and InvoiceHeader/Footer) and are composed of many fields of different types
The first treatment I want to apply is to parse such messages from the filesystem and construct a graph of objects (in order to save it to DB afterwards via hibernate)
How would you organise a suitable and proper-OO parser package in order to minimize code redundancy, allow TDD... I have already written dozens of solutions, none satisfies me 100%...
If you want further info / description / excerpts of my precedent solutions, no problem ! I don't want to be TOO specific, since I have the feeling that there's some kind of generic pattern that could apply to this problem elegantly... Am I wrong ?

Answer 1

Parsing is seperate from the graph.sure ok about that
So what exactly do you think needs improvement?I don't like these long parse(InputStream) methods
that just call some "basic parser" (implementing
methods like parse, parseInt, parseShort, and so on)
again and again, especially when a lot of parsing
activity code is duplicated (same sub-sequences of
fields in records of different types)
It gives you something like that :
public RecordX parse(InputStream inputStream) {
....I don't see a problem.
You can generalizing it using a decription component if you wish.
But, speaking from experience, unless you have a lot of these which are very similar then doing so it makes it harder to maintain (due to special cases.) And it doesn't make it faster.
>
>>
And "many fields" doesn't say much. In particular
just because an attribute exists or doesn't exist
does not necessarily imply that inheritance is
needed. For example I am still a human whether I
have a job or not.what I meant is : some records of different types
have many attributes and the most part of behaviour
in common so parsing them in separate classes
(specialised parsers) induces lengthy methods and
lots of duplicate code... This couldn't possibly
scale if there were many more types of records (I
feel I'm deserving a good "YAGNI" here ;-))
I suspect that you look at something like the following two snippets and see 'duplicate' code (pseudo representation of what you are doing.)
         fieldx1 = parse.parseInt(5)
         parse.skip(3)
         fieldx2 = parse.parseInt(7)
         fieldy1 = parse.parseInt(5)
         parse.skip(3)
         fieldy2 = parse.parseInt(4)
         fieldy3 = parse.parseInt(7)
There are similarities in the above but they are not duplicate.
And you could create a routine that handles both but there are so many dissimilarities between them that the duplicate part would become obscure by the part that isn't similar.
roughly, when I'm looking at the parser module of the
project and the classes involved, I can't help the
feeling there's something wrong :
It isn't quick and easy but learning how to generate code makes the production of such code a lot easier.
Instead of writing code you would produce something like this....
       fieldx1 int 5
       skip 3
       fieldx2 int 7
This would then be fed to a generator which then produces the appropriate code.
Various ways exist to do this...
- JavaCC
- XDoclet
- roll your own
And there are probably others (pretty sure there are alternatives to JavaCC.)
However whether this "helps" depends on what is being parsed. (Although for myself I find it more interesting to generate code even if it takes me as long to do it as if I did it manually.)
This couldn't possibly scale if there were many more types of recordsDon't strive to build code that meets all possibilities. Strive to build code that meets the current requirements and which meets the known future possibilities. (Known is a tentative term and isn't limited to just requirements in hand but can be based on a comment from someone that suggests something like this might be needed in the future.)
Having built systems that parse various structures I find it unlikely that additions in the future are going to easily fit a current model. Many times they are completely different.