Text parsing tool - need suggestion

Question

I need to do the following on a text file, tab separated.
The lines I'm interested in begin all with 4 digits; other lines must remain untouched.
The structure of a line is:
4-digit code
start
end
desciptive string
If the "end" of line X is equal to the "start" of line X+1, the end of the first line must be replaced by (for instance) --- ; I need to compare line 1 and 2, then line 2 and 3, then line 3 and 4, and so on. If a line doesn't begin with 4 digits, it must be printed literally and the comparison must be done with the following line.
Something like:
line 1 begins with 4 digit, so compare it with the following line
line 2 begins with text: print it and skip to the following for the comparison with line 1
line 3 begins with 4 digit: use it for the comparison with line 1 and then compare it with the following line
line 4 begins with 4 digit: use it for the comparison with the previous line and then compare it with the following line
### example input ###
Some garbage text
0123 001 003 dfghdfhg
4569 002 003 fdsygiusdg
Some other garbage text
5001 003 005 fguoiauhg
4786 005 007 fdhgsdhg
5641 007 008 kgjhlks
4455 009 010 dlkhsjsdf
### example desidered output ###
Some garbage text
0123 001 003 dfghdfhg
4569 002 --- fdsygiusdg
Some other garbage text
5001 003 --- fguoiauhg
4786 005 --- fdhgsdhg
5641 007 008 kgjhlks
4455 009 010 dlkhsjsdf
Can someone suggest the right tool for this and give some hints?

Answer 1

I've added my own comments to the code:
#!/usr/bin/perl
use strict;
use warnings;
# initialize an array to hold the matches from a line that fits our regular expression
# we initialize it here to make it global so we can use it in the loop
my @last_line = ();
# the stack will temporarily hold all the lines that don't match the regular expression
# we initialize it here too so that we can use it in the loop
my $stack = '';
# here's the main loop... it gets run once for each line of input
# the "my $line = <STDIN>" sets "$line" to the value of the next line on STDIN
# the "defined()" function is just to make sure that we get all of the input because
# perl can interpret some strings and digits as "False"
# without the "defined()" part, the actual value of each line would be used
while (defined(my $line = <STDIN>))
# first we need to decide how to print the lines
# the tricky part is that before we print one line,
# we need to check the NEXT line to see if it's "start"
# is this line's "end"
# we also need to keep the junk text in the right order,
# which means that we can't print the last line or any of
# the junk lines between it and the next matching line
# until we've checked it so we dump the junk lines on
# a stack that we can print after we've printed the last
# matching line
# here we check for lines that begin with 4 digits,
# followed by spaces, digits, spaces, more digits,
# then whatever's left on the line
# we capture the first 4 digits and the following spaces in first group
# the "start" in the second group
# the spaces between the "start" and the "end" in the third group
# the "end" in the fourth group
# and then the rest of the line in the fifth group
# the "m" at the beginning is not necessary as it defaults
# to matching, but I prefer to be explicit when sharing code
# to make it clearer... you could skip it
if ($line =~ m/^(\d{4}\s+)(\d+)(\s+)(\d+)(.+\n)$/)
# not we assign the matches to an array
# remember that arrays are 0-indexed, so
# group 1 will be at [0], etc
my (@current_line) = ($1,$2,$3,$4,$5);
# check if the "start" of the current line
# (group 2, index 1) matches the "end" of the
# last line (group 4, index 3)
if ($last_line[3] eq $current_line[1])
# if it does, replace each character in
# the "end" of the last line with "-"
$last_line[3] =~ s/./-/g;
# print the last line by joining the 5 groups
# together in a string without spaces then
# print all the junk lines that came after it
# before the current matching line
print join ('',@last_line), $stack;
# make the current line the "last line" for
# the comparison with the next matching line
@last_line=@current_line;
# clear the stack because we've printed those
# lines that were in it already
$stack = '';
# otherwise, the line didn't match our regular express
else
# so drop it on the stack for later
# if we printed it now, it would be printed before
# the last matching line
$stack .= $line;
# now we've run through all the input but we haven't
# printed the last matching line because we always waited
# for the next matching line to compare its "end" with the
# next "start"
# now there's nothing more to compare so we can print
# the last line and the stack
print join ('',@last_line), $stack;
Let's look at a modified version of your example input (with line numbers) and what happens when the script is run
1 Some garbage text
2 0123 001 003 dfghdfhg
3 4569 002 003 fdsygiusdg
4 Some other garbage text
5 Even more garbage text
6 5001 003 005 fguoiauhg
7 4786 005 007 fdhgsdhg
8 5641 007 008 kgjhlks
9 4455 009 010 dlkhsjsdf
@last_line is initalized (empty) and so is $stack
line 1 doesn't match the regex so it get's dropped on the stack
line 2 matches the regex
line 2's "start" (001) doesn't match @last_line's "end" (which is empty) so no changes are made to @last_line's "end"
print @last_line (""), followed by the stack (line 1) and clear the stack
@last_line = line 2
line 3 matches the regex
line 3's "start" (002) doesn't match @last_line's "end" (003) so no changes are made
print @last_line (line 2) followed by the stack ("")
@last_line = line 3
line 4 doesn't match the regex so it's dropped on the stack
same for line 5 so the stack is now line 4 + line 5
line 6 matches the regex
line 6's "start" (003) matches @last_line's (line 3's) "end" (003) so replace "003" with "---"
print @last_line (line 3) followed by the stack (line 4 + line 5)
etc
Without the stack, lines 4 and 5 would have been printed before line 3 because we were holding on to it for comparison with line 6.
(I sense another one of those "you have too much time on your hands" comments in this thread... I can't help it though, I like to help, plus it's good exercise to explain how you've programmed something and why)
Last edited by Xyne (2009-03-10 20:34:35)