[solved]how to extract recent log entries from a file (based on time)?

Question

I have a daily log file with hundreds of thousands of entries in the following format. 
field1,field2,field3,field4,field5,field6,field7,field8,field9,20110516192001.100
field1,field2,field3,field4,field5,field6,field7,field8,field9,20110516192002.200
field1,field2,field3,field4,field5,field6,field7,field8,field9,20110516192003.300
field1,field2,field3,field4,field5,field6,field7,field8,field9,20110516192004.400
field1,field2,field3,field4,field5,field6,field7,field8,field9,20110516192005.500
It's always in the same format and the 10th field is always the timestamp (YYYYMMDDHHMMSS.MS)
Since the file rotates daily, the 10th field will always be 20110516xxxxxx.xxx for today and will be 20110517xxxxxx.xxx tomorrow
What I want to do is only look at entries that have been written in the last 30 minutes.
At a high level, here's my plan
1) Get the date/time from 30 minutes ago... write it to a variable
2) Iterate through the file line by line comparing the 10th field to the variable, if it's larger write the line to a tmp file
3) Use tmp file for my analysis
This seems incredibly inefficient to me...  what would be a more graceful way to do it?  I have regular solaris tools at my disposal (plus python)
Thanks
Last edited by oliver (2011-05-17 12:41:43)

Answer 1

The algorith you describe really is a viable approach.  Since this is a log file, each line should have a time stamp later than all lines that preceed it in the file. A more efficient algoithm could do a binary search through the file for the time stamp you are interested in.  This would be easy enough in to do in C or python, but your algoithm could be fast enough. If this is the case, you could try the following quick & dirty bash script.
#!/bin/bash
seconds() {
secs=$(($1 % 100))
mins=$(($1 / 100 % 100))
hrs=$(($1 / 10000 % 100))
days=$(($1 / 1000000 % 100))
month=$(($1 / 100000000 % 100))
year=$(($1 / 10000000000))
(LC_TIME=C date +%s -d $(printf "%d-%02d-%02d %2d:%02d:%02d" $year $month $days $hrs $mins $secs))
found=0
now=$(date +%s)
while read line
do
if [ "$found" -eq "0" ]
then
ts=${line##*,}
ts=$(seconds ${ts%.*})
diff=$(( ($now - $ts)/60 ))
[[ $diff -lt "30" ]] && found=1
fi
[[ $found -ne 0 ]] && echo "$line"
done < $1
It will write (to stdout) all lines following the first line that has been time stamped within the last 30 minutes (ignoring milliseconds). You could redirect the output of this script to a file of your choice for analysis as follows:
$ ./script logfile > tmp
Last edited by rockin turtle (2011-05-17 06:58:41)