Pdftotext extracting from image files mystery

Question

hello all, just had a bit of a shock when I ran pdftotext (accidentally) on an unocr'd pdf file, and it extracted all the text. Running pdftohtml (as I'd intended) produced the expected output - i.e. png dumps. Curious, I then tried running it on a bunch of other downloaded files, and with the exception of one, pdftotext extracted the text from ALL of them. The mystery then is that it isnt using any ocr (no tesseract dependency, plus it's way too quick), so clearly it must be pulling it directly from the files. But if the text really is there in the original, then presumably the people who scanned them didn't know, else they'd have left it there. The only thing I can think is that some common piece of pdf software (probably acrobat) has ocr built in, so it's scanning them automatically and then encoding the hidden text in the pdf.
take for example (legal):
http://www.cd3wd.com/cd3wd_40/JF/JF_VE/SMALL/27-714.pdf
open up in a normal pdfreader - very clearly a (poorly) scanned document. Now run 'pdftotext -layout'  on it and you get a pretty impressive text file, considering the source. Sure, some of the formatting is messed up, but I'm sure anyone with sed knowledge could quickly sort most of that out. Besides, that's one of the worst documents -  on most others it was perfect, maintaining columns (even tesseract can't do that) and everything.
It makes for a fantastic command line pdf reader that works on almost all the files I've thrown at it, and since pdf's are almost the only reason I ever have to load up X, consider me tickled pink.
Just thought I'd mention it here, as a google search brings up nothing on the issue. hope someone finds it useful

Answer 1

Yes, Adobe Acrobat has a feature that allows you OCR the text and insert the text as a separate layer behind the image. Such PDFs are called "Searchable image PDFs". There are no doubt some other commercial software options that can do this too. The copier/scanner/multifunction device in my office actually does this automatically when you scan in a text PDF.
Theoretically, it is even possible to create PDFs like this using free software on linux (a combination of ExactImage and Cuneiform; or WatchOCR).
See, e.g., this blog post:
Searchable PDFs with linux
and this slashdot story:
Open Source OCR that makes Searchable PDFs
My own experiments with trying to get something like this to work have not been very successful--at least the quality of the result is nowhere near what I'm getting with our work printer.
If you like pdftohtml, you should also know about pdfreflow, which takes the XML output of pdftohtml with the -xml flag and creates an html file that can be reflowed (I.e., is smart about paragraph breaks, removes page numbers and recombines words broken through hyphenation and so on).
pdfreflow
Last edited by frabjous (2010-10-15 13:03:00)