Mail corruption after 10.5 - 10.6 script migration: fix?

Question

After moving 153,000 (!!) emails from our old 10.5 (cyrus) server to a new 10.6 (dovecot) server using the migratemaildata.pl script, I was surprised to find a huge number of error messages in the mailaccess logs to the effect of "Corrupted index cache file /var/spool/imap/dovecot/mail/\[GUID\]/dovecot.index.cache: Corrupted physical size for uid=\[xxx\]: 13975 != 13678". After some investigation I discovered:
- the message was a result of the "S=" flag containing a file size that did not match the actual file size;
- the reason that the actual file size was different was because garbage had been added to the end of the mail file. This garbage contained several lines beginning with various control characters (usually control-M) followed by text repeated from the last several lines of the original message.
- removing this garbage manually from the end of the message resulted in the file size being reduced to match the S= flag in the message filename.
Has anyone run into this? Any clues as to a solution? I don't relish writing a script to process 153,000 emails and examine each one to remove this garbage, but it may be my only solution.
Message was edited by: Robb Allan

Answer 1

Oh, the mailboxes made it, mail is started and running. That's why the error messages are filling the mailaccess.log file. Dovecot's documentation specifically says that it does not alter the "S=xxx" flag in a message's filename, so it's not healing the erroneous message size. More importantly, dovecot absolutely doesn't fix the garbage inserted at the end of each mail message during the migration. That looks like it may be from a bug in cvt_cyrusdb, but that's a binary so I can't tell.
I've ended up writing a quick and dirty shell script to strip the garbage out, which is not especially efficient but does the job. But I am uncomfortable about running it wholesale across my entire mail store. Yikes.
I've tried re-migrating individual mailboxes by account, and the garbage problem occurs each time, so that's not a fix. Here's an example (note the repetition of the last few lines, the first one preceded with a null (^@) and all of them ending with a CR (^M):
Here is a link that you should save for managing your account on-line:
https://www=2EPODS=2Ecom/orderapp/login=2Easpx=3Fcid=3D100160114
This e-mail has been automatically generated=2E Please do not reply to it=2E=
(message 5456165)
^@d=2E Please do not reply to it=2E=^M
(message 5456165)^M
^M
^M
^M
"