Sun*MS body rewriting -- relay implementation per RFCs

Question

I read and understand RFC 2821 to mean that relays should generally try to leave messages alone. To quote just one section of the RFC...
   A 'relay' SMTP system (usually referred to just as a 'relay') receives
   mail from an SMTP client and transmits it, without modification to
   the message data other than adding trace information, to another SMTP
   server fo r further relaying or fo r delivery.There are 76 character length references in RFC2045, but that refers to the encoded body lines and not headers/boundaries nor SMTP.
S1MS rewrites From: lines, message boundaries?, and wraps longish lines (76?). Most of the time, this rewriting is unnecessary and IMHO violates RFC2821 (and RFC2045 is not relevant, reread) and does nothing except involve more work and no doubt reduce throughput.
I can understand the option to rewrite long lines if they are exceptionally long (1000 characters per RFC), but S1MS seems to break all lines longer than 76 characters. I can understand rewriting local From: addresses if needed, but it seems to rewrites all From: lines remote or otherwise. I've also noticed different S1MS sometimes also rewrite Content boundaries as "Boundary_(ID_blah)". I'm wondering what exactly is rewritten, controlled by what options, and under what circumstances? Why the extra work? Because any subsequent smtp server will still need to parse everything anyways, I don't appreciate all the work and don't understand why it is done.
The crux of this was a recent incident where an 83 character attachment filename was wrapped across two lines causing problems with third party software (it has been fixed, but we both looked at RFCs and I felt S*MS was in the wrong on two counts).
Despite this particular incident, I've long had other reservations (delays, truncated messages) along with the how/why modifications. The end result obfuscates spam with each line that is modified (or corrected, dates inserted, etc).
I looked through this message base looking for other posts perhaps related to this and nothing stuck out. It seems from the messages I saw that S*MS only bothers with the exceptionally long 1000 character lines so I'm now wondering what is exactly is responsible for changing the Content Boundary and wrongly wrapping all lines longer than 76 lines which I see time and time again on many different S*MS systems.

Answer 1

Authoratative answers:
The customer is correct that the MTA will under a number of circumstances
rewrite boundary markers, force wrapping of "long" lines, etc. In general, we
process all outermost header lines coming through the MTA, and may well process
inner header lines as well.
We are fully aware of the RFC 2821 wording regarding relay SMTP machines
"leaving messages alone". The problem is that in our experience, that does
not in fact work well. There are NUMEROUS, NUMEROUS message problems that
our MTA "fixes up", without customers ever having been aware of there having
been a problem in the first place. The sorts of problem we used to encounter
tended to be with messages gatewayed in from non-Internet environments; I consider
nowadays the problems tend to be more with messages submitted from "light weight",
and unfortunately sloppy, e-mail clients. (The MTA does in fact have a configurable
approach for "leave relayed messages entirely alone". We've left it undocumented,
and so far continue to leave it undocumented, and beyond strongly discouraging use
of it don't even talk about it with customers, because in real life, it tends to
not work well.)
Customers, being unaware of how much is getting "fixed up", tend to think that
things would be "better" if we'd "just leave the messages alone". Or rather,
what customers really want is "fix up (a), (b), and (c) problems, but leave
(d) alone, except when it's coming from (e) or going to (f), or except on
every other Thursday when the moon is full..." blah, blah. In other words,
customers want to have their cake and eat it too, when it comes to message
"fix up". Our overwhelming experience is that our fix up is overall
beneficial -- and that the supposed "downsides" of message fixup are mostly
bogus: that inherent processing that customers do in fact want gets you so
far down the road of message processing that the supposed "extra" overhead
is not in fact "extra" because it's already had to be incurred, and that
most of the supposed "problems" of the fixup are due to broken other
e-mail software that's not correctly handling the correct message forms
(and that for every one case of broken software that's stumbling on the
corrected message form, there were NUMEROUS cases of which the customer
was blissfully unaware where our correction of the message form improved
interoperability). Also, the MTA has numerous configuration "knobs" that
can affect exactly what we emit, so even when the real problem is a bug
(lack of standards-compliance) on the remote side, it's often possible
to configure something on the MTA side to emit a form that the broken
remote side will be able to handle.
Now it is true that we try to avoid any more processing than necessary
-- but the fact is that there are extremely many cases, triggered by a
wide variety of original message factors and configuration choices, where
processing is necessary.
As regards line folding: Please refer the customer to section 2.1.1
Line Length Limits, of RFC 2822, in particular the second sentence:
Each line of characters MUST be no more than 998 characters and
SHOULD be no more than 78 characters, excluding the CRLF.
Also see 3.5 Overall message syntax, in particular the second sentence:
Lines in a message MUST be a maximum of 998 characters
excluding the CRLF, but it is RECOMMENDED that lines be limited to 78
characters excluding the CRLF.
The MTA's default is therefore to fold header lines at 80 characters,
though this is configurable via the headerlinelength channel keyword
(or individually for individual header lines via header trimming
and the LINELENGTH header trimming option).
Now as regards clients not being able to correctly handle the
correctly wrapped header lines -- well, that I can certainly
believe since correct implementation of header line unfolding
rules (especially when combined with use of RFC 2047 encoded-words
in header lines) seems to be an area where client implementations are
WOEFULLY poor. It is really very irritating: the standards exist
exactly so that data can be preserved, and the clients just get this
area so wrong.
(And by the way, it is legal to wrap within quoted-strings.)
As for message bodies line wrapping -- that is controlled by factors
such as whether the contents had to be encoded with an encoding (such
as QUOTED-PRINTABLE or BASE64) that requires also keeping the lines
short, and note that such encoding can be triggered by use of the
linelength channel keyword. But lacking a need, body lines can
normally be up to the 998 legal limit and the MTA won't be folding
them. If the customer thinks they are seeing something different,
then I would want to see lots of details.
As for what can trigger the MTA to use its own boundary markers:
that would be anything that triggers MIME processing, including
(a) routing through the conversion channel, (b) use of the "thurman"
keyword, (c) application of a CHARSET-CONVERSION, (d) incoming
raw (illegal) 8 bit data, (e) the "inner" keyword, etc., etc.
They're all useful, and typically needed by customers at-some
-time-or-another, for-some-purpose-or-another, sorts of features,
that inherently require that the MTA do MIME processing, at which
point we redo the boundary markers.
Oh, other comments: Yes, the MTA does reorder all the headers.
We output them in an order that the MTA considers default, though
that order is extremely configurable: see header trimming and
the PRECEDENCE header trimming option. And yes, we change
the case on header line names; although the names are per the
standards case-insensitive, we have encountered quite a few cases
of broken software that (incorrectly) expects a particular case,
so what we emit tends towards the "seems to be most interoperable"
case. And that also can be configured, via header trimming and
the RELABEL header trimming option.
Anyhow, Jay, it's perhaps unfair to just dump this information on
you, but I'm not certain I have the patience at the moment
to go through all the questions the customer raised one by one and
point out how each one is either: (a) possibly interesting (e.g.,
supposedly "lost" e-mail) though we'd have to get lots of details
(since as I think you wrote, it's pretty dubious that the messages
were actually lost by the MTA), or (b) the customer is wrong in
their understanding of the RFCs, or (c) the customer is dealing
with standards-incompliant other software, or (d) explain
exactly why the MTA does what it does, or etc...Each question
probably ought to be dealt with as a separate question. The
customer appears to have lumped a bunch of (from my point of view)
separate issues together and decided that "MTA processing of messages
is bad -- the MTA should be leaving the messages alone". I completely
disagree with their conclusion, although many of the details that the
customer has observed are correct -- but it's probably going to take
discussion of each issue individually to enlighten the customer.
and
(Basic complaint) .. . ... almost entirely at odds with reality. For starters, he begins by assuming
we operate as a relay. We don't and have never claimed to. As such, any
restrictions that apply to "relays" don't apply to us. So the argument that
we're incompliant with the relevant RFCs is completely specious.
The role of iMS fills is closest to what the RFCs call a "gateway", although in
practice few sites these days take advantage of the full gateway functionality
we provide. But by the same token, most sites find at least some of the various
operations we perform on messages useful and don't disable them (although most
of them can in fact be disabled). So in practice most sites operate something
that falls between a "relay" and "gateway". (There's also the issue of the
specific operations a submission server is supposed to perform but I don't want
to get into that here.)
He goes on to argue that the various processing steps we perform on messages
are unnecessary and/or result in unacceptable loss of performance. Wrong on
both counts. Everything we do is done because there's been a demonstrable need
for it. We don't spend time implementing stuff nobody wants. Indeed, a lot of
people find many of the operations we perform to be essential and purchase our
product because it has rewriting and message alteration capabilities lacking in
other messaging servers. As for performance, we have one of the fastest MTAs
around. I'd say that pretty much proves the performance impact is negligable.
A specific point of complaint is that we change boundary markers. The reason for
this is simple: When performing MIME processing the only way to do it correctly
in a single pass is to randomize the boundary markers. (I'll leave it as an
exercise to the reader to figure out why if you're so inclined.) In any case,
I see no evidence that there is anything wrong in our boundary handling.
He then starts complaining about header wrapping and reorderng. Header
wrapping and rewrapping is (a) Entirely legal and (b) Wrapping at 80 columns
or less is strongly encouraged by the relevant RFCs.
This brings me to the one point he makes that I think is somewhat valid: Header
reordering. We're RFC compliant here as well: If you read the relevant RFCs
you'll find that reordering of trace (Received:) fields is forbidden and
certain headers are required to be prepended, but aside from that pretty much
anything goes. We conform to all this but I don't think the RFCs are stringent
enough in this area. Accordingly, changes have been made for JES 5 that will
result in our preserving header order by default. (Obviously this all goes
out the window if you explicit request header reordering of some sort.)
One final point. Much of the behavior he doesn't like can easily be disabled in
iMS. If all this stuff is really so abhorrent, you'd think we'd see the
majority of sites turning it off. But what we see in practice is exactly the
opposite: Not only do sites not disable this stuff, they spend a considerable
amount of time yoeling and howling about we don't do enough of these things.
The addition of disclaimers is an excellent example of this: We don't think
this is a good idea and try to discourage it, but our customers seem not to
agree...