Mangling of non-ASCII attachment file-names

Question

It is not a 100%-reproducible problem that seems to happen in several cases involving both an OUCS 7u4 and an old JCS 5 server (when exchanging messages between each other or even within a server bypassing any milters for example, or sending to other domains such as google, posting with web or SMTP-Auth).
  The common thing is that the received attachment's headers lose the piece about character encoding of the first line of the filename:
--Boundary_(ID_aNbA9YOhlTUcvRNjPMb27Q)
Content-type: application/vnd.ms-excel;
name="0LPRgNCw0LzQvNGDINC+INC/0L7Rh9GC?=
=?UTF-8?B?0L7QstC+0Lkg0YHQuNGB0YLQtdC80LUg0LIg0KDQvtGB0L/QtdGH0LDRgtC4Lng=?=
=?UTF-8?B?bHM=?="
Content-transfer-encoding: base64
Content-disposition: attachment;
filename="0LPRgNCw0LzQvNGDINC+INC/0L7Rh9GC?= =?UTF-8?B?0L7QstC+0Lkg0YHQuNGB0YLQtdC80LUg0LIg0KDQvtGB0L/QtdGH0LDRgtC4Lng=?=
=?UTF-8?B?bHM=?=";
filename*0*=UTF-8''%D0%9E%D1%82%D0%B2%D0%B5%D1%82%20%D0%BD%D0%B0%20%D1%82;
filename*1*=%D0%B5%D0%BB%D0%B5%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D1%83%20%D0;
filename*2*=%BE%20%D0%BF%D0%BE%D1%87%D1%82%D0%BE%D0%B2%D0%BE%D0%B9%20%D1;
filename*3*=%81%D0%B8%D1%81%D1%82%D0%B5%D0%BC%D0%B5%20%D0%B2%20%D0%A0%D0;
filename*4*=%BE%D1%81%D0%BF%D0%B5%D1%87%D0%B0%D1%82%D0%B8%2E%78%6C%73
Sometimes messages do pass correctly, and in all cases they are saved into "Sent" folder correctly, so it is not a client error for example:
--------------090501050302010603060805
Content-Type: application/vnd.ms-excel;
name="=?UTF-8?B?0J7RgtCy0LXRgiDQvdCwINGC0LXQu9C10LPRgNCw0LzQvNGDINC+INC/0L7Rh9GC?=
=?UTF-8?B?0L7QstC+0Lkg0YHQuNGB0YLQtdC80LUg0LIg0KDQvtGB0L/QtdGH0LDRgtC4Lng=?=
=?UTF-8?B?bHM=?="
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename*0*=UTF-8''%D0%9E%D1%82%D0%B2%D0%B5%D1%82%20%D0%BD%D0%B0%20%D1%82;
filename*1*=%D0%B5%D0%BB%D0%B5%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D1%83%20%D0;
filename*2*=%BE%20%D0%BF%D0%BE%D1%87%D1%82%D0%BE%D0%B2%D0%BE%D0%B9%20%D1;
filename*3*=%81%D0%B8%D1%81%D1%82%D0%B5%D0%BC%D0%B5%20%D0%B2%20%D0%A0%D0;
filename*4*=%BE%D1%81%D0%BF%D0%B5%D1%87%D0%B0%D1%82%D0%B8%2E%78%6C%73
As can be seen in this example, the boundary identifiers have changed (meaning IMHO that the message was processed - taken apart and put together again), and in this example only part of the first line makes it into the received message's filename header.
Another example:
"Sent":
--------------010609000006000200040201
Content-Type: application/pdf;
name="=?UTF-8?B?0YLQtdC70LXQs9GA0LDQvNC80LAg0L4g0YDQsNCx0L7RgtC1INGB0LjRgdGC0LXQvA==?=
=?UTF-8?B?0Ysg0L7QsdC80LXQvdCwINGN0LvQtdC60YLRgC4g0L/QvtGH0YLQvtCy0YvQvNC4?=
=?UTF-8?B?INGB0L7QvtCx0YnQtdC90LjRj9C80Lgg0L7RgiAxNC4xMC4xMy5wZGY=?="
"Received":
--Boundary_(ID_kv6CyB9foC+l8rLg7THT2g)
Content-type: application/pdf;
name="QsdC80LXQvdCwINGN0LvQtdC60YLRgC4g0L/QvtGH0YLQvtCy0YvQvNC4?=
=?UTF-8?B?INGB0L7QvtCx0YnQtdC90LjRj9C80Lgg0L7RgiAxNC4xMC4xMy5wZGY=?="
Since this happens both on a server with a spam/virus filters and on a server without any of those, and it is not 100% reproducible (though frequent) I can guess that sometimes messages pass through channels such as (de-)fragmentation, or Sieve, or something, and this breaks them.
Any ideas? Is this a misconfiguration or a CommSuite bug?
Thanks,
//Jim Klimov

Answer 1

There is no reconstruction from anything involved, although perhaps there should be. The code simply checks the length of the parameter, and if it's bigger than the limit, truncates the string from the beginning. The idea behind truncating from the beginning is that way the file extension may perhaps be preserved.
I should have noted that use of encoded-words in parameters, while fairly common in practice, is nevertheless explicitly forbidden by RFC 2047. So this usage is problematic from the start. The standard way to encode utf-8 in content-type and content-disposition parameters is specified in RFC 2231, and it is what is being used in the content-disposition field. So whatever is producing this field is trying to work with both standards-compliant and standards-incompliant receivers. This sounds great right up to the point where an interaction like this occurs.
Additionally, agents should be keying preferentially off content-disposition information since this usage of a name parameter on a content-type for which it isn't defined is also incompliant.
I don't see any way how the intermittent nature of the problem could have anything to do with the charset used, other than the length of the same information encoded in different charsets can vary considerably. (iso-2022-jp is especially problematic in headers for various reasons I won't bother describing here.)
As for known issues, there certainly aren't any problems handling long parameters in Messaging Server. Outside of the known problem with parameters longer than 128 characters, I don't know the point at which other agents start to malfunction. But you also have to ask yourself: Given the hard limit of 255 octets imposed by many if not most filesystems, is it in way sensible to be sending stuff longer than that around? As such, it's hard to argue that something that exceeds, say, a 1K limit is worth accomodating.