Re: [Nmh-workers] Handling non-ASCII

[Ken Hornstein]

>If the only source of these malformed messages is spam then we should
>just refuse to display them.

I recently ran into this when working on replyfilter (the core issue was that
par mangles UTF-8 and replyfilter would end up with invalid UTF-8 sequences).

Refusing to display the message is rather unfriendly, and also could
end up messing up utilities that sit on top of nmh (exmh, mh-e,
mh-v).  Continuing on but replacing the bad sequences with something
obvious (in this case it was U+FFFD) seems to be the best solution.
It turns out perl does print a warning in this case, but because
replyfilter is being invoked right before the editor it's very easy to
miss.

Whatever is done I think it will have to be small; I'm reluctant
to do more until after the Great MIME Rewrite (I like Earl's suggestion
about a selectable default character set, but I think that's going to have
to wait).

--Ken