Re: [Nmh-workers] Fixing decoding of quoted messages in replies

[Peter Maydell]

Peter Maydell wrote:
>markus schnalke wrote:
>>during the next weeks I intend to fix the annoying problem of still
>>encoded message text in quoted replies.

>I completely agree. I'll try to make some time to take another
>look at this some time this week. (I'm at the Ubuntu Developer
>Summit so I'm in a free software kind of mood anyway :-))

Having thought about this a little more, here are six
proposed changes to nmh which I think will together make
dealing with MIME rather nicer. I have attempted to
stick to "what are the changes we want to make to UI,
config and to individual nmh binaries?" -- we should
agree on these before we start getting into the technical
details of how to change things. So:


(1) things which currently claim to mean "columns" or "characters"
but actually count bytes (eg scan -width, the "%4(from)" syntax
in format fields) should count columns. [Aside: 'columns' is
usually the same as 'characters' except that some Chinese/Japanese
characters are two columns wide on screen. Since the affected
switches/syntax are generally for making lined up displays on
screen we should care about the display width of the text.]

(2) the default setup should always MIME-decode headers. Either:
 (a) we need to add decode() calls to all the default format
     templates where they aren't already present
 (b) have a general config switch for "default to raw vs
     default to decode", set it to 'decode' and remove any
     existing decode() calls from format templates

(3) scan's "body" component escape should give you decoded
text, by which I mean:
 * select the first text/plain bit of multipart MIME
 * decode quoted-printable and base-64
 * convert to the character encoding of the current locale
[we can probably steal a lot of code from mhshow for this]
We can add a 'raw' option to this for the rare case where
you really did want the undecoded text.

(4) mhl's body component should decode as per (3).
This is what will allow 'repl' to put you in an editor
with the decoded text rather than the raw MIME.

(5) (Contrary to what I first suggested) I think that the
conversion of outbound messages to a suitable charset and 
adding of MIME headers should be done in send. We may be
able to use the existing -mime option to control this, or
we could play safe and add a new switch.

Rationale: (a) I want MIME-ing to be automatic but I don't
want to auto invoke
mhbuild for all users because it does a lot of interpretation
of the body text which would be very user-unfriendly to
have happen without you explicitly asking for it. 
(b) send / post is the code which does things like looking
up addresses in the aliases file, so we can't mime encode
headers until it's done. And it already is the code which
wraps headers to sensible lengths, ensures mandatory headers
are present and generally makes the mail be a valid format,
so it's not out of place to implement it there.

So send should: 
 * check for non-ASCII characters
 * when present: pick an encoding (first usable one from
    a user-configurable list like us-ascii,iso-8859-1,utf-8)
 * encode body and headers as per MIME, adding MIME headers
   to match
[We need to take care not to trip mhbuild's earlier work
up, but in theory mhbuild should already have got rid
of all the non-ascii characters, so if send just passes
through anything that's already a valid MIME message then
mhbuild and any preexisting script-based hacks by users
should be unaffected.]

(6) Reading MIME email currently works but can feel a bit
clunky. Since the changes in (4) basically mean that
plain old 'show' can cope with at least some MIME
messages, I think we could let the user choose to use
show for everything. We would probably need some new
mhl component file syntax to say "display a summary
of the MIME parts that weren't shown (ie attachments)".


So do people think I've got the changes to the "user
visible" bits of interface/config right?

-- PMM