Re: mhfixmsg character set conversion

[Steven Winikoff]

>Steven wrote:
>
>> That's probably a helpful thing to do, but the question I was wondering
>> about wasn't why the UTF-to-UTF conversion was reported, but rather why
>> the iso-8859-1-to-UTF conversion wasn't reported.
>
>Ok, commit 41ce4490ac5d might fix the problem for you.

Thank you!  I'll check it out tomorrow and see what happens.  Do you
think the patch will apply to 1.7.1, or will I need to install the
latest version from the repository?


>The cause was mismatch between the character set of content generated by
>the external program (lynx, in this case) and mhfixmsg -textcharset UTF-8.
>mhfixmsg wasn't capturing the charset of the generated output, assuming it
>was unchanged.  It then converted it again.

I'm afraid I have to admit I'm not entirely clear on how lynx is even
involved.  I know I have these entry in my system-wide mhn.defaults file:

   $ grep lynx /local/pkg/nmh/root-nmh-1.7.1/etc/mhn.defaults
   mhbuild-convert-text/html: charset="%{charset}"; /sbin/lynx -child -dump 
-force_html ${charset:+--assume_charset} ${charset:+"$charset"} %F | sed 
's/^\(.\)/> \1/; s/^$/>/;' | par 64
   mhfixmsg-format-text/html: charset="%{charset}"; /sbin/lynx -child -dump 
-force_html ${charset:+--assume_charset} ${charset:+"$charset"} %F | expand | 
sed -e 's/^   //' -e 's/  *$//'
   mhshow-show-text/html: charset="%{charset}"; %l/sbin/lynx -child -dump 
-force-html ${charset:+--assume_charset} ${charset:+"$charset"} %F

...and the second of these certainly looks relevant, but:

   - While testing on Friday, I emptied that file out completely and still
     observed the same behaviour.

   - In your message from 12:35 today in the "crufty mhn.default.sh stuff"
     thread, you wrote:

        > There is a way.  etcpath() looks for mhn.defaults in this order:
        >  * 3) Next, check in nmh Mail directory.
        >  * 4) Next, check in nmh `etc' directory.
        >
        > So if the user puts an mhn.defaults in their Mail directory, then
        > only it will be read.  They'd have to copy any entries that they
        > do want from /etc/nmh/mhn.defaults to their own.

     I do have an mhn.defaults file in my Mail directory, with (only) these
     entries in it:

        mhshow-show-application/pdf: %pmime_helper %F %s "%{name}"
        mhshow-show-application:     %pmime_helper %F %s "%{name}"
        mhshow-show-audio:           %pmime_helper %F %s "%{name}"
        mhshow-show-video:           %pmime_helper %F %s "%{name}"
        mhshow-show-image:           %pmime_helper %F %s "%{name}"
        mhshow-show-text/richtext:   %pmime_helper %F %s "%{name}"

...so while I believe that lynx is involved, I don't know where that
involvement is coming from.

While I'm replying to you anyway, I realize I forgot to reply to your
question from yesterday morning.  You asked:

   Have you tried the -decodeheaderfieldbodies switch to mhfixmsg?

I haven't, mainly because I didn't know that switch existed.  I don't know
what it does (other than what I can infer from the name, of course), and I
can't find any mention of it in the man page for mhfixmsg, or anywhere in
the source code for version 1.7.1.  Was this switch added after 1.7.1 was
released?
   

>The fix is for mhfixmsg to detect the charset of the content,
>using file --brief --mime-encoding if it can.  If it can't, it
>falls back to the -textcharset value.  If that wasn't used, it
>gets it from the locale and advises the user.

That sounds reasonable to me.


>I'm not completely sure that this will fix your problem because
>it's aimed at added text/plain parts.  But with -noreplacetextplain
>I think that's the path to your issue.

Please advise on the easiest way to try it (between applying 41ce4490ac5d to
1.7.1, or just downloading and building the current version of the master
branch), and I'll do so tomorrow (I'm unable to do it before then due to
a prior commitment).

     - Steven
-- 
___________________________________________________________________________
Steven Winikoff      | "The thing is, I mean, there's times when
Montreal, QC, Canada |  you look at the universe and you think,
smw@smwonline.ca     |  'What about me?' and you can just hear
http://smwonline.ca  |  the universe replying, 'Well, what about
                     |  you?'"
                     |         - Terry Pratchett (Thief of Time)