Re: [nmh-workers] Formatting HTML to Text: netrik.

[Ronald F. Guilmette]

In message <address@hidden>, 
Ken Hornstein <address@hidden> wrote:

>>I had just sort-of jury-rigged my own very local and very idiosyncratic
>>mechanism for dealing with the problem/issue some years ago, and I have
>>been just trying to struggle along with it for all this time because
>>I just haven't had the time to find a proper fix... which I've always
>>assumed is burred in the documentation someplace.  But if there really
>>is no good solution in the case of nmh, then I guess that eventually...
>>and proabbly sooner rather than later... I'm going to have to switch
>>mail clients at long last, although I sure will miss some of the nicer
>>nmh features.
>
>Geez, I thought we handled that pretty well.

Apologies if I offended, unintentionally. I should have gone further to
clarify that I'm quite completely sure that all of the issues I've
encountered are due either to my own ignorance, or to the fact that
I had jury-rigged my personal nmh install, long long ago, approximately
in the bronze age, and I've just never had the time to re-do any of
this stuff properly, or even to -learne- the specifics of what I have
been doing wrong, let alone the "new" and proper way of doing things.
(I';ve been using nmh and its predecessor nmh since about 1982, so that
will give you some idea of the fact that I'm set in my ways, more than
a little bit.)

>For base64 emails, we should handle that fine for display, full stop.

Perhaps, but for display, since so much of the email I've received for
low these many years now has been either partially or totally HTMLized
crap, *and* because I have some rather specialized needs, long ago I
invented and implemenmted my own personal quirky little "solution",
which involved running HTMLized MIME parts thrugh the linx browser to
render them as plain text.

I freely admit that it was most probably a *really* bad idea for me to
have strayed so far from the beaten parth, but I do a lot of anti-spam
work, and since the dawn of time this case caused me to insist on always
seeing the FULL HEADERS of each and every email message I read.  (And
I'm sure this was part of what motivated me to go down this route, although
I have paid for it in daily agony ever since.)

So anyway, here's what I have now, and you're all invited to tell me what
obvious garbage it is, and how I should be doing things The Right Way...

For starters, I have in my ~/.mh_profile the following lines:

showproc: mtext
showmimeproc: mtext

On my system "mtext" is a trivial Perl script I wrote that simply
figures out the full pathname of the "current" NMH message and then
passes that to a different small Perl script I wrote (called "textualize")
that does most of the heavy lifting.  Here are those two short Perl scripts:

    https://pastebin.com/raw/XgqbekaZ
    https://pastebin.com/raw/EMSgdXaD

Together, these two work well enough to show me what I want to see when I do
"show", but the whole mess breaks down badly (and is very neraly entirely
useless) when I try to "repl" any message.

>If you are building from source, _if_ you have one of the common text-based
>HTML browsers available (I use w3m, but we also support lynx and elinks),
>then show(1) does the right thing, and pretty reasonably I would say.

OK, well, but as noted above, I've screwed things up, badly, so what should
I *actually* have in my .mh_profile file for the definitions of showproc
and showmimeproc?  what are the current (proper) defaults for those?  And
do the defaults have options to show me FULL headers for each message?
If not, how do I make that happen?

>If you are installing from an OS package ... well, what happens there varies.
>At least for MacOS X, it will use w3m (this all is configured at install
>time).  But if you've put your own entry in for mhshow-show-text/html, then
>you wouldn't see that.  We've done that since 1.6 (released in 2014).

Here is my current, much fiddled ~/.mh_profile.  Tell me -everything- I
should fix and I'll fix it all:

   https://pastebin.com/raw/PEMJ06VH

(Yes, this .mh_profile has been reused and recycled and fiddled and ignored
ad infinitum from literally EONS ago.  In you see anything in there that
hasn't even been supported for 10+ years, don't be surprised.)

>As for repl(1), we've shipped with replyfilter in $(DOCDIR)/contrib
>since 1.5 (released in 2012), and I use it on every message.  I won't
>say it's perfect, but 95% of the time it does the right thing for me,
>and handles HTML and base64 email just fine.

I've just upgraded everything, so I'm on a fresh new FreeBSD 12.0 system
with a fresh and shiny new nmh-1.7.1 pre-built package installed, so I do
believe that I have everything needed in order to get rolling.

Checking now, i see that I *do* have the following:

   /usr/local/share/doc/nmh/contrib/replyfilter

>Unfortunately without a
>complete rewrite of mhl it was hard to put it in there transparently, so
>it does require extra configuration (see the comments at the top of
>replyfilter).  We talked about this a LOT on the mailing list during it's
>development, and here's the snippet from the NEWS file:
>
>- Preliminary support for improved MIME handling when replying to messages!
>  Yes, a long requested feature has a solution.  A perl script
>  called replyfilter is available; it is designed to act as a mhl
>  external filter to process MIME messages in a more logical way.

So basically, you guys hacked together a solution, sort of like I did!
That's OK.  I'm 100% sure that your solution is WAY better than my crappy
and ham-fisted one.

(I'm just real glad that you folks did your's in Perl, cuz that happens to
be one of the few languages that I actually speak somewhat. It would all
have been utterly Greek to me if it was in Python or Ruby or Go or something.)

I'm looking at the comments at the top of the replyfilter Perl script.
I'll try to just do everything it tells me to do.  Wish me luck!  If you
never hear from me again (because I have screwed up and totally broken
my email entirely) please tell my family that I said I love them. :-)

>  It is available in $(srcdir)/docs/contrib/replyfilter or is
>  typically installed as $(prefix)/share/doc/nmh/contrib/replyfilter.

I guess that I should make an exexcutable copy of that in /usr/local/bin
alongside show and repl and the rest of the gang, yes?  It doesn't seem
quite right to just be executing it from a /docs/ directory.

>  See the comments at the top of replyfilter for usage information;
>  it will likely require some adjustment for your site.

I don't follow.  What will need adjusting, exactly?

>  replyfilter
>  requires the MIME-Tools and MailTools perl modules.

In FreeBSD parlance, that would be "p5-MIME-Tools" and "p5-Mail-Tools".
I have just successfully installed both, so I'm good.

>So, I hope it doesn't seem like I'm crapping on you, because you're
>not the first person who is unaware of replyfilter (you can search the
>mailing list archives for people who have asked this question before,
>and I can write this email in my sleep now).

Sir, you mistake me for someone who gives a crap if someone craps on me in
a mailing list!  I don't.  I have thick skin and I'm just damned greatful
for the help.  I came here perfectly prepared to admit my abundant ignorance,
which I believe I have done.  If not, allow me to do so now.  I'm ignorant.
I've always felt that ignornace isn't something to be at all ashamed of.
There's a BIG difference between ignorance, which can be easily cured, and
stupidity, which can't be.

>I thought we did a pretty
>good job of putting this information out there, but clearly in the 7
>years since 1.5 has come out everyone hasn't gotten the word yet.

Um.... I don't suppose it will help my case any of I were to say something
like "I was busy, and had a LOT of stuff to do", would it?  (But that's
actually true, and I hope that I've done some Good Things for the Internet
and for humanity in that time.)

>So my
>question to YOU is ... what could we have done better to let you know?

See above.  Not your fault.  Nobody's fault in fact.  I am just Rip Van
Winnkle with respect to {N}MH.  I've continued to use it, year after year
after year, even while never taking the time to catch up on recent
developments... even ones that I personally could greatly benefit from.

If you do figure out a way... or if you have figured out a way... to pack
25 hours into every day and if you then neglect to tell me about it, THEN
I will have something to blame you for, but not until then.  Till then,
the fault is all on my end.

>(Yes, in a perfect world 'repl' would just do the right thing automatically
>without any extra configuration, but that's a heavy lift).

I could talk a LOT about this, but probably shouldn't here.

I just "upgraded" two of my systems here.  The wall-clock time needed in
each case for me to get -everything- that I previously had running back
and perfectly configured and running again is shown below:

     Ubuntu  176.04 -> 18.04  (Time: approx 1.5 days.)
     FreeBSD 9.1 -> 12.0      (Time: approx 2.5 WEEKS)

The disparity here has to do with several factors:

    1) MUCH shorer "upgrade gap" in the case of Ubuntu (many YEARS different)

    2) In the FreeBSD case, that system is *both* my main desktop *and*
       the central file & mail server for my whole home network, so there
       were a lot more little things that had to be made to work... again.

    3) Ubuntu has a for-profit corp, working night and day to sandpaper down
       all of the rough edges, especially when it comes to using the OS as
       a desktop system.  FreeBSD doesn't have this.  It's the absolute
       best for -any- "server" type application, but as a desktop, not so
       much.

My point is that having "repl" just "do the right thing", 100% automagically,
is a nice goal, but nmh is not a commercial product of anybody.  So some
allowances can and should be made.

I, for one, am just damn glad the thing works at all.


Regards,
rfg