[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Nmh-workers] nmh architecture discussion: format engine character s

From: Earl Hood
Subject: Re: [Nmh-workers] nmh architecture discussion: format engine character set
Date: Tue, 11 Aug 2015 11:43:12 -0500
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0

On 8/11/2015 11:08 AM, Jon Steinhart wrote:

It seems to me that the only solution is to use Unicode internally.
Disgusting as it seems to those of us who are old enough to hoard
bytes, we might want to consider using something other than UTF-8
for the internal representation.  Using UTF-16 wouldn't be horrible
but I recall that the Unicode folks made a botch of things so that
one really needs 24 bits now, which really means using 32 internally.

UTF-8 should be sufficient, and it does not suffer from byte-ordering
and byte order mark problems that other encodings suffer from.

Some general comments about this discussion:

- Message should be stored in their original forms.  I.e.  The
  character encoding transformation should only be done for
  display/access purposes.

  Main reasons:
    - Protects against bugs in encoding transformation code.  The
      original message is always left untouched.
    - Maintains compatibility with folks that use external tools and
      scripts to access nmh messages.
    - Avoids invalidating any digital signatures since any
      modification of the message will invalidate them.

  If there is a desire to support transformation of incoming mail, so
  mail is stored in "normalized form", then it should be a configurable
  option.  I personally would not want all my stored email converted to
  UTF-8 due to reasons cited above.

- I think using a character encoding library is unavoidable.  Is iconv()
  sufficient?.  If UTF-8 is to be used as the normalized encoding
  format, a library is needed that can transform the various encodings
  into it, and likely from it.  Maybe it is not as big an issue as it
  was in the past, but not everyone was sold on Unicode.  In my
  mail-related project, I had users that preferred they local character
  encoding formats over anything Unicode related.

  Character encoding choices can get quite political.

  If a library is adopted, then users have full control of what encoding
  they prefer.

  Side Note: My project is in Perl, so I can leverage Perl's Encode
  module for character encoding conversion.  However, I even had code
  that dealt with character encoding conversion pre-Encode days.

- As for parsing message headers, make it a configurable option
  on what the default character encoding should be.  UTF-8 could be the
  default (which is fortunately is US-ASCII compatible).

  Real-world note: I have encountered emails that actually use a
  non-ASCII default encoding for message header data.  Messages in
  non-English locale.  Technically, these message are not conformant to
  the RFCs, but such messages actually exist.  Hence, in my project, I
  support an option that specifies what the default encoding is.

- I think it is perfectly reasonable to leverage the current locale
  setting to determine defaults, but one should be able to explicit
  override such defaults via .mh_profile and command-line options.

On the output side, we just have to do the best we can if characters in
the input locale can't be represented in the output locale.  This is
independent of the internal representation.

A '?' is commonly used when one cannot map a character in one encoding
to another, likely since '?' is likely representable in all encodings.
Some systems will use a special glyph, but I think that is not possible
for nmh to do.

Warning message(s) should be generated when character data is lost due
to conversion.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]