bug#31679: 26.1; detect-coding-string does not detect UTF-16

[Eli Zaretskii]

> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
> Date: Fri, 01 Jun 2018 21:40:32 +0200
> 
> I have been trying this (in real life the strings are often longer, of
> course):
> 
>   (detect-coding-string "h\0t\0m\0l\0")
> 
> And I was surprised that this does not detect UTF-16 but instead gives
> (no-conversion).

First, you should lose the trailing null (or add one more), since
UTF-16 strings must, by definition, have an even number of bytes.

Next, you should disable null byte detection by binding
inhibit-null-byte-detection to a non-nil value, because otherwise
Emacs's guesswork will prefer no-conversion, assuming this is binary
data.

If you do that, you get

  (let ((inhibit-null-byte-detection t))
    (detect-coding-string "h\0t\0m\0l"))
  => (undecided)

Why? because it is perfectly valid for a plain-ASCII string to include
null bytes, so Emacs prefers to guess ASCII.

As another example, try this:

  (prefer-coding-system 'utf-16)
  (let ((inhibit-null-byte-detection t))
    (detect-coding-string (encode-coding-string "áçðë" 'utf-16-be) t))
  => utf-16

but

  (let ((inhibit-null-byte-detection t))
    (detect-coding-string
      (substring (encode-coding-string "áçðë" 'utf-16-be) 2) t))
  =>iso-latin-1

So even when UTF-16 is the most preferred encoding, just removing the
BOM is enough to let Emacs prefer something other than UTF-16.

Morale: detecting an encoding in Emacs is based on heuristic
_guesswork_, which is heavily biased to what is deemed to be the most
frequent use cases.  And UTF-16 is quite infrequent, at least on Posix
hosts.

IOW, detecting encoding in Emacs is not as reliable as you seem to
expect.  If you _know_ the text is in UTF-16, just tell Emacs to use
that, don't let it guess.