Re: Inadequate documentation of silly characters on screen.

[Stephen J. Turnbull]

David Kastrup writes:

 > > However, I think a well-behaved platform should by default error
 > > (something derived from invalid-state, in XEmacs's error hierarchy) in
 > > such a case; normally this means corruption in the file.
 > 
 > We take care that it does not mean corruption.

I meant pre-existing corruption, like your pre-existing disposition to
bash XEmacs.  Please take it elsewhere; it doesn't belong on Emacs
channels.  (Of course I'd prefer not to see it on XEmacs channels
either, but at least it wouldn't be entirely off-topic there.)

 > And more often it means that you might have been loading with the
 > wrong encoding (people do that all the time).  If you edit some
 > innocent ASCII part

You can't do that if the file is not in a buffer because the encoding
error aborted the conversion.  Aborting the conversion is what the
Unicode Consortium requires, too, IIRC: errors in UTF-8 (or any other
UTF for that matter) are considered *fatal* by the standard.  Exactly
what that means is up to the application to decide.  One plausible
approach would be to do what you do now, but make the buffer read-only.

 > Sometimes there is no "right encoding".

So what?  The point is that there certainly are *wrong* encodings,
namely ones that will result in corruption if you try to save the file
in that encoding.  There are usually many "usable" encodings (binary
is always available, for example).  Some will be preferred by users,
and that will be reflected in coding system precedence.

But when faced with ambiguity, it is best to refuse to guess.

 > We currently _have_ [a scheme for encoding invalid sequences of
 > code units] in place.  We just use different Unicode-invalid code
 > points [from Python].

Conceded.  I realized that later; the important difference is that
Python only uses that scheme when explicitly requested.