Re: lexer.ll: Warn about non-UTF-8 characters (issue 5505090)

[Keith OHara]

On Sun, 01 Jan 2012 01:56:04 -0800, <address@hidden> wrote:

> http://codereview.appspot.com/5505090/diff/1/lily/lexer.ll#newcode134
> lily/lexer.ll:134: A            [a-zA-Z\200-\377]
> On 2012/01/01 02:01:11, Keith wrote:
> > non-ASCII characters are used internally as-read, tested below only to
> warn the
> > user if the input is invalid as UTF-8
>
> You mean, you want a comment here?

Nope; I was writing what I understood the code to be doing, so you could 
complain if the innocent reader's understanding is different from your intent.
I confirmed that a Latin1-encoded file using French variable names continues to 
work, with warnings.

> http://codereview.appspot.com/5505090/diff/1/lily/lexer.ll#newcode1083
> lily/lexer.ll:1083: LexerWarning (_ ("non-UTF-8 characters").c_str ());
> On 2012/01/01 02:01:11, Keith wrote:
> > I suggest "unsupported file encoding, expected UTF-8"
>
> I think that error message would be appropriate once per run, either
> because of making it a fatal error, or as a summary at the end.  But
> when flagging the actual place where there is a problematic character?

My suggestion was to emphasize the encoding as the problem, rather than the 
character(s) as the problem --- even if the error is output at each character 
for which the encoding makes a difference.