Re: [datasets 01/18] gui: Always convert file names to UTF-8 for use in syntax.

[Ben Pfaff]

I have to agree that testing is warranted, but let me add some
more background here to explain.

Before commit 9ade26c8349 "lexer: Reimplement for better
testability and internationalization", the encoding of syntax
files was not well-defined.  It was reasonable, then, to put file
names in generated syntax in the file name encoding.

Commit 9ade26c8349 changed the encoding of syntax so that it was
always in UTF-8.  This meant that file names in syntax had to be
converted back into the file name encoding before trying to open
the files, and I made that change (you can see, for example, the
call to utf8_to_filename in do_insert() in
src/language/utilities/include.c).

But I forgot that the GUI needs to convert its file names into
UTF-8 when it is generating syntax, so this commit fixes that up.

(I've now added the above background to the commit message.)

Does the reason for this change make sense now?

John Darrington <address@hidden> writes:

> Messing with this has always caused a lot of problems in the past.
> I'd like to see a fair bit of testing on different OSes and with 
> different locales - using non-ascii filenames of course.

> On Sat, Apr 30, 2011 at 10:36:29PM -0700, Ben Pfaff wrote:
>      Syntax as understood by the lexer is always in UTF-8, so file names
>      have to be in UTF-8 too.  (The PSPP code that opens files based on
>      strings from syntax should already be using utf8_to_filename() to
>      convert them properly before opening.)

-- 
Ben Pfaff 
http://benpfaff.org