Re: MinGW vs. setlocale

[Ludovic Courtès]

Eli Zaretskii <address@hidden> skribis:

>> From: address@hidden (Ludovic Courtès)
>> Date: Mon, 09 Jun 2014 21:30:46 +0200
>> 
>> > 1. i18n.test completely fails, because it depends on the ability to
>> >    change the program's locale at run time.  I wish this whole test
>> >    were skipped on Windows.  (I'm quite sure I reported this last
>> >    year.)
>> 
>> What does ‘setlocale’ return when called more than once on Windows?  Is
>> there an exception thrown or something that would allow i18n.test to
>> determine that tests should be skipped?
>
> A very good question, thank you for asking it.  The short answer is
> that yes, it threw an exception.  The long answer is that while
> looking into the reasons of these exceptions, I found a few snafus,
> and succeeded to fix some, so that most of the i18n test now works on
> Windows.

Heh, thanks for investigating!

> First, make-locale threw an exception, because it tried to call
> 'setlocale' with LC_MESSAGES, which the Windows runtime doesn't
> support.  locale-categories.h tried to avoid that by conditioning that
> call by LC_MESSAGES being defined, but the oh-so-helpful libintl.h
> header file just happens to define it to some arbitrary large
> constant.  So the ifdef didn't work, and setlocale barfed.  Here's the
> suggested solution:

Pushed as c84f25b.

> The next problem is that i18n.test uses Posix locale strings, whereas
> the Windows runtime names the same locales by different names.
> Moreover, Windows 'setlocale' doesn't support UTF-8 encoding (even
> though a Windows UTF-8 codepage exists).  So every test for a locale
> other than "C" was failing, because setlocale failed.  I replaced
> Posix locales with similar Windows ones; see the following patch, in
> which I also emoved all but one LC_MESSAGES, because these always fail
> on Windows:

Ditto.

>  (define %french-locale-name
> -  "fr_FR.ISO-8859-1")
> +  (if (string-contains %host-type "-mingw32")
> +      "fra_FRA.850"
> +      "fr_FR.ISO-8859-1"))
>  
>  (define %french-utf8-locale-name
> -  "fr_FR.UTF-8")
> +  (if (string-contains %host-type "-mingw32")
> +      "fra_FRA.1252"
> +      "fr_FR.UTF-8"))
>  
>  (define %turkish-utf8-locale-name
> -  "tr_TR.UTF-8")
> +  (if (string-contains %host-type "-mingw32")
> +      "tur_TRK.1254"
> +      "tr_TR.UTF-8"))
>  
>  (define %german-utf8-locale-name
> -  "de_DE.UTF-8")
> +  (if (string-contains %host-type "-mingw32")
> +      "deu_DEU.1252"
> +      "de_DE.UTF-8"))
>  
>  (define %greek-utf8-locale-name
> -  "el_GR.UTF-8")
> +  (if (string-contains %host-type "-mingw32")
> +      "grc_ELL.1253"
> +      "el_GR.UTF-8"))
>  
>  (define %american-english-locale-name
>    "en_US")
> @@ -148,13 +161,14 @@
>    (under-locale-or-unresolved %french-utf8-locale thunk))
>  
>  (define (under-turkish-utf8-locale-or-unresolved thunk)
> -  ;; FreeBSD 8.2 and 9.1, Solaris 2.10, and Darwin 8.11.0 have a broken
> -  ;; tr_TR locale where `i' is mapped to uppercase `I' instead of `Ä°',
> -  ;; so disable tests on that platform.
> +  ;; FreeBSD 8.2 and 9.1, Solaris 2.10, Darwin 8.11.0, and MinGW have
> +  ;; a broken tr_TR locale where `i' is mapped to uppercase `I'
> +  ;; instead of `İ', so disable tests on that platform.
>    (if (or (string-contains %host-type "freebsd8")
>            (string-contains %host-type "freebsd9")
>            (string-contains %host-type "solaris2.10")
> -          (string-contains %host-type "darwin8"))
> +          (string-contains %host-type "darwin8")
> +       (string-contains %host-type "-mingw32"))
>        (throw 'unresolved)
>        (under-locale-or-unresolved %turkish-utf8-locale thunk)))
>  
> @@ -192,7 +206,10 @@
>          ;; strings.
>          (dynamic-wind
>            (lambda ()
> -            (setlocale LC_ALL "fr_FR.UTF-8"))
> +         (setlocale LC_ALL
> +                    (if (string-contains %host-type "-mingw32")
> +                        "fra_FRA.1252"
> +                        "fr_FR.UTF-8")))
>          (lambda ()
>            (string-locale-ci=? "œuf" "ŒUF"))
>          (lambda ()
>            (setlocale LC_ALL "C"))))))

Pushed that as 700f6cd.

> After all these changes, some tests still fail or throw exceptions:
>
>   UNRESOLVED: i18n.test: text collation (French): string-locale-ci=?
>   UNRESOLVED: i18n.test: text collation (French): string-locale-ci=? (2 args, 
> wide strings)
>   UNRESOLVED: i18n.test: text collation (French): string-locale-ci=? (3 args, 
> wide strings)
>   UNRESOLVED: i18n.test: text collation (French): string-locale-ci<>?
>   UNRESOLVED: i18n.test: text collation (French): string-locale-ci<>? (wide 
> strings)
>   UNRESOLVED: i18n.test: text collation (French): string-locale-ci<>? (wide 
> and narrow strings)
>   UNRESOLVED: i18n.test: text collation (French): char-locale-ci<>?
>   UNRESOLVED: i18n.test: text collation (French): char-locale-ci<>? (wide)
>   UNRESOLVED: i18n.test: text collation (Greek): string-locale-ci=?
>   UNRESOLVED: i18n.test: character mapping: char-locale-upcase Turkish
>   UNRESOLVED: i18n.test: character mapping: char-locale-downcase Turkish
>   UNRESOLVED: i18n.test: string mapping: string-locale-upcase Greek
>   UNRESOLVED: i18n.test: string mapping: string-locale-upcase Greek (two 
> sigmas)
>   UNRESOLVED: i18n.test: string mapping: string-locale-downcase Greek
>   UNRESOLVED: i18n.test: string mapping: string-locale-downcase Greek (two 
> sigmas)
>   UNRESOLVED: i18n.test: string mapping: string-locale-upcase Turkish
>   UNRESOLVED: i18n.test: string mapping: string-locale-downcase Turkish
>
> I don't know why these fail.

Note that “UNRESOLVED” is not a failure; it means “we can’t run this
test here, so skip it.”

> Is it possible that the underlying functions assume that the string
> arguments are encoded according to the locale's codeset?  If so, since
> the source file is encoded in UTF-8, that won't work on Windows, and
> the strings need to be recoded before they are passed to libunistring
> functions.  Any ideas for debugging this are welcome.

‘under-locale-or-unresolved’ catches any failure to install the French,
Greek, etc. locale (which can happens if the locale is missing on the
system), and throws 'unresolved when that happens.

So presumably, those UNRESOLVED mean that (setlocale LC_ALL "fra_FRA.850")
and similar calls just fail.

>   FAIL: i18n.test: nl-langinfo et al.: locale-day (French)
>   FAIL: i18n.test: nl-langinfo et al.: locale-day (French, using 
> `%global-locale')
>
> This is because gnulib's nl_langinfo only supports C locale for the
> day names.  I'm taking this up with gnulib maintainers.

Great.

>   FAIL: i18n.test: number->locale-string: French: integer
>   FAIL: i18n.test: number->locale-string: French: fraction
>   FAIL: i18n.test: number->locale-string: French: fraction, 1 digit
>   FAIL: i18n.test: monetary-amount->locale-string: French: integer
>   FAIL: i18n.test: monetary-amount->locale-string: French: fraction
>
> There's no blank after the 7th digit, where the test expects it.  Not
> sure what kind of problem is that, perhaps again due to gnulib's
> nl_langinfo.

You could try this:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,m (ice-9 i18n)
scheme@(ice-9 i18n)> (locale-decimal-point (make-locale LC_ALL "fr_FR"))
$2 = ","
scheme@(ice-9 i18n)> (locale-thousands-separator (make-locale LC_ALL "fr_FR"))
$3 = " "
--8<---------------cut here---------------end--------------->8---

(Using the right locale name.)

>   UNRESOLVED: i18n.test: format ~h: French: 12345.5678
>   UNRESOLVED: i18n.test: format ~h: English: 12345.5678
>
> ~h is not supported on Windows.

~h is implemented using ‘number->locale-string’.

Thank you!

Ludo’.