Re: Prefer luatex for documentation

[Werner LEMBERG]

Date: Mon, 28 Nov 2022 09:18:25 +0100

>> `texindex` is just one part of the problem.[*] We also need a UTF-8
>> locale for both `makeinfo` and `lilypond-book` – without it, the
>> output for both PDF and HTML is simply wrong.  My standard example
>> is the incorrect display of the degree sign in the documentation of
>> the '\rotate' markup command:
>> 
>>  https://lilypond.org/doc/v2.23/Documentation/notation/align
>> 
>> In other words, such a test for `configure.ac` is inevitable.  We
>> might fix `lilypond-book` to emit a warning or even abort if it
>> can't switch to a UTF-8 locale internally, but `makeinfo` (and soon
>> `texi2any`, as a replacement for `texi2html`) is out of our
>> control.
> 
> I would like to understand why this happens. The French translation
> has accents everywhere, and I never saw problems while reading
> it. Judging from the image, the texi input itself output by
> lilypond-book is buggy (since LilyPond does work on UTF-8).

Yes, it seems so.  Maybe there is still a writing instruction in
`lilypond-book` not marked as being Unicode?

> Any reliance of lilypond-book on the locale encoding sounds
> unexpected (except for encoding file paths for the
> filesystem). lilypond-book should just use UTF-8 Everywhere, UTF-8
> Always, UTF-8 Only (TM, or rather ™).

Yep.

> At any rate, Python provides all the tools to work independently
> from the locale encoding. (I look forward to dropping Python 3.6
> support. Then we’ll be able to use UTF-8 mode, which makes file
> reads and the like default to UTF-8. Then we won’t need
> encoding="utf-8" anymore. This mode will be the default in Python
> 3.15.)

I'm not sure when we can drop Python 3.6...

>> [*] Even if rewritten in Perl, we eventually need language-specific
>>     locales to get correct collation in indices.  This means that
>>     for building the German documentation, `de_DE.UTF-8` should be
>>     used, etc., etc.  For this, however, there are ready-to-use
>>     autoconf macros available in 'gnulib'.  If there are better
>>     suggestions to avoid such additional locale tests, I'm all
>>     ears.
> 
> Doesn’t Perl implement the Unicode collation algorithm?

It does, but AFAICS it does *not* have the locale data for sorting
built in.  In https://perldoc.perl.org/perllocale you can read

  Definitions for locales that you use must be installed.

This means again that we have to test for language-specific UTF-8
locales...


    Werner