Re: Prefer luatex for documentation

[Jean Abou Samra]

> Le 28 nov. 2022 à 06:54, Werner LEMBERG <wl@gnu.org> a écrit :
> 
> 
>>> What I'm now working on are macros for `configure.ac` to find a
>>> UTF-8 locale, which is important for the documentation generation
>>> in the long run.  [...]
>> 
>> Phooey, that sounds complicated.  I wonder if it would not take just
>> as much time to reimplement texindex in Perl, as suggested in
>> https://lists.gnu.org/archive/html/bug-texinfo/2022-11/msg00022.html
>> … (It would also benefit everyone using indices in Texinfo.)
> 
> `texindex` is just one part of the problem.[*]  We also need a UTF-8
> locale for both `makeinfo` and `lilypond-book` – without it, the
> output for both PDF and HTML is simply wrong.  My standard example is
> the incorrect display of the degree sign in the documentation of the
> '\rotate' markup command:
> 
>  https://lilypond.org/doc/v2.23/Documentation/notation/align
> 
> In other words, such a test for `configure.ac` is inevitable.  We
> might fix `lilypond-book` to emit a warning or even abort if it can't
> switch to a UTF-8 locale internally, but `makeinfo` (and soon
> `texi2any`, as a replacement for `texi2html`) is out of our control.


I would like to understand why this happens. The French translation has accents 
everywhere, and I never saw problems while reading it. Judging from the image, 
the texi input itself output by lilypond-book is buggy (since LilyPond does 
work on UTF-8). Any reliance of lilypond-book on the locale encoding sounds 
unexpected (except for encoding file paths for the filesystem). lilypond-book 
should just use UTF-8 Everywhere, UTF-8 Always, UTF-8 Only (TM, or rather ™).

At any rate, Python provides all the tools to work independently from the 
locale encoding. (I look forward to dropping Python 3.6 support. Then we’ll be 
able to use UTF-8 mode, which makes file reads and the like default to UTF-8. 
Then we won’t need encoding="utf-8" anymore. This mode will be the default in 
Python 3.15.)



> [*] Even if rewritten in Perl, we eventually need language-specific
>    locales to get correct collation in indices.  This means that for
>    building the German documentation, `de_DE.UTF-8` should be used,
>    etc., etc.  For this, however, there are ready-to-use autoconf
>    macros available in 'gnulib'.  If there are better suggestions to
>    avoid such additional locale tests, I'm all ears.



Doesn’t Perl implement the Unicode collation algorithm?

Jean