Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Mark H Weaver <address@hidden> skribis:

> For compatibility reasons, I think we should add "/2.22" _only_ to the
> entries of GUIX_LOCPATH.  IMO, it's unwise to change the meaning of
> LOCPATH.  Some programs, build systems, or user scripts might use
> "localedef" in a directory, set LOCPATH to that directory, and expect it
> to work.

Yes, makes sense.  I noticed that the recipe for ‘tre’ relies on
‘LOCPATH’, and I agree that it’s best to keep ‘LOCPATH’ semantics.

So, as discussed on IRC, here’s a second patch that:

  • introduces ‘GUIX_LOCPATH’, which is like ‘LOCPATH’, except
    versioned;

  • keeps ‘LOCPATH’, without appending “/X.Y” to its entries, and giving
    it less precedence than ‘GUIX_LOCPATH’;

We would tell people to migrate away from ‘LOCPATH’ in favor of
‘GUIX_LOCPATH’.

I think we should use this patch as a starting point for a discussion
for the libc people.  I’d argue that would be acceptable even for
upstream (with s/GUIX_LOCPATH/LOCALE_PATH/ for instance.)

Please review carefully.  Ideally I’d like to push this patch or a
variant thereof tomorrow, so we can start the full rebuild.

TIA!

Ludo’.

>From 7258d8c6e2a464f911723cabc2e35f7e05c49192 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <address@hidden>
Date: Thu, 1 Oct 2015 21:32:50 +0200
Subject: [PATCH] gnu: glibc: Look for locale data in versioned
 sub-directories.

* gnu/packages/patches/glibc-versioned-locpath.patch: New file.
* gnu-system.am (dist_patch_DATA): Add it.
* gnu/packages/base.scm (glibc)[source]: Use it.
  [arguments]: Add explicit version sub-directory to
  libc_cv_localedir.
  [native-search-paths]: Use 'GUIX_LOCPATH' instead of 'LOCPATH'.
  (glibc-locales, glibc-utf8-locales): Write to a VERSION
  sub-directory.
---
 doc/guix.texi                                      |  23 +-
 gnu-system.am                                      |   1 +
 gnu/packages/base.scm                              |  17 +-
 gnu/packages/patches/glibc-versioned-locpath.patch | 240 +++++++++++++++++++++
 4 files changed, 268 insertions(+), 13 deletions(-)
 create mode 100644 gnu/packages/patches/glibc-versioned-locpath.patch

diff --git a/doc/guix.texi b/doc/guix.texi
index 68ee451..1ad4229 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -933,24 +933,24 @@ daemons on the same machine.
 @node Application Setup
 @section Application Setup
 
-When using Guix on top of GNU/Linux distribution other than GuixSD, a
-few additional steps are needed to get everything in place.  Here are
-some of them.
+When using Guix on top of GNU/Linux distribution other than GuixSD---a
+so-called @dfn{foreign distro}---a few additional steps are needed to
+get everything in place.  Here are some of them.
 
 @subsection Locales
 
 @anchor{locales-and-locpath}
 @cindex locales, when not on GuixSD
 @vindex LOCPATH
address@hidden GUIX_LOCPATH
 Packages installed @i{via} Guix will not use the host system's locale
 data.  Instead, you must first install one of the locale packages
-available with Guix and then define the @code{LOCPATH} environment
-variable (@pxref{Locale Names, @code{LOCPATH},, libc, The GNU C Library
-Reference Manual}):
+available with Guix and then define the @code{GUIX_LOCPATH} environment
+variable:
 
 @example
 $ guix package -i glibc-locales
-$ export LOCPATH=$HOME/.guix-profile/lib/locale
+$ export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
 @end example
 
 Note that the @code{glibc-locales} package contains data for all the
@@ -958,6 +958,15 @@ locales supported by the address@hidden and weighs in at 
around
 address@hidden  Alternately, the @code{glibc-utf8-locales} is smaller but
 limited to a few UTF-8 locales.
 
+The @code{GUIX_LOCPATH} variable plays the exact same role as
address@hidden (@pxref{Locale Names, @code{LOCPATH},, libc, The GNU C
+Library Reference Manual}).  However, since it is honored only by Guix's
+libc, and not by the libc provided by foreign distros, using
address@hidden allows you to make sure the the foreign distro's
+programs will not end up loading incompatible locale data.  This is
+important because the locale data format used by different libc versions
+may be incompatible.
+
 @subsection X11 Fonts
 
 The majority of graphical applications use Fontconfig to locate and
diff --git a/gnu-system.am b/gnu-system.am
index f2f7e17..571e660 100644
--- a/gnu-system.am
+++ b/gnu-system.am
@@ -472,6 +472,7 @@ dist_patch_DATA =                                           
\
   gnu/packages/patches/glibc-ldd-x86_64.patch                  \
   gnu/packages/patches/glibc-locales.patch                     \
   gnu/packages/patches/glibc-o-largefile.patch                 \
+  gnu/packages/patches/glibc-versioned-locpath.patch           \
   gnu/packages/patches/gmp-arm-asm-nothumb.patch               \
   gnu/packages/patches/gnucash-price-quotes-perl.patch         \
   gnu/packages/patches/gnutls-doc-fix.patch                    \
diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index a7d9459..1402a44 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -476,6 +476,7 @@ store.")
             (modules '((guix build utils)))
             (patches (map search-patch
                           '("glibc-ldd-x86_64.patch"
+                            "glibc-versioned-locpath.patch"
                             "glibc-o-largefile.patch")))))
    (build-system gnu-build-system)
 
@@ -606,9 +607,11 @@ store.")
 
    (native-search-paths
     ;; Search path for packages that provide locale data.  This is useful
-    ;; primarily in build environments.
+    ;; primarily in build environments.  Use 'GUIX_LOCPATH' rather than
+    ;; 'LOCPATH' to avoid interference with the host system's libc on foreign
+    ;; distros.
     (list (search-path-specification
-           (variable "LOCPATH")
+           (variable "GUIX_LOCPATH")
            (files '("lib/locale")))))
 
    (synopsis "The GNU C Library")
@@ -649,10 +652,11 @@ the 'share/locale' sub-directory of this package.")
             (alist-delete 'install ,phases)))
          ((#:configure-flags flags)
           `(append ,flags
-                   ;; Use $(libdir)/locale as is the case by default.
+                   ;; Use $(libdir)/locale/X.Y as is the case by default.
                    (list (string-append "libc_cv_localedir="
                                         (assoc-ref %outputs "out")
-                                        "/lib/locale")))))))))
+                                        "/lib/locale/"
+                                        ,(package-version glibc))))))))))
 
 (define-public glibc-utf8-locales
   (package
@@ -661,7 +665,7 @@ the 'share/locale' sub-directory of this package.")
     (source #f)
     (build-system trivial-build-system)
     (arguments
-     '(#:modules ((guix build utils))
+     `(#:modules ((guix build utils))
        #:builder (begin
                    (use-modules (srfi srfi-1)
                                 (guix build utils))
@@ -669,7 +673,8 @@ the 'share/locale' sub-directory of this package.")
                    (let* ((libc      (assoc-ref %build-inputs "glibc"))
                           (gzip      (assoc-ref %build-inputs "gzip"))
                           (out       (assoc-ref %outputs "out"))
-                          (localedir (string-append out "/lib/locale")))
+                          (localedir (string-append out "/lib/locale/"
+                                                    ,version)))
                      ;; 'localedef' needs 'gzip'.
                      (setenv "PATH" (string-append libc "/bin:" gzip "/bin"))
 
diff --git a/gnu/packages/patches/glibc-versioned-locpath.patch 
b/gnu/packages/patches/glibc-versioned-locpath.patch
new file mode 100644
index 0000000..bc76521
--- /dev/null
+++ b/gnu/packages/patches/glibc-versioned-locpath.patch
@@ -0,0 +1,240 @@
+The format of locale data can be incompatible between libc versions, and
+loading incompatible data can lead to 'setlocale' returning EINVAL at best
+or triggering an assertion failure at worst.  See
+https://lists.gnu.org/archive/html/guix-devel/2015-09/msg00717.html
+for background information.
+
+To address that, this patch changes libc to honor a new 'GUIX_LOCPATH'
+variable, and to look for locale data in version-specific sub-directories of
+that variable.  So, if GUIX_LOCPATH=/foo:/bar, locale data is searched for in
+/foo/X.Y and /bar/X.Y, where X.Y is the libc version number.
+
+That way, a single 'GUIX_LOCPATH' setting can work even if different libc
+versions coexist on the system.
+
+--- a/locale/newlocale.c
++++ b/locale/newlocale.c
+@@ -30,6 +30,7 @@
+ /* Lock for protecting global data.  */
+ __libc_rwlock_define (extern , __libc_setlocale_lock attribute_hidden)
+ 
++extern error_t compute_locale_search_path (char **, size_t *);
+ 
+ /* Use this when we come along an error.  */
+ #define ERROR_RETURN                                                        \
+@@ -48,7 +49,6 @@ __newlocale (int category_mask, const char *locale, 
__locale_t base)
+   __locale_t result_ptr;
+   char *locale_path;
+   size_t locale_path_len;
+-  const char *locpath_var;
+   int cnt;
+   size_t names_len;
+ 
+@@ -102,17 +102,8 @@ __newlocale (int category_mask, const char *locale, 
__locale_t base)
+   locale_path = NULL;
+   locale_path_len = 0;
+ 
+-  locpath_var = getenv ("LOCPATH");
+-  if (locpath_var != NULL && locpath_var[0] != '\0')
+-    {
+-      if (__argz_create_sep (locpath_var, ':',
+-                           &locale_path, &locale_path_len) != 0)
+-      return NULL;
+-
+-      if (__argz_add_sep (&locale_path, &locale_path_len,
+-                        _nl_default_locale_path, ':') != 0)
+-      return NULL;
+-    }
++  if (compute_locale_search_path (&locale_path, &locale_path_len) != 0)
++    return NULL;
+ 
+   /* Get the names for the locales we are interested in.  We either
+      allow a composite name or a single name.  */
+diff --git a/locale/setlocale.c b/locale/setlocale.c
+index ead030d..0c0e314 100644
+--- a/locale/setlocale.c
++++ b/locale/setlocale.c
+@@ -215,12 +215,65 @@ setdata (int category, struct __locale_data *data)
+     }
+ }
+ 
++/* Return in *LOCALE_PATH and *LOCALE_PATH_LEN the locale data search path as
++   a colon-separated list.  Return ENOMEN on error, zero otherwise.  */
++error_t
++compute_locale_search_path (char **locale_path, size_t *locale_path_len)
++{
++  char* guix_locpath_var = getenv ("GUIX_LOCPATH");
++  char *locpath_var = getenv ("LOCPATH");
++
++  if (guix_locpath_var != NULL && guix_locpath_var[0] != '\0')
++    {
++      /* Entries in 'GUIX_LOCPATH' take precedence over 'LOCPATH'.  These
++       entries are systematically prefixed with "/X.Y" where "X.Y" is the
++       libc version.  */
++      if (__argz_create_sep (guix_locpath_var, ':',
++                           locale_path, locale_path_len) != 0
++        || __argz_suffix_entries (locale_path, locale_path_len,
++                                  "/" VERSION) != 0)
++      goto bail_out;
++    }
++
++  if (locpath_var != NULL && locpath_var[0] != '\0')
++    {
++      char *reg_locale_path = NULL;
++      size_t reg_locale_path_len = 0;
++
++      if (__argz_create_sep (locpath_var, ':',
++                           &reg_locale_path, &reg_locale_path_len) != 0)
++      goto bail_out;
++
++      if (__argz_append (locale_path, locale_path_len,
++                       reg_locale_path, reg_locale_path_len) != 0)
++      goto bail_out;
++
++      free (reg_locale_path);
++    }
++
++  if (*locale_path != NULL)
++    {
++      /* Append the system default locale directory.  */
++      if (__argz_add_sep (locale_path, locale_path_len,
++                        _nl_default_locale_path, ':') != 0)
++      goto bail_out;
++    }
++
++  return 0;
++
++ bail_out:
++  free (*locale_path);
++  *locale_path = NULL;
++  *locale_path_len = 0;
++
++  return ENOMEM;
++}
++
+ char *
+ setlocale (int category, const char *locale)
+ {
+   char *locale_path;
+   size_t locale_path_len;
+-  const char *locpath_var;
+   char *composite;
+ 
+   /* Sanity check for CATEGORY argument.  */
+@@ -251,17 +304,10 @@ setlocale (int category, const char *locale)
+   locale_path = NULL;
+   locale_path_len = 0;
+ 
+-  locpath_var = getenv ("LOCPATH");
+-  if (locpath_var != NULL && locpath_var[0] != '\0')
++  if (compute_locale_search_path (&locale_path, &locale_path_len) != 0)
+     {
+-      if (__argz_create_sep (locpath_var, ':',
+-                           &locale_path, &locale_path_len) != 0
+-        || __argz_add_sep (&locale_path, &locale_path_len,
+-                           _nl_default_locale_path, ':') != 0)
+-      {
+-        __libc_rwlock_unlock (__libc_setlocale_lock);
+-        return NULL;
+-      }
++      __libc_rwlock_unlock (__libc_setlocale_lock);
++      return NULL;
+     }
+ 
+   if (category == LC_ALL)
+diff --git a/string/Makefile b/string/Makefile
+index 8424a61..f925503 100644
+--- a/string/Makefile
++++ b/string/Makefile
+@@ -38,7 +38,7 @@ routines     := strcat strchr strcmp strcoll strcpy strcspn  
        \
+                  swab strfry memfrob memmem rawmemchr strchrnul       \
+                  $(addprefix argz-,append count create ctsep next     \
+                                    delete extract insert stringify    \
+-                                   addsep replace)                    \
++                                   addsep replace suffix)             \
+                  envz basename                                        \
+                  strcoll_l strxfrm_l string-inlines memrchr           \
+                  xpg-strerror strerror_l
+diff --git a/string/argz-suffix.c b/string/argz-suffix.c
+new file mode 100644
+index 0000000..505b0f2
+--- /dev/null
++++ b/string/argz-suffix.c
+@@ -0,0 +1,56 @@
++/* Copyright (C) 2015 Free Software Foundation, Inc.
++   This file is part of the GNU C Library.
++   Contributed by Ludovic Courtès <address@hidden>.
++
++   The GNU C Library is free software; you can redistribute it and/or
++   modify it under the terms of the GNU Lesser General Public
++   License as published by the Free Software Foundation; either
++   version 2.1 of the License, or (at your option) any later version.
++
++   The GNU C Library is distributed in the hope that it will be useful,
++   but WITHOUT ANY WARRANTY; without even the implied warranty of
++   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
++   Lesser General Public License for more details.
++
++   You should have received a copy of the GNU Lesser General Public
++   License along with the GNU C Library; if not, see
++   <http://www.gnu.org/licenses/>.  */
++
++#include <argz.h>
++#include <errno.h>
++#include <stdlib.h>
++#include <string.h>
++
++
++error_t
++__argz_suffix_entries (char **argz, size_t *argz_len, const char *suffix)
++
++{
++  size_t suffix_len = strlen (suffix);
++  size_t count = __argz_count (*argz, *argz_len);
++  size_t new_argz_len = *argz_len + count * suffix_len;
++  char *new_argz = malloc (new_argz_len);
++
++  if (new_argz)
++    {
++      char *p = new_argz, *entry;
++
++      for (entry = *argz;
++         entry != NULL;
++         entry = argz_next (*argz, *argz_len, entry))
++      {
++        p = stpcpy (p, entry);
++        p = stpcpy (p, suffix);
++        p++;
++      }
++
++      free (*argz);
++      *argz = new_argz;
++      *argz_len = new_argz_len;
++
++      return 0;
++    }
++  else
++    return ENOMEM;
++}
++weak_alias (__argz_suffix_entries, argz_suffix_entries)
+diff --git a/string/argz.h b/string/argz.h
+index bb62a31..d276a35 100644
+--- a/string/argz.h
++++ b/string/argz.h
+@@ -134,6 +134,16 @@ extern error_t argz_replace (char **__restrict __argz,
+                            const char *__restrict __str,
+                            const char *__restrict __with,
+                            unsigned int *__restrict __replace_count);
++
++/* Suffix each entry of ARGZ & ARGZ_LEN with SUFFIX.  Return 0 on success,
++   and ENOMEN if memory cannot be allocated.  */
++extern error_t __argz_suffix_entries (char **__restrict __argz,
++                                    size_t *__restrict __argz_len,
++                                    const char *__restrict __suffix);
++extern error_t argz_suffix_entries (char **__restrict __argz,
++                                  size_t *__restrict __argz_len,
++                                  const char *__restrict __suffix);
++
+
+ /* Returns the next entry in ARGZ & ARGZ_LEN after ENTRY, or NULL if there
+    are no more.  If entry is NULL, then the first entry is returned.  This
-- 
2.5.0