01/131: gnu: python: Replace PYTHONPATH by GUIX_PYTHONPATH.

[guix-commits]

apteryx pushed a commit to branch core-updates
in repository guix.

commit cb72f9a773e0931ee3758c851d96007ded034e4c
Author: Maxim Cournoyer <maxim.cournoyer@gmail.com>
AuthorDate: Thu Jan 21 23:26:01 2021 -0500

    gnu: python: Replace PYTHONPATH by GUIX_PYTHONPATH.
    
    Using PYTHONPATH as a mean to discover the Python packages had the following
    issues:
    
            1. It is not versioned, so different versions of Python would clash 
if
            installed in a shared profile.
    
            2. It would interfere with the host Python site on foreign
            distributions, sometimes preventing a a user to login their GDM
            session (!).
    
            3. It would take precedence over user installed Python packages
            installed through pip.
    
            4. It would leak into Python virtualenvs, which are supposed to 
create
            isolated Python environments.
    
    This changes fixes the above issues by making use of a sitecustomize.py
    module.  The newly introduced GUIX_PYTHONPATH environment variable is read
    from the environment, filtered for the current Python version of the
    interpreter, and spliced in 'sys.path' just before Python's own site 
location,
    which provides the expected behavior.
    
    * gnu/packages/aux-files/python/sitecustomize.py: New file.
    * Makefile.am: Register it.
    * gnu/packages/python.scm (customize-site)
    (guix-pythonpath-search-path): New procedures.
    (python-2.7)[phases]{install-sitecustomize.py}: New phase.
    [native-inputs]{sitecustomize.py}: New input.
    [native-search-paths]: Replace PYTHONPATH with GUIX_PYTHONPATH.
    (python-3.9)[native-search-paths]: Likewise.
    [phases]{install-sitecustomize}: Override with correct version.
    [native-search-paths]: Replace PYTHONPATH with GUIX_PYTHONPATH.
    * gnu/packages/commencement.scm (python-boot0):
    [phases]{install-sitecustomize}: Likewise.
    [native-inputs]{sitecustomize.py}: New input.
    [native-search-paths]: Replace PYTHONPATH with GUIX_PYTHONPATH.
    * guix/build/python-build-system.scm (site-packages): Do not add a trailing
    '/'.
    
    squash! gnu: python: Replace PYTHONPATH by GUIX_PYTHONPATH.
---
 Makefile.am                                    |  1 +
 gnu/packages/aux-files/python/sitecustomize.py | 41 ++++++++++++++++++
 gnu/packages/commencement.scm                  | 14 +++++--
 gnu/packages/python.scm                        | 57 ++++++++++++++++++--------
 guix/build/python-build-system.scm             |  4 +-
 5 files changed, 95 insertions(+), 22 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index dc5cf9b..7ef8d4d 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -380,6 +380,7 @@ AUX_FILES =                                         \
   gnu/packages/aux-files/linux-libre/4.4-i686.conf     \
   gnu/packages/aux-files/linux-libre/4.4-x86_64.conf   \
   gnu/packages/aux-files/pack-audit.c                  \
+  gnu/packages/aux-files/python/sitecustomize.py       \
   gnu/packages/aux-files/run-in-namespace.c
 
 # Templates, examples.
diff --git a/gnu/packages/aux-files/python/sitecustomize.py 
b/gnu/packages/aux-files/python/sitecustomize.py
new file mode 100644
index 0000000..65d3c7d
--- /dev/null
+++ b/gnu/packages/aux-files/python/sitecustomize.py
@@ -0,0 +1,41 @@
+# -*- coding: utf-8 -*-
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import sys
+
+python_root = os.path.realpath(sys.executable).split('/bin/')[0]
+major_minor = '{}.{}'.format(sys.version_info[0], sys.version_info[1])
+site_packages_prefix = 'lib/python' + major_minor + '/site-packages'
+python_site = python_root + '/' + site_packages_prefix
+
+try:
+    all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(':')
+except KeyError:
+    all_sites_raw = []
+# Normalize paths, otherwise a trailing slash would cause it to not match.
+all_sites_norm = [os.path.normpath(p) for p in all_sites_raw]
+matching_sites = [p for p in all_sites_norm
+                  if p.endswith(site_packages_prefix)]
+
+# Insert sites matching the current version into sys.path, right before
+# Python's own site.
+sys_path_absolute = [os.path.realpath(p) for p in sys.path]
+index = sys_path_absolute.index(python_site)
+sys.path = sys.path[:index] + matching_sites + sys.path[index:]
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 3e91179..22ec5a0 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -9,6 +9,7 @@
 ;;; Copyright © 2019, 2020 Marius Bakke <mbakke@fastmail.com>
 ;;; Copyright © 2020 Timothy Sample <samplet@ngyro.com>
 ;;; Copyright © 2020 Guy Fleury Iteriteka <gfleury@disroot.org>
+;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -54,6 +55,7 @@
   #:use-module (guix download)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system trivial)
+  #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix memoization)
   #:use-module (guix utils)
@@ -3095,7 +3097,9 @@ memoized as a function of '%current-system'."
     (inputs
      `(,@(%boot0-inputs)
        ("expat" ,expat-sans-tests)))              ;remove OpenSSL, zlib, etc.
-    (native-inputs '())                           ;and pkg-config
+    (native-inputs                                ;and pkg-config
+     `(("sitecustomize.py" ,(local-file (search-auxiliary-file
+                                         "python/sitecustomize.py")))))
     (arguments
      `(#:implicit-inputs? #f
        #:guile ,%bootstrap-guile
@@ -3136,8 +3140,12 @@ memoized as a function of '%current-system'."
                              (substitute* "Lib/plat-generic/regen"
                                (("/usr/include/")
                                 (string-append libc "/include/")))))))
-                     '())))
-           ((#:tests? _ #f) #f))))))
+                     '())
+               (replace 'install-sitecustomize.py
+                 ,(customize-site version))))
+           ((#:tests? _ #f) #f))))
+    (native-search-paths
+     (list (guix-pythonpath-search-path version)))))
 
 (define/system-dependent ld-wrapper-boot0
   ;; The first 'ld' wrapper, defined with 'define/system-dependent' because
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index d3df1a4..258e074 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -96,13 +96,41 @@
   #:use-module (gnu packages tcl)
   #:use-module (gnu packages tls)
   #:use-module (gnu packages xml)
+  #:use-module (guix gexp)
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix utils)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system trivial)
   #:use-module (srfi srfi-1)
-  #:use-module (srfi srfi-26))
+  #:use-module (srfi srfi-26)
+
+  #:export (customize-site
+            guix-pythonpath-search-path))
+
+(define* (customize-site version)
+  "Generate a install-sitecustomize.py phase, using VERSION."
+  `(lambda* (#:key inputs outputs #:allow-other-keys)
+     (let* ((out (assoc-ref outputs "out"))
+            (site-packages (string-append
+                            out "/lib/python"
+                            ,(version-major+minor version)
+                            "/site-packages"))
+            (sitecustomize.py (assoc-ref inputs "sitecustomize.py"))
+            (dest (string-append site-packages "/sitecustomize.py")))
+       (mkdir-p site-packages)
+       (copy-file sitecustomize.py dest)
+       ;; Set the correct permissions on the installed file, else the byte
+       ;; compilation phase fails with a permission denied error.
+       (chmod dest #o644))))
+
+(define (guix-pythonpath-search-path version)
+  "Generate a GUIX_PYTHONPATH search path specification, using VERSION."
+  (search-path-specification (variable "GUIX_PYTHONPATH")
+                             (files (list (string-append
+                                           "lib/python"
+                                           (version-major+minor version)
+                                           "/site-packages")))))
 
 (define-public python-2.7
   (package
@@ -266,8 +294,7 @@
                                             (not
                                              (string-prefix? "test_support."
                                                              file))))))
-                      (call-with-output-file "__init__.py" (const #t))
-                      #t)))))))
+                      (call-with-output-file "__init__.py" (const #t)))))))))
          (add-after 'remove-tests 'rebuild-bytecode
            (lambda* (#:key outputs #:allow-other-keys)
              (let ((out (assoc-ref outputs "out")))
@@ -313,7 +340,9 @@
                                     "/site-packages")))
                       (install-file tkinter.so target)
                       (delete-file tkinter.so)))))
-               #t))))))
+               #t)))
+         (add-after 'install 'install-sitecustomize.py
+           ,(customize-site version)))))
     (inputs
      `(("bzip2" ,bzip2)
        ("expat" ,expat)
@@ -327,15 +356,15 @@
        ("tk" ,tk)))                     ; for tkinter
     (native-inputs
      `(("pkg-config" ,pkg-config)
+       ("sitecustomize.py" ,(local-file (search-auxiliary-file
+                                         "python/sitecustomize.py")))
        ;; When cross-compiling, a native version of Python itself is needed.
        ,@(if (%current-target-system)
              `(("python2" ,this-package)
                ("which" ,which))
              '())))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files '("lib/python2.7/site-packages")))))
+     (list (guix-pythonpath-search-path version)))
     (home-page "https://www.python.org";)
     (synopsis "High-level, dynamically-typed programming language")
     (description
@@ -438,8 +467,7 @@ data types.")
                (setenv "TZDIR"
                        (string-append (assoc-ref
                                        (or native-inputs inputs) "tzdata")
-                                      "/share/zoneinfo"))
-               #t))
+                                      "/share/zoneinfo"))))
            (replace 'rebuild-bytecode
              (lambda* (#:key outputs #:allow-other-keys)
                (let ((out (assoc-ref outputs "out")))
@@ -462,8 +490,9 @@ data types.")
                                ;; Don't build lib2to3, because it's Python 2 
code.
                                "-x" "lib2to3/.*"
                                ,out))))
-                  (list "none" "-O" "-OO"))
-                 #t)))))))
+                  (list "none" "-O" "-OO")))))
+           (replace 'install-sitecustomize.py
+             ,(customize-site version))))))
     (native-inputs
      `(("tzdata" ,tzdata-for-tests)
        ,@(if (%current-target-system)
@@ -471,11 +500,7 @@ data types.")
              '())
        ,@(package-native-inputs python-2)))
     (native-search-paths
-     (list (search-path-specification
-            (variable "PYTHONPATH")
-            (files (list (string-append "lib/python"
-                                        (version-major+minor version)
-                                        "/site-packages"))))
+     (list (guix-pythonpath-search-path version)
            ;; Used to locate tzdata by the zoneinfo module introduced in
            ;; Python 3.9.
            (search-path-specification
diff --git a/guix/build/python-build-system.scm 
b/guix/build/python-build-system.scm
index aad8625..c6c88e6 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -167,9 +167,7 @@
   "Return the path of the current output's Python site-package."
   (let* ((out (python-output outputs))
          (python (assoc-ref inputs "python")))
-    (string-append out "/lib/python"
-                   (python-version python)
-                   "/site-packages/")))
+    (string-append out "/lib/python" (python-version python) 
"/site-packages")))
 
 (define (add-installed-pythonpath inputs outputs)
   "Prepend the Python site-package of OUTPUT to PYTHONPATH.  This is useful