[RFC PATCH v2 07/10] mkvenv: work around broken pip installations on Debian 10

[John Snow]

This is a workaround intended for Debian 10, where the debian-patched
pip does not function correctly if accessed from within a virtual
environment.

We don't support Debian 10 any longer, but it's possible that this bug
might appear on other derivative platforms and this workaround may prove
useful.

RFC, a note from Paolo:

"BTW, another way to repair Debian 10's pip is to create a symbolic link
to sys.base_prefix + '/share/python-wheels' in sys.prefix +
'/share/python-wheels'. Since this is much faster, perhaps it can be
done unconditionally [...] ?"

I was slightly apprehensive about this as it felt "hackier", but it is
indeed a lot less code and much faster. It's probably low-risk. Should
we do that instead, or should we just scrap any fix at all under the
premise that Debian 10 support is dropped anyway?

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/mkvenv.py | 61 +++++++++++++++++++++++++++++-----------
 1 file changed, 44 insertions(+), 17 deletions(-)

diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py
index 317697a953..6e1a979355 100644
--- a/python/scripts/mkvenv.py
+++ b/python/scripts/mkvenv.py
@@ -103,27 +103,25 @@ def need_ensurepip() -> bool:
     return True
 
 
-def check_ensurepip(with_pip: bool) -> None:
+def check_ensurepip(prefix: str = '', suggest_remedy: bool = False) -> None:
     """
     Check that we have ensurepip.
 
     Raise a fatal exception with a helpful hint if it isn't available.
     """
-    if not with_pip:
-        return
-
     if not find_spec("ensurepip"):
         msg = ("Python's ensurepip module is not found.\n"
 
                "It's normally part of the Python standard library, "
                "maybe your distribution packages it separately?\n"
 
-               "Either install ensurepip, or alleviate the need for it in the "
-               "first place by installing pip and setuptools for "
-               f"'{sys.executable}'.\n"
-
-               "(Hint: Debian puts ensurepip in its python3-venv package.)")
-        raise Ouch(msg)
+               "(Debian puts ensurepip in its python3-venv package.)\n")
+        if suggest_remedy:
+            msg += (
+                "Either install ensurepip, or alleviate the need for it in the 
"
+                "first place by installing pip and setuptools for "
+                f"'{sys.executable}'.\n")
+        raise Ouch(prefix + msg)
 
     # ensurepip uses pyexpat, which can also go missing on us:
     if not find_spec("pyexpat"):
@@ -132,12 +130,13 @@ def check_ensurepip(with_pip: bool) -> None:
                "It's normally part of the Python standard library, "
                "maybe your distribution packages it separately?\n"
 
-               "Either install pyexpat, or alleviate the need for it in the "
-               "first place by installing pip and setuptools for "
-               f"'{sys.executable}'.\n\n"
-
-               "(Hint: NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)")
-        raise Ouch(msg)
+               "(NetBSD's pkgsrc debundles this to e.g. 'py310-expat'.)\n")
+        if suggest_remedy:
+            msg += (
+                "Either install pyexpat, or alleviate the need for it in the "
+                "first place by installing pip and setuptools for "
+                f"'{sys.executable}'.\n")
+        raise Ouch(prefix + msg)
 
 
 def make_venv(  # pylint: disable=too-many-arguments
@@ -169,7 +168,8 @@ def make_venv(  # pylint: disable=too-many-arguments
         with_pip = True if not system_site_packages else need_ensurepip()
         logger.debug("with_pip unset, choosing %s", with_pip)
 
-    check_ensurepip(with_pip)
+    if with_pip:
+        check_ensurepip(suggest_remedy=True)
 
     if symlinks is None:
         # Default behavior of standard venv CLI
@@ -349,6 +349,31 @@ def _get_entry_points() -> Iterator[Dict[str, str]]:
         logger.debug("wrote '%s'", script_path)
 
 
+def checkpip():
+    """
+    Debian10 has a pip that's broken when used inside of a virtual environment.
+
+    We try to detect and correct that case here.
+    """
+    try:
+        import pip._internal
+        logger.debug("pip appears to be working correctly.")
+        return
+    except ModuleNotFoundError as exc:
+        if exc.name == 'pip._internal':
+            # Uh, fair enough. They did say "internal".
+            # Let's just assume it's fine.
+            return
+        logger.warning("pip appears to be malfunctioning: %s", str(exc))
+
+    check_ensurepip("pip appears to be non-functional, and ")
+
+    logging.debug("Attempting to repair pip ...")
+    subprocess.run((sys.executable, '-m', 'ensurepip'),
+                   stdout=subprocess.DEVNULL, check=True)
+    logging.debug("Pip is now (hopefully) repaired!")
+
+
 def post_venv_setup(bin_path: str, packages: Sequence[str] = ()) -> None:
     """
     This is intended to be run *inside the venv* after it is created.
@@ -358,6 +383,8 @@ def post_venv_setup(bin_path: str, packages: Sequence[str] 
= ()) -> None:
                  bin_path, packages)
     generate_console_scripts(python_path, bin_path, packages)
 
+    # Test for a broken pip (Debian 10 or derivative?) and fix it if needed
+    checkpip()
 
 def main() -> int:
     """CLI interface to make_qemu_venv. See module docstring."""
-- 
2.39.2