On Mon, Apr 24, 2023 at 04:02:28PM -0400, John Snow wrote:
> GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/846869409
> (All green, except Python self-tests, see below)
>
> This patch series creates a mandatory python virtual environment
> ("venv") during configure time and uses it to ensure the availability of
> meson and sphinx.
>
> See https://www.qemu.org/2023/03/24/python/ for details. The summary is
> that the goal of this series is to ensure that the `python` used to run
> meson is the same `python` used to run Sphinx, tests, and any build-time
> python scripting we have. As it stands, meson and sphinx (and their
> extensions) *may* run in a different python environment than the one
> configured and chosen by the user at configure/build time.
>
> The effective change of this series is that QEMU will now
> unconditionally create a venv at configure-time and will ensure that
> meson (and sphinx, if docs are enabled) are available through that venv.
>
> Some important points as a pre-emptive "FAQ":
>
> - This venv is unconditionally created and lives at {build_dir}/pyvenv.
>
> - The python interpreter used by this venv is always the one identified
> by configure. (Which in turn is always the one specified by --python
> or $PYTHON)
>
> - *almost* all python scripts in qemu.git executed as part of the build
> system, meson, sphinx, avocado tests, vm tests or CI are always
> executed within this venv.
>
> (iotests are not yet integrated; I plan to tackle this separately as a
> follow-up in order to have a more tightly focused scope on that
> series.)
>
> - It remains possible to build and test fully offline.
> (In most cases, you just need meson and sphinx from your distro's repo.)
>
> - Distribution packaged 'meson' and 'sphinx' are still utilized whenever
> possible as the highest preference.
>
> - Vendored versions of e.g. 'meson' are always preferred to PyPI
> versions for speed, repeatability and ensuring tarball builds work
> as-is offline.
>
> (Sphinx will not be vendored, just like it already isn't.)
>
> - Missing dependencies, when possible, are fetched and installed
> on-demand automatically to make developer environments "just work".
>
> - Works for Python 3.7 and up, on Fedora, OpenSuSE, Red Hat, CentOS,
> Alpine, Debian, Ubuntu, NetBSD, OpenBSD, and hopefully everywhere
>
> - No new dependencies (...for most platforms. Debian and NetBSD get an
> asterisk.)
>
> - The meson git submodule is unused after this series and can be removed.
>
> For reviewers, here's how the series is broken up:
>
> Patch 1 is a testing pre-req. Note that even with this patch,
> 'check-python-minreqs' and 'check-python-tox' CI jobs will both still
> fail on origin/master because this series requires 3.7+, but
> origin/master is currently still 3.6+.
>
> - python: update pylint configuration
>
> Patches 2-8 add the mkvenv script. The first patch checks in the barest
> essentials, and each subsequent patch adds a workaround or feature one
> at a time.
>
> - python: add mkvenv.py
> - mkvenv: add console script entry point generation
> - mkvenv: Add better error message for missing pyexapt module
> - mkvenv: generate console entry shims from inside the venv
> - mkvenv: work around broken pip installations on Debian 10
> - mkvenv: add nested venv workaround
> - mkvenv: add ensure subcommand
>
> Patches 9-11 modify our testing configuration to add new dependencies as
> needed.
>
> - tests/docker: add python3-venv dependency
> - tests/vm: Configure netbsd to use Python 3.10
> - tests/vm: add py310-expat to NetBSD
>
> Patch 12 changes how we package release tarballs.
>
> - scripts/make-release: download meson==0.61.5 .whl
>
> Patches 13-16 wire mkvenv into configure and tests.
>
> - configure: create a python venv unconditionally
> - configure: use 'mkvenv ensure meson' to bootstrap meson
> - configure: add --enable-pypi and --disable-pypi
> - tests: Use configure-provided pyvenv for tests
>
> Patches 17-20 delegate Sphinx bootstrapping to mkvenv. Some of these
> changes could be folded earlier in the series (like the diagnose()
> patch), but I'm keeping it separate for review for now.
>
> - configure: move --enable-docs and --disable-docs back to configure
> - mkvenv: add diagnose() method for ensure() failures
> - configure: use --diagnose option with meson ensure
> - configure: bootstrap sphinx with mkvenv
I'm not sure this last bit is working.
I uninstalled meson and python3-sphinx from my F38 host and ran
configure --target-list=x86_64-softmmu and got this:
$ ./configure --target-list=x86_64-softmmu
Using './build' as the directory for build output
python determined to be '/usr/bin/python3'
python version: Python 3.11.3
MKVENV pyvenv
Configured python as '/home/berrange/src/virt/qemu/build/pyvenv/bin/python3 -B'
MKVENV ensure meson>=0.61.5
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Location 'file:///home/berrange/src/virt/qemu/python/wheels' is ignored: it is neither a file nor a directory.
ERROR: Could not find a version that satisfies the requirement meson>=0.61.5 (from versions: none)
ERROR: No matching distribution found for meson>=0.61.5
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
MKVENV ensure sphinx>=1.6.0
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
WARNING: Skipping /usr/lib/python3.11/site-packages/virt_firmware-1.5-py3.11.egg-info due to invalid metadata entry 'name'
ERROR: Could not find a version that satisfies the requirement sphinx>=1.6.0 (from versions: none)
ERROR: No matching distribution found for sphinx>=1.6.0
*** Ouch! ***
Could not ensure availability of 'sphinx>=1.6.0':
• Python package 'sphinx' was not found nor installed.
• No local package directory was searched.
• mkvenv was configured to operate offline and did not check PyPI.
Sphinx not found/usable, disabling docs.
MKVENV ok!
It says mkvenv was configured to run offline, but I didn't do
anything. I thought the intention was for developers it would
live download from PyPI ?
Ah. So... with enable pypi being the default and with docs set to "auto", I actually fall back to not installing sphinx from pypi *by default*. It tries to locate it on your system and will enable docs if it can, but it doesn't try too hard and doesn't get upset if it fails.
(Though all of those errors and warnings sure are noisy for meaning "we couldn't find an optional package". mkvenv just doesn't distinguish between optional and required right now so it just leaves it to the caller to interpret. Any suggestions for improving this?)
Try passing --enable-docs to convince the build system you'd really definitely like docs, and it'll force the pypi access.
I can make it try PyPI in this double-default case too, I was just being very conservative about when we tried PyPI - this solution is fairly reluctant by design to do it.
If we all agree that it shouldn't be so reluctant, and anyone who doesnt want it at all should just pass --disable-pypi, I can make that change easily.
I wonder if I should make an "auto" setting for the pypi access which represents this behavior ("only if I have to"), and make "enable" more aggressive (consult pypi even for optional features.)
I was just being conservative on the first pass. Trying to strike a balance between convenience, speed, and least surprise.