Re: [RFC PATCH] docs: build-platforms: refine requirements on Python build dependencies

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> Historically, the critical dependency for both building and running
> QEMU has been the distro packages.  Because QEMU is written in C and C's
> package management has been tied to distros (at least if you do not want
> to bundle libraries with the binary, otherwise I suppose you could use
> something like conda or wrapdb), C dependencies of QEMU would target the
> version that is shipped in relatively old but still commonly used distros.
>
> For non-C libraries, however, the situation is different, as these
> languages have their own package management tool (cpan, pip, gem, npm,
> and so on).  For some of these languages, the amount of dependencies
> for even a simple program can easily balloon to the point that many
> distros have given up on packaging non-C code.  For this reason, it has
> become increasingly normal for developers to download dependencies into
> a self-contained local environment, instead of relying on distro packages.
>
> Fortunately, this affects QEMU only at build time, as qemu.git does
> not package non-C artifacts such as the qemu.qmp package; but still,
> as we make more use of Python, we experience a clash between a support
> policy that is written for the C world, and dependencies (both direct
> and indirect) that increasingly do not care for the distro versions
> and are quick at moving past Python runtime versions that are declared
> end-of-life.
>
> For example, Python 3.6 has been EOL'd since December 2021 and Meson 0.62
> (released the following March) already dropped support for it.  Yet,
> Python 3.6 is the default version of the Python runtime for RHEL/CentOS
> 8 and SLE 15, respectively the penultimate and the most recent version
> of two distros that QEMU would like to support.  (It is also the version
> used by Ubuntu 18.04, but QEMU stopped supporting it in April 2022).
>
> Fortunately, these long-term support distros do include newer versions of
> the Python runtime.  However, these more recent runtimes only come with
> a very small subset of the Python packages that the distro includes.
> Because most dependencies are optional tests (avocado, mypy, flake8)
> and Meson is bundled with QEMU, the most noticeably missing package is
> Sphinx (and the readthedocs theme).
>
> Assuming QEMU would like to move forward with the deprecation of
> Python 3.6 (for which there are some good reasons: completing the
> configure->meson switch, which requires Meson 0.63, or making qapidoc

I think you mean "the QAPI generator".

> fully typed which requires newer versions of mypy and also Python due
> to PEP563), there are four possibilities:
>
> * we change the support policy and stop supporting CentOS 8 and SLE 15;
>   not a good idea since CentOS 8 is not an unreasonable distro for us to
>   want to continue to support
>
> * we keep supporting Python 3.6 until CentOS 8 and SLE 15 stop being
>   supported.  This is a possibility---but we may want to revise the support
>   policy anyway because SLE 16 has not even been released, so this would
>   mean delaying those desirable reasons for perhaps three years;
>
> * we support Python 3.6 just for building documentation, i.e. we are
>   careful not to use Python 3.7+ features in our Sphinx extensions but are
>   free to use them elsewhere.  Besides being more complicated to understand
>   for developers, this is difficult to apply because qapidoc runs at

Suggest "some QAPI generator code runs".

>   sphinx-build time, and it is one of the areas which would benefit from
>   a newer version of the runtime;
>
> * we only support Python 3.7+, which means CentOS 8 CI and users
>   have to either install Sphinx from pip or disable documentation.
>
> This proposed update to the support policy chooses the last of these
> possibilities.  It does by modifying two aspects of the support policy:
>
> * it introduces different support periods for *native* vs. *non-native*
>   dependencies.  Non-native dependencies are currently Python ones only,
>   and for simplicity the policy only mentions Python; however, the concept
>   generalizes to other languages with a well-known upstream package
>   manager, that users of older distributions can fetch dependencies from;
>
> * it limits the support period for non-native dependencies to a fixed
>   amount of 4 years.  This is intended to be close to the Python 5-year
>   lifecycle while accounting for the time between a distro's feature freeze
>   and the day it's released.  This limit applies to all distro versions,
>   not just the previous one, in order to cater for the delay of SLE 16.
>
> The 4 year cutoff in practice means that QEMU will be able to drop Python
> 3.6 support for QEMU 7.1 (RHEL8 becomes 4 year old next May, while SLE

We released 7.1 last August, do you mean 8.0 or 8.1?

> is already over the threshold).
>
> Note that all "non-native" packages are currently build dependencies.
> If in the future some non-native packages became runtime dependencies for
> parts of QEMU, it would still be possible to choose any of the first
> three possibilities for them.
>
> Another possible future change is to the way that these dependencies
> have to be obtained by the person building QEMU.  Right now they have to
> run pip before the build; it may be desirable for configure to set up a
> virtual environment and download them in the same way that it populates
> git submodules.  Just like with submodules, this would make things
> easier for people that can afford accessing the network in their build
> environment; the option to populate the build environment manually with
> pip would remain for people whose build machines lack network access.
> The change to the support policy neither requires nor forbids this change.
>
> [Thanks to Daniel P. Berrangé, Peter Maydell and others for discussions
>  that were copied or summarized in the above commit message]
>
> Cc: Markus Armbruster <armbru@redhat.com>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Cc: John Snow <jsnow@redhat.com>
> Cc: Kevin Wolf <kwolf@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  docs/about/build-platforms.rst | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
>
> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> index 1c1e7b9e11c3..e1ea09789107 100644
> --- a/docs/about/build-platforms.rst
> +++ b/docs/about/build-platforms.rst
> @@ -86,6 +86,25 @@ respective ports repository, while NetBSD will use the 
> pkgsrc repository.
>  For macOS, `Homebrew`_ will be used, although `MacPorts`_ is expected to 
> carry
>  similar versions.
>  
> +Python build dependencies
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The minimum supported version of Python is currently 3.6.
> +
> +Distributions with long-term support often provide multiple
> +versions of the Python runtime.  QEMU aims to support the default
> +Python runtime for 4 years after the initial release of a new version.

Just to be crystal clear: new version of what?

> +Afterwards, you may have to point QEMU to a newer version of the Python
> +runtime using the ``--python`` command line option of the ``configure``
> +script.
> +
> +Some of QEMU's build dependencies are written in Python and available
> +through the Python Package Index (PyPI).  QEMU aims to be compatible
> +with the versions packaged by common Linux distributions for the first
> +4 years after the major release of the distribution.  After 4 years,
> +you may have to use ``pip`` to install some of these build dependencies.
> +
> +
>  Windows
>  -------