Re: [PATCH v2 6/7] CI: Stop building docs on centos8

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Feb 14, 2023 at 08:40:20AM +0100, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>> [...]
>> 
>> > We don't have to drop python 3.6. It is a choice because
>> > of a desire to be able to use some shiny new python
>> > features without caring about back compat.
>> 
>> I read this on Friday, and decided to let it sit until after the
>> weekend.  Well, it's now Tuesday, and to be frank, it's still as
>> offensively flippant as it was on Friday.  It shows either ignorance of
>> or cavalier disregard for the sheer amount of work some of us have had
>> to put into keeping old versions of Python viable.
>> 
>> The latter would be quite unlike you, so it must be the former.
>
> I'm sorry, I don't mean it to be offensive. I'm genuinely not seeing
> from the descriptions in the series what the functional benefits are
> from dropping 3.6.

Apology accepted, and point well taken.

>> John has sunk *man-months* into keeping old versions of Python viable.
>> I've put in a lot less myself, but still enough to be painfully aware of
>> it.  I figure Cleber and Beraldo are somewhere in between
>> 
>> Insinuating John's proposal is motivated by "a desire to be able to use
>> some shiny new python features without caring about back compat"
>> disrespects all this work.
>
> I'm writing my comments based on what is described in the cover letter
> as the motivations for the change:
>
> [quote]
> The motivation for this series is that Python 3.6 was EOL at the end of
> 2021; upstream tools are beginning to drop support for it, including
> setuptools, pylint, mypy, etc. As time goes by, it becomes more
> difficult to support and test against the full range of Python versions
> that QEMU supports. The closer we get to Python 3.12, the harder it will
> be to cover that full spread of versions.
> [/quote]
>
> this is all about new/eol versions of software upstream, and I don't
> think that's a justification. QEMU explicitly aims to use distro provided
> versions and upstream EOL status is not relevant in that context. Even
> if using "pip" to install it is possible to limit yourself to upstream
> releases which still support 3.6.
>
> There is the separate issue of Meson dropping python 3.6 which motivates
> Paolo's series. Again though, we don't have to increase our minimum meson
> version, because meson is working today. It is our choice to to increase
> it to use latest available meson features. At some point we can decide
> what we have is good enough and we don't have to keep chasing the latest
> features. Maybe we're not there yet, but we should think about when that
> would be.
>
> [quote]
> The qemu.qmp library and the avocado testing framework both have
> motivations for dropping 3.6 support, but are committed to not doing so
> until QEMU drops support.
> [/quote]
>
> I suspect that this is more of a driver for the drop of 3.6, but I
> don't see any details.
>
> IOW overall justification come across as wanting to use new features,
> and follow upstream EOL, without any real detail of what we're going
> to gain from a functional POV.

I think what we gain is there: "As time goes by, it becomes more
difficult to support and test against the full range of Python versions
that QEMU supports. The closer we get to Python 3.12, the harder it will
be to cover that full spread of versions."  In other words, the gain is
"we make something that's hard and getting harder easier".

What isn't in the cover letter, only in commit message 6+7/7, i.e. this
patch and the next one, is what we pay for it: basically nothing (next
patch's commit message) except for the issue of Sphinx in CentOS 8 (this
patch's commit message).  Putting at least a summary it in the cover
letter would have been better.

John did explain this again and in more detail in reply to Peter's "This
confuses me.  We work fine with Python 3.6 today."  Quote:

    That won't last - Many tools such as mypy, pylint and flake8 which I use to
    manage our python codebase have been dropping support for 3.6 and I've had
    to implement an increasing number of workarounds to help keep it possible
    to test 3.6 via CI while also ensuring our newest platforms work as dev
    environments.

    Our testing matrix for Python is novel and thorough enough that it's
    revealed  several bugs in other downstream Python distributions for Debian
    and Fedora, and dozens of bugs for the linters themselves.

    I'm concerned that when 3.7 is EOL'd in just a few months that the support
    and testing gap is going to get even uglier.

    [...]

    The argument I'm making is:

    - CentOS 8 is a supported build platform
    - All platforms *do* have a Python modern enough to allow us to drop 3.6
    - CentOS's repo version of sphinx is hardcoded to use the older 3.6, though
    - You expressed a preference to me in the past to NOT use a pip installed
    version of sphinx in preference to the system version in "configure"
    - It's still possible to build docs on CentOS 8 after this patchset, you
    just need a pip version.
    - We've used the justification that our build platform promise does not
    necessarily extend to docs and tests in the past.
    - So just skip docs building for CentOS 8, only in the CI.

    If you believe docs in CI for CentOS 8 is a hard deal breaker, then I want
    to go back to discussing the possibility of using sphinx versions from pip.

You even quoted most of it in your message :)

The second paragraph is evidence we've been doing something basically
nobody else does.

>> We should have a sober discussion on which versions of Python to work
>> with, and the tradeoffs involved.  But before I engage in that, I insist
>> on resetting the frame: no, this is not about shiny, new Python
>> features.  It is about stopping the bleeding.  It is about reducing what
>> feels more and more like bullshit work to me, so we can actually
>> accomplish stuff that matters.
>
> Every applications developer chooses an arbitrary cut off points for
> minimum software versions, depending on their particular needs. With
> our support policy we tried to express a reasonable tradeoff between
> keeping back compat, and being able to adopt new features.
>
> Obviously that tradeoff is not currently looking acceptable on the
> python side, but it is not clear why that is ?
>
> Can someone simply explain what we wil see as the benefit from dropping
> 3.6 / adopting 3.7 as the baseline ? 

Let me start with the costs, then move on to the benefits.

Every single supported build platform already provides Python >= 3.7,
and all the Python tools we use continue to work with it, with one
exception: Sphinx on CentOS 8.

Solutions for this issue proposed so far:

1. Do nothing.  To build documentation on CentOS 8, you have to pip
   install a working version of Sphinx, which is easy enough.  As long
   as we don't in CI, we have to disable building docs there (this
   patch).

2. Have the build system pip install it for you.  A bit of a paradigm
   shift: so far our build system doesn't install anything.

3. Split the Python dependency for Sphinx off the general one.  On
   CentOS 8, we'd use 3.6 to run Sphinx, and 3.7+ for everything else.
   On all other systems, we'd use the same 3.7+ for everything.

So the cost is our choice of "you have to pip install Sphinx to build
docs on CentOS 8" / "build system pip installs Sphinx" / "special Python
dependency for Sphinx".  Feels quite modest to me.

The main benefit is we reduce the bleeding caused by trying to support
Python all the way back to 3.6 when basically nobody else does.  Linters
in particular.  How much of a benefit is this?  I'd like you all to
trust John it's plenty big enough to justify the (modest!) cost.  He's
been doing much of the bleeding, after all.

New features provided by 3.7 will be nice to have (especially for
typing), but that's not why we're doing this.