Re: [RFC PATCH v3 00/20] configure: create a python venv and ensure meson, sphinx

[Daniel P . Berrangé]

On Tue, Apr 25, 2023 at 02:58:53PM -0400, John Snow wrote:
> On Tue, Apr 25, 2023 at 2:10 PM Daniel P. Berrangé <berrange@redhat.com> 
> wrote:
> > How about having --enable-pypi never|auto|force with the following
> > semantics for --enable-docs + --enable-pypi
> >
> >
> >   * docs=no   - pypi never used
> >
> >   * docs=auto + pypi=never  => docs only enable if sphinx is already
> >                                installed locally, otherwise disabled
> >
> >   * docs=auto + pypi=auto  => docs enable if sphinx is already
> >                               installed locally, or can download from
> >                               pypi as fallback
> >
> 
> As long as this doesn't cause needless trouble for existing configure
> invocations baked into scripts and the like. It's a bit more
> aggressive about enabling docs than we have been in the past, but
> maybe that's not a bad thing.
> 
> >   * docs=auto + pypi=force  => always download sphinx from pypi
> >
> 
> So if you already have Sphinx, this should perform an upgrade to the
> latest version?

Essentially I meant 'force' to mean *never* use the host python
installation packages. Always install all the deps in the venv,
even if they exist in the host with sufficient version met.

> >   * docs=yes + pypi=never  => ERROR if sphinx is not already
> >                               installed locally
> >
> >   * docs=yes + pypi=auto  => docs enable if sphinx is already
> >                              installed locally, or can download from
> >                              pypi as fallback
> >
> >   * docs=yes + pypi=force  => always download sphinx from pypi
> 
> Yeah, there's room for settings like these, and the above mostly makes
> sense to me.
> 
> Paolo and I had written up a more elaborate set of flags at one point
> under the premise of integrating testing, too -- for instance,
> pre-loading the testing dependencies for iotests (local qemu package,
> qemu.qmp, etc) or for avocado tests (avocado-framework and other deps
> needed for those tests). I wanted to try and check in a "minimum
> viable" version of this patch set first close to the 8.1 tree opening
> to iron out any bugs in the core feature and then I'd work to expand
> the flags and capability of the system as differential patches.
> 
> So, I think changing --enable-pypi too much is beyond the scope of the
> current patchset, but it's on my mind for what will follow almost
> immediately after it. With that in mind, we can brainstorm a little
> here and throw some darts at the wall:

Yep, I don't consider it a pre-requisite for this series.

> 
> The version of flags we originally came up with was this:
> 
> --python=... # runtime used to create venv
> --enable-pip-groups=testing,devel,avocado,meson,sphinx
> --enable-pip=now  # install all python deps now
> --enable-pip=on-demand  # install qemu.git/meson/sphinx, delay the rest
> --enable-pip=no    # offline
> --{enable,disable}-isolated-venv # let venv use system/distro if disable

This feels like a bit of overkill to me, and would create a hell
of a lot of combinations to test if you expand the matrix of
options.

> I may have pulled us in a slightly different direction with the
> version of the patches I actually sent here today, but the key ideas
> here were:
> 
> - The ability to opt into or out of different package groups at
> configure-time. meson is always required, docs/sphinx is optional, and
> the other three groups are testing related. (testing: iotests and
> qemu.qmp, devel: python self-tests, avocado: avocado tests.)

I think this is especially overkill. While you can probably come up
with some scenarios where you could use this fine grained selection,
I don't think it would be very compelling.

> - The ability to decide when packages get installed to the venv;
> either at configure time ("now") or as-needed ("on-demand") or
> functionally never ("no")

The distinction between now vs on-demand is really just about
avoiding the overhead of downloading bits that you might not
end up using. eg not downloading avocado unless you will be
running 'make check-avocado'. That's saving a few 10s or 100s
of KB of download, but doesn't feel like it'd make a noticable
difference in the overall QEMU build time.

THe 'now' and 'no' options look sufficient (corresponding to
the 'auto' and 'never' options I suggested above)

> - The ability to enable or disable venv isolation (turning on or off
> system packages).

Corresponds to the 'force' option I suggested above.

> 
> I think I abandoned the idea of configuring the venv isolation and
> have it set to "always off". We may revisit this later, but I think
> for this series it's fine to have glossed over it.
> I also skipped over the idea of having package installation being
> "now", "on-demand" or "no" -- this patch-set more or less is using a
> hybrid of "now" and "on-demand" where meson and sphinx are "now" but
> avocado is "on-demand". Unifying that discrepancy can occur in
> subsequent patches as it closely resembles what we had in 8.0 and
> earlier.

I guess the hybrid of 'now' and 'on-demand' could equally well
map to what I suggested by 'auto'. 


> > So eg distros could use pypi=never, devs would use pypi=auto mostly,
> > while CI might use pypi=force to test specific versions indepenant
> > of distros ?
> >
> 
> Sounds reasonable in general, I think, but I have some questions about
> what 'force' does, exactly.

Turning of system packages in the venv, and always installing everything
from pip.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|