Re: Migrating custom qemu.org infrastructure to GitLab

[Daniel P . Berrangé]

On Wed, Jul 08, 2020 at 01:17:11PM +0200, Paolo Bonzini wrote:
> On 08/07/20 12:53, Daniel P. Berrangé wrote:
> > Consider if qemu-web.git is hosted on gitlab, using GitLab CI to
> > generate the static site content, and GitLab pages to host the
> > website. If a user wants to contribute to qemu-web, they can fork
> > the repo and be confident that the CI jobs in their fork are going
> > to work in the exact same way as in the master repo. They can use
> > GitLab Pages to browse the generated content for their fork to
> > see exactly how it will look.
> > 
> > This eliminates the big pain point in qemu-web contribution. Many
> > times I was tripped up with the problem of qemu-web.git requiring
> > a version of Jekyll that is incompatible with the version that
> > ships on my distro.
> 
> I would have no issue with using a pull request workflow for qemu-web.
> The lack of "git range-diff" functionality for gitlab is an absolute
> showstopper for using it in QEMU, though.

FWIW, I tried to avoid mentioning merge requests, as I didn't want
to derail the discussion & believe use of gitlab for infrastructure
is worthwhile even if you never use merge requests.  I do tend to
think that merge requests would be a reasonable thing to use for
qemu-web in particular.

> >> GitLab's Continuous Integration (CI) system provides a powerful way to
> >> perform actions defined in yaml files in qemu.git. This includes
> >> running scripts, builds, publishing build artifacts, etc. We have
> >> already begun using it for automated builds and tests:
> >> https://gitlab.com/qemu-project/qemu/-/blob/master/.gitlab-ci.yml
> > 
> > The CI integration has probably been the single best thing about
> > libvirt's switch to GitLab.
> 
> How do you handle non-x86 platforms?  Has there been any progress in
> gitlab runner support for s390 and PPC?

We don't do any native non-x86 stuff right now.  We do all non-x86
builds using cross-compilers. Obviously that means we don't get any
testing coverage, only build coverage.

We did update our lcitool to be able to build VMs to serve as gitlab
runners, principally to provide us FreeBSD runners. After doing all
the work though, we discovered a way to integrate Cirrus CI into
GitLab CI, using a shim called "cirrus-run".  So now we have macOS
and FreeBSD jobs in GitLab CI, that use a shim to call out to Cirrus
CI todo the actual work:

>From this pipeline:

  https://gitlab.com/libvirt/libvirt/-/pipelines/164291392

These two jobs actually ran on Cirrus CI:

  https://gitlab.com/libvirt/libvirt/-/jobs/628816093
  https://gitlab.com/libvirt/libvirt/-/jobs/628816086

We don't get a nice build log output in gitlab but that's just a
limitation of the current cirrus-run impl - we could easily enhance
it to pull the buld log out of Cirrus CI, so it is immediately
visible in GitLab logs.

IOW, the cirrus-run hack shows that even if you can't get a native runner
integration for GitLab CI, you can always use a shim to call out to pretty
much any external system you desire. So that provides a backup plan for
non-x86 CI support.

> > The Documentation/Platforms content arguably should be part of
> > the main qemu.git docs.
> > 
> > Many of the feature pages are probably better as part of the formal
> > QEMU documentation too.
> 
> Yes, definitely; but someone has to do the work.  At least the obsolete
> features are clearly marked as so.

Even if no one has time todo the work now, if we can spend a bit of time
to categorize current wiki pages and document a desired "plan", then we
can publicise it as a "bite sized task" and potentially motivate future
contributors to do some of the work as a useful way in to the QEMU world.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|