Re: Migrating custom qemu.org infrastructure to GitLab

[Daniel P . Berrangé]

On Wed, Jul 08, 2020 at 10:52:38AM +0100, Stefan Hajnoczi wrote:
> Dear QEMU community,
> QEMU currently has a static website, wiki, git repo hosting, and
> special-purpose cronjobs/containers running in VMs. There is currently
> no system administrator looking after our infrastructure so the most
> urgent tasks fall onto me, the remainder are ignored/postponed. The
> current situation exposes qemu.org to the risk of downtime and
> security issues.

I would further say that maintaining infrastructure is not a good use
of our contributors' time. It is better to have our contributors spend
time delivering features that are beneficial to our users, rather than
playing at sysadmin.

> Another limitation is that each piece of infrastructure is managed
> separately and one-time contributors cannot easily propose changes
> because they do not have access. It would be much better to use our
> existing code review process so that anyone can make changes to
> infrastructure by sending a patch.

When I was thinking about this for libvirt I came to better understand
the appeal of the gitforge for project infrastructure. I've written
about it in detail for libvirt:

   https://www.redhat.com/archives/libvir-list/2020-March/msg00009.html

The distributed SCM model of git democratized access to source control
features across contributors. Drive-by contributors have the same level
of SCM features as the main project contributors, instead of being
second class citizens as they are with a non-distributed SCM.

With the hosted Git forges, this concept covers all aspects of project
infrastructure. If your project is using GitLab for all its infra
needs, then when a user forks the repo they get access to all the
same features as the main project. The fork is no longer a second
class citizen lacking access to certain parts of the infra. I find
this is especially compelling in the area of CI, containers and web
hosting.

Consider if qemu-web.git is hosted on gitlab, using GitLab CI to
generate the static site content, and GitLab pages to host the
website. If a user wants to contribute to qemu-web, they can fork
the repo and be confident that the CI jobs in their fork are going
to work in the exact same way as in the master repo. They can use
GitLab Pages to browse the generated content for their fork to
see exactly how it will look.

This eliminates the big pain point in qemu-web contribution. Many
times I was tripped up with the problem of qemu-web.git requiring
a version of Jekyll that is incompatible with the version that
ships on my distro.

> GitLab's Continuous Integration (CI) system provides a powerful way to
> perform actions defined in yaml files in qemu.git. This includes
> running scripts, builds, publishing build artifacts, etc. We have
> already begun using it for automated builds and tests:
> https://gitlab.com/qemu-project/qemu/-/blob/master/.gitlab-ci.yml

The CI integration has probably been the single best thing about
libvirt's switch to GitLab.

> GitLab also offers git repo hosting, wikis, issue tracking, and other
> features. It is possible to log in using GitHub, Google, or Twitter
> single sign-on if you do not want to create another account. As more
> open source projects use GitLab it becomes easier for one-time
> contributors who will already be familiar with the tools from other
> projects.

> Here is a full list of GitLab's features:
> https://about.gitlab.com/features/
> 
> GitLab offers the gold/ultimate tier for free to open source projects:
> https://about.gitlab.com/solutions/open-source/
> 
> GitLab itself is open source and can be self-hosted if we decide to
> leave in the future.

NB self-hosting would reduce features compared to gitlab.com unless
you paid for the enterprise edition. You'd also have to bring your
own infra for running CI jobs which quickly gets very expensive
as freedesktop.org admins have learnt the hard way. The CI resources
provided by gitlab.com are incredibly generous.

> With this in mind I propose moving qemu.org infrastructure to GitLab
> incrementally. This needs to be done carefully to avoid disruption and
> only where GitLab meets the requirements. The QEMU project will
> continue to have access to cloud hosting for running custom
> infrastructure or adding runners to GitLab CI to improve CI
> performance.
> 
> The following infrastructure components can be considered for GitLab 
> migration:
> 
> 1. qemu-web.git static site generation. GitLab CI/CD can build the
> static website on each qemu-web.git commit and publish the HTML
> artifacts.

This is an easy win. it should be especially simple for qemu, compared to
libvirt, since qemu.org already used separate sub-domains for other content
like git & downloads.

> 2. wiki.qemu.org is a MediaWiki instance. Account creation is a hurdle
> to one-time or new contributors. It is unclear whether GitLab's wiki
> is expressive enough for a lossless conversion of the existing QEMU
> wiki. Any volunteers interested in evaluating the wiki migration would
> be appreciated.

Yeah, this is a potentially big piece of work. We didn't finish this
in libvirt either. Looking at the libvirt mediawiki though, I decided
not todo a straight export/import of all content.

A bunch of content was horribly out of date and needed deleting as
it was irrelevant to anyone, or outright misleading wrt current state
of the world.

A lot of content on the wiki really only existed in the wiki, because
contributing to our libvirt.org website was harder than it needed to
be. We switched the main website over to use RST instead of HTML so
authoring content is much closer to what you'd experience with the
wiki markup. The wiki became a burden for contributors as due to
spam attacks we can't allow anonymous access, and need manual account
creation as with QEMU wiki.

Every page on the main website now has a link "edit this page"
that takes you straight to the source file in the gitlab.com UI,
where you can interactively edit the text in your fork and then
submit a merge request with the result.

The wiki also suffered from the content not having any level of review.
While the intentions of many contributors were good, the results often
left alot to be desired. 

So overall we decided that while GitLab does offer a wiki, we are not
intending to use it. Instead we're going to aim to turn the subset of
wiki content that is genuinely useful into pages on the main website.

IOW, I'd encourage whoever tackles the wiki to look at the content
with a critical eye and decide whether it really belongs in the wiki
at all. Considering:

  https://wiki.qemu.org/index.php?title=Special:AllPages

The Planning / Changes pages for each release are probably reasonable
as wiki pages.

The Interships/ProjectIdeas pages would possibly be better off as
GitLab Issue tickets, tagged with a "interships" label. That way
when a project starts you can have the issue to track the work
right from the start.

The Documentation/Platforms content arguably should be part of
the main qemu.git docs.

Many of the feature pages are probably better as part of the formal
QEMU documentation too.

> 3. Git repo hosting is a core feature of GitLab and we already have a
> qemu.git mirror. Hosting the repos on GitLab reduces the need for
> qemu.org ssh access.

With libvirt we had long used both gitlab.com/libvirt and github.com/libvirt
as read-only mirrors of the master libvirt.org/git/ repos.  When we formally
adopted gitlab as primary infrastructure, we simply flipped the mirroring
processes around. So now libvirt.org/git is a read-only mirror, and gitlab
is the master.  Gitlab has built-in support to push to github.com as a
mirror which is useful.

So turning git.qemu.org into a readonly mirror would ease the transition
to avoid breaking any scripts / workflows relying on that URL in the short
term.

> 4. The QEMU release process can be moved to CI/CD so that publishing
> stable releases and release candidates is less dependent on one
> committer's machine or scripts.
> 
> 5. Issue tracking. Launchpad more or less works, but the login always
> bothers me. If we move git repo hosting then it makes sense to do
> issue tracking on GitLab too.

The big thing that always bothers me about launchpad is how easy it
is to get confused between issues for QEMU upstream and issues for
legacy releases in Ubuntu distro. This is the curse of using a distro
issue tracker - libvirt had the same problem with its use of Red Hat
bugzilla.

Integrated issue tracking alongside the primary git repo makes a lot
of sense. Once you declare gitlab.com the primary repo, contributors
will immediately look to gitlab for the issue tracking too whether
you tell them to or not. 

The gitlab issue tracker is fairly simple especially compared to the
complexity horror of red hat bugzilla, so it was a massive win for
libvirt to switch away from bugzilla for issues. Probably not such
a big difference compared to launchpad in terms of complexity. The
little things though like magic git commit messages to close issues
are nice though.

There is a question of what todo with existing bugs in launchpad.

Essentially three choices

 1. Move all the open bugs to gitlab
 2. Move some relevant bugs to gitlab, but close outdated ones
 3. Leave existing launchpad bugs but don't allow new ones filed

So far for libvirt we did option 3 there, with a little bit of
option 2.

> There is a snowball effect where the experience is improved the more
> GitLab features we use, so I hope that most of these migrations will
> be possible.
> 
> Next steps:
>  * If you have an interest in one or more of these infrastructure
> components, please join the discussion.
>  * If there are no volunteers for an infrastructure component I'll
> slowly work my way through evaluating GitLab and propose migrations

The experiance of moving infra to gitlab.com has been quite positive
for libvirt, and me in particular as the single point of failure for
much of libvirt's old infrastructure. I think it will be beneficial
for QEMU too, for many of the reasons I mention above and in the
email link I quoted.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|