[Qemu-devel] converting build system to Meson?

[Paolo Bonzini]

Hi all,

lately I have been thinking of converting the QEMU build system to
Meson.  Meson is a relatively new build system that can replace
Autotools or hand-written Makefiles such as QEMU; as a die-hard
Autotools fan, I must say that Meson is by far better than anything else
that has ever tried to replace Autotools, and actually has the potential
to do so.

Advantages of Meson that directly matter for QEMU include:

- build definitions in a very readable and user friendly DSL, which
supports looping and conditions.

- ability to introspect the build definitions so that you can find out
what is built without building it (the DSL is not Turing complete and
most objects in it are immutable, so it cannot be abused that much :))

- support for a non-recursive build from per-subdirectory input (similar
to Makefile.objs)

- ease of distributing a full copy of Meson to support distros that ship
an older version (no dependencies apart from Python 3.5).  At 40000
lines of Python, Meson is relatively small.


Unlike Autoconf, meson.build files include both feature detection and
build rules.  Also unlike Autoconf, Meson must be present on the system
of whoever builds QEMU, which is why the last bullet above matters.
However, it is possible to keep a configure script, if only to provide
backwards command line compatibility and support a bundled copy of Meson.

Of course a full conversion of QEMU's build system, including the 7k
lines configure script is not easy.  In addition, even though Meson
supports extension modules that are written in Python, those are really
used only for namespacing purposes as the extensions need to live in the
Meson tree and not in the QEMU tree.  Therefore I have tried to identify
the parts of QEMU's Makefiles that could complicate the transition, and
see in advance if they can be addressed:

- --enable-modules's usage of special linker options -u SYMBOL.  For
this I initially wrote patches to Meson that would make everything
automatic, however they were rejected.  Instead, the Meson maintainer
prodded me to find a way to implement the behavior in QEMU's build
system without having to extend Meson, and we managed to find one.  This
was a good exercise because it showed that, despite being very
"opinionated", Meson manages to be pretty flexible too.

- ease of use for test logs and the ability to cut and paste test
invocations from the logs to the command line.  For this I have started
"probing" how the Meson developers feel about this kind of change[1],
and intend to follow up until the meson test driver is comparable in
usability to QEMU's "make check",

- ease of converting Makefile.objs files.  The Makefile.objs files are
very nice to change for simple modifications, and any replacement should
have the same feature.  This will require a Meson extension which I have
proposed already[2], where adding a source file will look like

    obj.add(['CONFIG_VIRTIO'], files('virtio.c'),
            if_false: files('virtio-stub.c'))
    common_obj.add(['CONFIG_SDL'], files('sdl.c'],
                   dependencies: sdl)

This is a little more verbose than Makefile.objs, but should be okay
assuming the extension is accepted.  Rejection of my extension would be
a blocker though, because QEMU is quite special in how it builds dozens
of large binaries, all *from the same sources* (unlike e.g. gstreamer
which has >100 build targets but they are all independent plugins).

Rules for generators like trace-tool or QAPI would becomes quite a bit
more verbose.  Hopefully this is offset by increased clarity if we don't
rely anymore on stuff like pattern rules and instead have foreach loops.

- unintended performance issues.  I wouldn't be surprised if QEMU
stressed Meson more than many other projects did.  However, this should
be mitigated by the fact that source selection is done in Python rather
than in Meson's DSL.  Everything else is O(#binaries) or O(#sources),
but not O(#sources*#binaries).

- ability to use the currently in progress Kconfig declarations for
dependencies.  Meson developers have expressed that they would accept an
extension module that loads Kconfig-like declarations from a file; this
would also be useful to start using Meson only as a Make replacement
while keeping the current configure script[4], since we could slurp
config-host.mak and config-target.mak from Meson.  However, this is only
half of the story, since we also have to compute the Kconfig
dependencies.  For this it should be possible to take the existing
minikconf and change it to produce Meson's DSL.  For example

   config FOO
     default y if BAR
     select BAZ

would become

   if config.get('BAR') == 'y' and not config.has('FOO') then
     config.set('FOO', 'y')
   endif
   if config.get('FOO') == 'y' then
     if config.has('BAZ') and config.get('BAZ') != 'y' then
       error('FOO selects BAZ, but BAZ is disabled')
     endif
     config.set('BAZ', 'y')
   endif

I have not tried doing this, but it seems feasible.

- Meson generates a build.ninja file rather than a Makefile, which also
complicates a bit interoperability with the current system.  This is
perhaps minor, since in most cases the change is just s/make/ninja/ but
it may matter for day-to-day development.  For this I have prototyped a
converter (written in Python) from ninja back to Make, so that we could
even just "include" the Meson-generated build system into Make and keep
a (progressively more) minimal Makefile veneer around it.  I benchmarked
Make a bit and its slowness of QEMU on a do-nothing build is simply due
to the complexity of the unnest-vars macro machinery; Make could parse a
synthetic 90000-target Makefile in 2 seconds (QEMU has 9000 object
files), and the ninja->Make converter is careful to avoid traps where
Make has quadratic complexity.


To be clear, this is not something I am going to work anytime soon.
Still in my opinion it's a worthwhile exercise to think about it.  As I
will see in the next few weeks what the Meson maintainers' reaction will
be to some of the proposed extensions, I wanted to gauge the reactions
of you all as well. :)

Thanks,

Paolo

[1] https://github.com/mesonbuild/meson/pull/5025
[2] https://github.com/mesonbuild/meson/pull/5028
[3] https://github.com/mesonbuild/meson/pull/1617