Re: [Qemu-devel] converting build system to Meson?

[Marc-André Lureau]

Hi Paolo

On Wed, Mar 6, 2019 at 7:12 PM Paolo Bonzini <address@hidden> wrote:
>
> Hi all,
>
> lately I have been thinking of converting the QEMU build system to
> Meson.  Meson is a relatively new build system that can replace
> Autotools or hand-written Makefiles such as QEMU; as a die-hard
> Autotools fan, I must say that Meson is by far better than anything else
> that has ever tried to replace Autotools, and actually has the potential
> to do so.
>
> Advantages of Meson that directly matter for QEMU include:
>
> - build definitions in a very readable and user friendly DSL, which
> supports looping and conditions.
>
> - ability to introspect the build definitions so that you can find out
> what is built without building it (the DSL is not Turing complete and
> most objects in it are immutable, so it cannot be abused that much :))
>
> - support for a non-recursive build from per-subdirectory input (similar
> to Makefile.objs)
>
> - ease of distributing a full copy of Meson to support distros that ship
> an older version (no dependencies apart from Python 3.5).  At 40000
> lines of Python, Meson is relatively small.
>
>
> Unlike Autoconf, meson.build files include both feature detection and
> build rules.  Also unlike Autoconf, Meson must be present on the system
> of whoever builds QEMU, which is why the last bullet above matters.
> However, it is possible to keep a configure script, if only to provide
> backwards command line compatibility and support a bundled copy of Meson.

Yes, we will probably have a transition period where both build-sys
have to co-exist (same story with libslirp, which I proposed as
meson-only first)

>
> Of course a full conversion of QEMU's build system, including the 7k
> lines configure script is not easy.  In addition, even though Meson

I don't think a complete transition in one go is feasible. Could we
start with some basic things?

For example, we could quite easily start converting tools (qemu-img
etc), some tests...

> supports extension modules that are written in Python, those are really
> used only for namespacing purposes as the extensions need to live in the

I didn't know, I suppose meson doesn't want to support an internal API?

> Meson tree and not in the QEMU tree.  Therefore I have tried to identify
> the parts of QEMU's Makefiles that could complicate the transition, and
> see in advance if they can be addressed:
>
> - --enable-modules's usage of special linker options -u SYMBOL.  For
> this I initially wrote patches to Meson that would make everything
> automatic, however they were rejected.  Instead, the Meson maintainer
> prodded me to find a way to implement the behavior in QEMU's build
> system without having to extend Meson, and we managed to find one.  This
> was a good exercise because it showed that, despite being very
> "opinionated", Meson manages to be pretty flexible too.
>
> - ease of use for test logs and the ability to cut and paste test
> invocations from the logs to the command line.  For this I have started
> "probing" how the Meson developers feel about this kind of change[1],
> and intend to follow up until the meson test driver is comparable in
> usability to QEMU's "make check",
>
> - ease of converting Makefile.objs files.  The Makefile.objs files are
> very nice to change for simple modifications, and any replacement should
> have the same feature.  This will require a Meson extension which I have
> proposed already[2], where adding a source file will look like
>
>     obj.add(['CONFIG_VIRTIO'], files('virtio.c'),
>             if_false: files('virtio-stub.c'))
>     common_obj.add(['CONFIG_SDL'], files('sdl.c'],
>                    dependencies: sdl)
>
> This is a little more verbose than Makefile.objs, but should be okay
> assuming the extension is accepted.  Rejection of my extension would be
> a blocker though, because QEMU is quite special in how it builds dozens
> of large binaries, all *from the same sources* (unlike e.g. gstreamer
> which has >100 build targets but they are all independent plugins).
>
> Rules for generators like trace-tool or QAPI would becomes quite a bit
> more verbose.  Hopefully this is offset by increased clarity if we don't
> rely anymore on stuff like pattern rules and instead have foreach loops.
>
> - unintended performance issues.  I wouldn't be surprised if QEMU
> stressed Meson more than many other projects did.  However, this should
> be mitigated by the fact that source selection is done in Python rather
> than in Meson's DSL.  Everything else is O(#binaries) or O(#sources),
> but not O(#sources*#binaries).
>
> - ability to use the currently in progress Kconfig declarations for
> dependencies.  Meson developers have expressed that they would accept an
> extension module that loads Kconfig-like declarations from a file; this
> would also be useful to start using Meson only as a Make replacement
> while keeping the current configure script[4], since we could slurp

(you meant [3])

> config-host.mak and config-target.mak from Meson.  However, this is only
> half of the story, since we also have to compute the Kconfig
> dependencies.  For this it should be possible to take the existing
> minikconf and change it to produce Meson's DSL.  For example
>
>    config FOO
>      default y if BAR
>      select BAZ
>
> would become
>
>    if config.get('BAR') == 'y' and not config.has('FOO') then
>      config.set('FOO', 'y')
>    endif
>    if config.get('FOO') == 'y' then
>      if config.has('BAZ') and config.get('BAZ') != 'y' then
>        error('FOO selects BAZ, but BAZ is disabled')
>      endif
>      config.set('BAZ', 'y')
>    endif
>
> I have not tried doing this, but it seems feasible.

I am confused, I thought minikconf was generating a .mak (I am not
very familiar with it, I am waiting for it to land!)

>
> - Meson generates a build.ninja file rather than a Makefile, which also
> complicates a bit interoperability with the current system.  This is
> perhaps minor, since in most cases the change is just s/make/ninja/ but
> it may matter for day-to-day development.  For this I have prototyped a
> converter (written in Python) from ninja back to Make, so that we could

wow, interesting, and no link? :)

> even just "include" the Meson-generated build system into Make and keep
> a (progressively more) minimal Makefile veneer around it.  I benchmarked
> Make a bit and its slowness of QEMU on a do-nothing build is simply due
> to the complexity of the unnest-vars macro machinery; Make could parse a
> synthetic 90000-target Makefile in 2 seconds (QEMU has 9000 object
> files), and the ninja->Make converter is careful to avoid traps where
> Make has quadratic complexity.
>
>
> To be clear, this is not something I am going to work anytime soon.
> Still in my opinion it's a worthwhile exercise to think about it.  As I
> will see in the next few weeks what the Meson maintainers' reaction will
> be to some of the proposed extensions, I wanted to gauge the reactions
> of you all as well. :)

I like meson better than anything else. It is still young, one can
find quickly limitations & documentation could be better (if you
compare it with make/autoconf/automake reference manual for ex). But I
also believe it has the potential to improve the qemu build-sys and
make it more readable & more modern (meson has some IDE integration
stuff, multiple targets, and various niceties).

Do you have a WIP branch somewhere we could contribute to?

Alternatively, could we start small upstream, with the tools for
example, and iterate there? (unless one bikeshed on our favorite
build-sys - then let's work on it in a seperate branch)