[Qemu-devel] Slow kernel/initrd loading via fw_cfg; Was Re: Hack integrating SeaBios / LinuxBoot option rom with QEMU trace backends

[Daniel P. Berrange]

On Mon, Oct 10, 2011 at 09:01:52PM +0200, Alexander Graf wrote:
> 
> On 10.10.2011, at 20:53, Anthony Liguori wrote:
> 
> > On 10/10/2011 12:08 PM, Daniel P. Berrange wrote:
> >> With the attached patches applied to QEMU and SeaBios, the attached
> >> systemtap script can be used to debug timings in QEMU startup.
> >> 
> >> For example, one execution of QEMU produced the following log:
> >> 
> >>   $ stap qemu-timing.stp
> >>   0.000 Start
> >>   0.036 Run
> >>   0.038 BIOS post
> >>   0.180 BIOS int 19
> >>   0.181 BIOS boot OS
> >>   0.181 LinuxBoot copy kernel
> >>   1.371 LinuxBoot copy initrd
> > 
> > Yeah, there was a thread a bit ago about the performance
> > of the interface to read the kernel/initrd.  I think at it
> > was using single byte access instructions and there were
> > patches to use string accessors instead?  I can't remember
> > where that threaded ended up.

There was initially a huge performance problem, which was
fixed during the course of the thread, getting to the current
state where it still takes a few seconds to load large blobs.
The thread continued with many proposals & counter proposals
but nothing further really came out of it.

   https://lists.gnu.org/archive/html/qemu-devel/2010-08/msg00133.html

One core point to take away though, is that -kernel/-initrd is
*not* just for ad-hoc testing by qemu/kernel developers. It is
critical functionality widely used by users of QEMU in production
scenarios and performance of it does matter, in some cases, alot.

> IIRC we're already using string accessors, but are still
> slow. Richard had a nice patch cooked up to basically have
> the fw_cfg interface be able to DMA its data to the guest.
> I like the idea. Avi did not.

That's here:

  https://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01037.html

> And yes, bad -kernel performance does hurt in some workloads. A lot.

Let me recap the 3 usage scenarios I believe are most common:

 - Most Linux distro installs done with libvirt + virt-manager/virt-install
   are done by directly booting the distro's PXE kernel/initrd files.
   The kernel images are typically < 5 MB, while the initrd images may
   be as large as 150 MB.  Both are compressed already. An uncompressed
   initrd image would be more like 300 MB,  so these are avoided for
   obvious reasons.

   Performance is not really an issue, within reason, since the overall
   distro installation time will easily dominate, but loading should
   still be measured in seconds, not minutes.

   The reason for using a kernel/initrd instead of a bootable ISO is
   to be able to set kernel command line arguments for the installer.

 - libguestfs directly boots its appliance using the regular host's
   kernel image and a custom built initrd image. The initrd does
   not contain the entire appliance, just enough to boot up and
   dynamically read files in from the host OS on demand. This is
   a so called "supermin appliance".

   The kernel is < 5 MB, while the initrd is approx 100MB. The initrd
   image is used uncompressed, because decompression time needs to be
   eliminated from bootup.  Performance is very critical for libguestfs.
   100's of milliseconds really do make a big difference for it.

   The reason for using a kernel/initrd instead of bootable ISO is to
   avoid the time required to actually build the ISO, and to avoid
   having more disks visible in the guest, which could confuse apps
   using libguestfs which enumerate disks.

 - Application sandbox, directly boots the regular host's kernel and
   a custom initrd image. The initrd does not contain any files except
   for the 9p kernel modules and a custom init binary, which mounts
   the guest root FS from a 9p filesystem export.

   The kernel is < 5 MB, while the initrd is approx 700 KB compressed,
   or 1.4 MB compressed. Performance for the sandbox is even more
   critical than for libguestfs. Even 10's of milliseconds make a
   difference here. The commands being run in the sandbox can be
   very short lived processes, executed reasonably frequently. The
   goal is to have end-to-end runtime overhead of < 2 seconds. This
   includes libvirt guest startup, qemu startup/shutdown, bios time,
   option ROM time, kernel boot & shutdown time.

   The reason for using a kerenl/initrd instead of a bootable ISO,
   is that building an ISO requires time itself, and we need to be
   able to easily pass kernel boot arguments via -append.


I'm focusing on the last use case, and if the phase of the moon
is correct, I can currently executed a sandbox command with a total
overhead of 3.5 seconds (if using a compressed initrd) of which
the QEMU execution time is 2.5 seconds.

Of this, 1.4 seconds is the time required by LinuxBoot to copy the
kernel+initrd. If I used an uncompressed initrd, which I really want
to, to avoid decompression overhead, this increases to ~1.7 seconds.
So the LinuxBoot ROM is ~60% of total QEMU execution time, or 40%
of total sandbox execution overhead.

For comparison I also did a test building a bootable ISO using ISOLinux.
This required 700 ms for the boot time, which is appoximately 1/2 the
time reqiured for direct kernel/initrd boot. But you have to then add
on time required to build the ISO on every boot, to add custom kernel
command line args. So while ISO is faster than LinuxBoot currently
there is still non-negligable overhead here that I want to avoid.

For further comparison I tested with Rich Jones' patches which add a
DMA-like inteface to fw_cfg. With this the time spent in the LinuxBoot
option ROM was as close to zero as matters.

So obviously, my preference is for -kernel/-initrd to be made very fast
using the DMA-like patches, or any other patches which could achieve
similarly high performance for -kernel/-initd.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|