Re: [Qemu-devel] [PATCH v3 0/4] Introduce the microvm machine type

[Stefan Hajnoczi]

On Fri, Jul 19, 2019 at 2:48 PM Sergio Lopez <address@hidden> wrote:
> Stefan Hajnoczi <address@hidden> writes:
> > On Thu, Jul 18, 2019 at 05:21:46PM +0200, Sergio Lopez wrote:
> >>
> >> Stefan Hajnoczi <address@hidden> writes:
> >>
> >> > On Tue, Jul 02, 2019 at 02:11:02PM +0200, Sergio Lopez wrote:
> >>  --------------
> >>  | Conclusion |
> >>  --------------
> >>
> >> The average boot time of microvm is a third of Q35's (115ms vs. 363ms),
> >> and is smaller on all sections (QEMU initialization, firmware overhead
> >> and kernel start-to-user).
> >>
> >> Microvm's memory tree is also visibly simpler, significantly reducing
> >> the exposed surface to the guest.
> >>
> >> While we can certainly work on making Q35 smaller, I definitely think
> >> it's better (and way safer!) having a specialized machine type for a
> >> specific use case, than a minimal Q35 whose behavior significantly
> >> diverges from a conventional Q35.
> >
> > Interesting, so not a 10x difference!  This might be amenable to
> > optimization.
> >
> > My concern with microvm is that it's so limited that few users will be
> > able to benefit from the reduced attack surface and faster startup time.
> > I think it's worth investigating slimming down Q35 further first.
> >
> > In terms of startup time the first step would be profiling Q35 kernel
> > startup to find out what's taking so long (firmware initialization, PCI
> > probing, etc)?
>
> Some findings:
>
>  1. Exposing the TSC_DEADLINE CPU flag (i.e. using "-cpu host") saves a
>     whooping 120ms by avoiding the APIC timer calibration at
>     arch/x86/kernel/apic/apic.c:calibrate_APIC_clock
>
> Average boot time with "-cpu host"
>  qemu_init_end: 76.408950
>  linux_start_kernel: 116.166142 (+39.757192)
>  linux_start_user: 242.954347 (+126.788205)
>
> Average boot time with default "cpu"
>  qemu_init_end: 77.467852
>  linux_start_kernel: 116.688472 (+39.22062)
>  linux_start_user: 363.033365 (+246.344893)

\o/

>  2. The other 130ms are a direct result of PCI and ACPI presence (tested
>     with a kernel without support for those elements). I'll publish some
>     detailed numbers next week.

Here are the Kata Containers kernel parameters:

var kernelParams = []Param{
        {"tsc", "reliable"},
        {"no_timer_check", ""},
        {"rcupdate.rcu_expedited", "1"},
        {"i8042.direct", "1"},
        {"i8042.dumbkbd", "1"},
        {"i8042.nopnp", "1"},
        {"i8042.noaux", "1"},
        {"noreplace-smp", ""},
        {"reboot", "k"},
        {"console", "hvc0"},
        {"console", "hvc1"},
        {"iommu", "off"},
        {"cryptomgr.notests", ""},
        {"net.ifnames", "0"},
        {"pci", "lastbus=0"},
}

pci lastbus=0 looks interesting and so do some of the others :).

Stefan