Re: [Qemu-devel] [PATCH v3 0/4] Introduce the microvm machine type

[Sergio Lopez]

Stefan Hajnoczi <address@hidden> writes:

> On Tue, Jul 02, 2019 at 02:11:02PM +0200, Sergio Lopez wrote:
>> Microvm is a machine type inspired by both NEMU and Firecracker, and
>> constructed after the machine model implemented by the latter.
>> 
>> It's main purpose is providing users a KVM-only machine type with fast
>> boot times, minimal attack surface (measured as the number of IO ports
>> and MMIO regions exposed to the Guest) and small footprint (specially
>> when combined with the ongoing QEMU modularization effort).
>> 
>> Normally, other than the device support provided by KVM itself,
>> microvm only supports virtio-mmio devices. Microvm also includes a
>> legacy mode, which adds an ISA bus with a 16550A serial port, useful
>> for being able to see the early boot kernel messages.
>> 
>> Microvm only supports booting PVH-enabled Linux ELF images. Booting
>> other PVH-enabled kernels may be possible, but due to the lack of ACPI
>> and firmware, we're relying on the command line for specifying the
>> location of the virtio-mmio transports. If there's an interest on
>> using this machine type with other kernels, we'll try to find some
>> kind of middle ground solution.
>> 
>> This is the list of the exposed IO ports and MMIO regions when running
>> in non-legacy mode:
>> 
>> address-space: memory
>>     00000000d0000000-00000000d00001ff (prio 0, i/o): virtio-mmio
>>     00000000d0000200-00000000d00003ff (prio 0, i/o): virtio-mmio
>>     00000000d0000400-00000000d00005ff (prio 0, i/o): virtio-mmio
>>     00000000d0000600-00000000d00007ff (prio 0, i/o): virtio-mmio
>>     00000000d0000800-00000000d00009ff (prio 0, i/o): virtio-mmio
>>     00000000d0000a00-00000000d0000bff (prio 0, i/o): virtio-mmio
>>     00000000d0000c00-00000000d0000dff (prio 0, i/o): virtio-mmio
>>     00000000d0000e00-00000000d0000fff (prio 0, i/o): virtio-mmio
>>     00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi
>> 
>> address-space: I/O
>>   0000000000000000-000000000000ffff (prio 0, i/o): io
>>     0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic
>>     0000000000000040-0000000000000043 (prio 0, i/o): kvm-pit
>>     000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
>>     00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic
>>     00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr
>>     00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr
>> 
>> A QEMU instance with the microvm machine type can be invoked this way:
>> 
>>  - Normal mode:
>> 
>> qemu-system-x86_64 -M microvm -m 512m -smp 2 \
>>  -kernel vmlinux -append "console=hvc0 root=/dev/vda" \
>>  -nodefaults -no-user-config \
>>  -chardev pty,id=virtiocon0,server \
>>  -device virtio-serial-device \
>>  -device virtconsole,chardev=virtiocon0 \
>>  -drive id=test,file=test.img,format=raw,if=none \
>>  -device virtio-blk-device,drive=test \
>>  -netdev tap,id=tap0,script=no,downscript=no \
>>  -device virtio-net-device,netdev=tap0
>> 
>>  - Legacy mode:
>> 
>> qemu-system-x86_64 -M microvm,legacy -m 512m -smp 2 \
>>  -kernel vmlinux -append "console=ttyS0 root=/dev/vda" \
>>  -nodefaults -no-user-config \
>>  -drive id=test,file=test.img,format=raw,if=none \
>>  -device virtio-blk-device,drive=test \
>>  -netdev tap,id=tap0,script=no,downscript=no \
>>  -device virtio-net-device,netdev=tap0 \
>>  -serial stdio
>
> Please post metrics that compare this against a minimal Q35.
>
> With qboot it was later found that SeaBIOS can achieve comparable boot
> times, so it wasn't worth maintaining qboot.
>
> Data is needed to show that microvm is really a significant improvement
> over a minimal Q35.

I've just ran some numbers using Stefano Garzarella's qemu-boot-time
scripts [1] on a server with 2xIntel Xeon Silver 4114 2.20GHz, using the
upstream QEMU (474f3938d79ab36b9231c9ad3b5a9314c2aeacde) built with
minimal features [2]. The VM boots a minimal kernel [3] without initrd,
using a kata container image as root via virtio-blk (though this isn't
really relevant, as we're just taking measurements until the kernel is
about to exec init).

To try to make the comparison as fair as possible, I've used a minimal
q35 machine with as few devices as possible. Disabling HPET and PIT at
the same time caused the kernel to get stuck on boot, so I ran two
iterations, one without HPET and the other without PIT:


-----------------
 | Q35 with HPET |
 -----------------

Command line:

./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M 
q35,smbus=off,nvdimm=off,pit=off,vmport=off,sata=off,usb=off,graphics=off 
-kernel /root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k panic=1 
root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev 
pty,id=virtiocon0,server -device virtio-serial -device 
virtconsole,chardev=virtiocon0 -drive 
id=test,file=/root/src/images/hello-rootfs.ext4,format=raw,if=none -device 
virtio-blk,drive=test

Average boot times after 10 consecutive runs:

 qemu_init_end: 77.637936
 linux_start_kernel: 117.082526 (+39.44459)
 linux_start_user: 364.629972 (+247.547446)

Memory tree:

 address-space: memory
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g @pc.ram 
0000000000000000-000000001fffffff
    0000000000000000-ffffffffffffffff (prio -1, i/o): pci
      00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
      00000000000e0000-00000000000fffff (prio 1, i/o): alias isa-bios @pc.bios 
0000000000020000-000000000003ffff
      00000000febf4000-00000000febf7fff (prio 1, i/o): virtio-pci
        00000000febf4000-00000000febf4fff (prio 0, i/o): virtio-pci-common
        00000000febf5000-00000000febf5fff (prio 0, i/o): virtio-pci-isr
        00000000febf6000-00000000febf6fff (prio 0, i/o): virtio-pci-device
        00000000febf7000-00000000febf7fff (prio 0, i/o): virtio-pci-notify
      00000000febf8000-00000000febfbfff (prio 1, i/o): virtio-pci
        00000000febf8000-00000000febf8fff (prio 0, i/o): virtio-pci-common
        00000000febf9000-00000000febf9fff (prio 0, i/o): virtio-pci-isr
        00000000febfa000-00000000febfafff (prio 0, i/o): virtio-pci-device
        00000000febfb000-00000000febfbfff (prio 0, i/o): virtio-pci-notify
      00000000febfe000-00000000febfefff (prio 1, i/o): virtio-serial-pci-msix
        00000000febfe000-00000000febfe01f (prio 0, i/o): msix-table
        00000000febfe800-00000000febfe807 (prio 0, i/o): msix-pba
      00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix
        00000000febff000-00000000febff01f (prio 0, i/o): msix-table
        00000000febff800-00000000febff807 (prio 0, i/o): msix-pba
      00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
    00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 
00000000000a0000-00000000000bffff
    00000000000c0000-00000000000c2fff (prio 1000, i/o): alias kvmvapic-rom 
@pc.ram 00000000000c0000-00000000000c2fff
    00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000c0000-00000000000c3fff [disabled]
    00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000c0000-00000000000c3fff [disabled]
    00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000c0000-00000000000c3fff
    00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pci 
00000000000c0000-00000000000c3fff [disabled]
    00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000c4000-00000000000c7fff [disabled]
    00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000c4000-00000000000c7fff [disabled]
    00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000c4000-00000000000c7fff
    00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pci 
00000000000c4000-00000000000c7fff [disabled]
    00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000c8000-00000000000cbfff [disabled]
    00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000c8000-00000000000cbfff [disabled]
    00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000c8000-00000000000cbfff
    00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pci 
00000000000c8000-00000000000cbfff [disabled]
    00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000cc000-00000000000cffff [disabled]
    00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000cc000-00000000000cffff [disabled]
    00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000cc000-00000000000cffff
    00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pci 
00000000000cc000-00000000000cffff [disabled]
    00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000d0000-00000000000d3fff [disabled]
    00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000d0000-00000000000d3fff [disabled]
    00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000d0000-00000000000d3fff
    00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pci 
00000000000d0000-00000000000d3fff [disabled]
    00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000d4000-00000000000d7fff [disabled]
    00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000d4000-00000000000d7fff [disabled]
    00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000d4000-00000000000d7fff
    00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pci 
00000000000d4000-00000000000d7fff [disabled]
    00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000d8000-00000000000dbfff [disabled]
    00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000d8000-00000000000dbfff [disabled]
    00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000d8000-00000000000dbfff
    00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pci 
00000000000d8000-00000000000dbfff [disabled]
    00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000dc000-00000000000dffff [disabled]
    00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000dc000-00000000000dffff [disabled]
    00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000dc000-00000000000dffff
    00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pci 
00000000000dc000-00000000000dffff [disabled]
    00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000e0000-00000000000e3fff [disabled]
    00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000e0000-00000000000e3fff [disabled]
    00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000e0000-00000000000e3fff
    00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pci 
00000000000e0000-00000000000e3fff [disabled]
    00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000e4000-00000000000e7fff [disabled]
    00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000e4000-00000000000e7fff [disabled]
    00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000e4000-00000000000e7fff
    00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pci 
00000000000e4000-00000000000e7fff [disabled]
    00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000e8000-00000000000ebfff
    00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000e8000-00000000000ebfff [disabled]
    00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000e8000-00000000000ebfff [disabled]
    00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pci 
00000000000e8000-00000000000ebfff [disabled]
    00000000000ec000-00000000000effff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000ec000-00000000000effff
    00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000ec000-00000000000effff [disabled]
    00000000000ec000-00000000000effff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000ec000-00000000000effff [disabled]
    00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pci 
00000000000ec000-00000000000effff [disabled]
    00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000f0000-00000000000fffff [disabled]
    00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000f0000-00000000000fffff [disabled]
    00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000f0000-00000000000fffff
    00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pci 
00000000000f0000-00000000000fffff [disabled]
    0000000020000000-0000000020000000 (prio 1, i/o): tseg-blackhole [disabled]
    00000000b0000000-00000000bfffffff (prio 0, i/o): pcie-mmcfg-mmio
    00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic
    00000000fed00000-00000000fed003ff (prio 0, i/o): hpet
    00000000fed1c000-00000000fed1ffff (prio 1, i/o): lpc-rcrb-mmio
    00000000feda0000-00000000fedbffff (prio 1, i/o): alias smram-open-high 
@pc.ram 00000000000a0000-00000000000bffff [disabled]
    00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi

 address-space: I/O
  0000000000000000-000000000000ffff (prio 0, i/o): io
    0000000000000000-0000000000000007 (prio 0, i/o): dma-chan
    0000000000000008-000000000000000f (prio 0, i/o): dma-cont
    0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic
    0000000000000060-0000000000000060 (prio 0, i/o): i8042-data
    0000000000000064-0000000000000064 (prio 0, i/o): i8042-cmd
    0000000000000070-0000000000000071 (prio 0, i/o): rtc
      0000000000000070-0000000000000070 (prio 0, i/o): rtc-index
    000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
    0000000000000080-0000000000000080 (prio 0, i/o): ioport80
    0000000000000081-0000000000000083 (prio 0, i/o): dma-page
    0000000000000087-0000000000000087 (prio 0, i/o): dma-page
    0000000000000089-000000000000008b (prio 0, i/o): dma-page
    000000000000008f-000000000000008f (prio 0, i/o): dma-page
    0000000000000092-0000000000000092 (prio 0, i/o): port92
    00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic
    00000000000000b2-00000000000000b3 (prio 0, i/o): apm-io
    00000000000000c0-00000000000000cf (prio 0, i/o): dma-chan
    00000000000000d0-00000000000000df (prio 0, i/o): dma-cont
    00000000000000f0-00000000000000f0 (prio 0, i/o): ioportF0
    00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr
    00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr
    0000000000000510-0000000000000511 (prio 0, i/o): fwcfg
    0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma
    0000000000000600-000000000000067f (prio 0, i/o): ich9-pm
      0000000000000600-0000000000000603 (prio 0, i/o): acpi-evt
      0000000000000604-0000000000000605 (prio 0, i/o): acpi-cnt
      0000000000000608-000000000000060b (prio 0, i/o): acpi-tmr
      0000000000000620-000000000000062f (prio 0, i/o): acpi-gpe0
      0000000000000630-0000000000000637 (prio 0, i/o): acpi-smi
      0000000000000660-000000000000067f (prio 0, i/o): sm-tco
    0000000000000cd8-0000000000000ce3 (prio 0, i/o): acpi-mem-hotplug
    0000000000000cf8-0000000000000cfb (prio 0, i/o): pci-conf-idx
    0000000000000cf9-0000000000000cf9 (prio 1, i/o): lpc-reset-control
    0000000000000cfc-0000000000000cff (prio 0, i/o): pci-conf-data
    000000000000c000-000000000000c07f (prio 1, i/o): virtio-pci
    000000000000c080-000000000000c0bf (prio 1, i/o): virtio-pci


 ----------------
 | Q35 with PIT |
 ----------------

Command line:

./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M 
q35,smbus=off,nvdimm=off,pit=on,vmport=off,sata=off,usb=off,graphics=off 
-no-hpet -kernel /root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k 
panic=1 root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev 
pty,id=virtiocon0,server -device virtio-serial -device 
virtconsole,chardev=virtiocon0 -drive 
id=test,file=/root/src/images/hello-rootfs.ext4,format=raw,if=none -device 
virtio-blk,drive=test

Average boot times after 10 consecutive runs:

 qemu_init_end: 77.467852
 linux_start_kernel: 116.688472 (+39.22062)
 linux_start_user: 363.033365 (+246.344893)

Memory tree:

address-space: memory
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g @pc.ram 
0000000000000000-000000001fffffff
    0000000000000000-ffffffffffffffff (prio -1, i/o): pci
      00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
      00000000000e0000-00000000000fffff (prio 1, i/o): alias isa-bios @pc.bios 
0000000000020000-000000000003ffff
      00000000febf4000-00000000febf7fff (prio 1, i/o): virtio-pci
        00000000febf4000-00000000febf4fff (prio 0, i/o): virtio-pci-common
        00000000febf5000-00000000febf5fff (prio 0, i/o): virtio-pci-isr
        00000000febf6000-00000000febf6fff (prio 0, i/o): virtio-pci-device
        00000000febf7000-00000000febf7fff (prio 0, i/o): virtio-pci-notify
      00000000febf8000-00000000febfbfff (prio 1, i/o): virtio-pci
        00000000febf8000-00000000febf8fff (prio 0, i/o): virtio-pci-common
        00000000febf9000-00000000febf9fff (prio 0, i/o): virtio-pci-isr
        00000000febfa000-00000000febfafff (prio 0, i/o): virtio-pci-device
        00000000febfb000-00000000febfbfff (prio 0, i/o): virtio-pci-notify
      00000000febfe000-00000000febfefff (prio 1, i/o): virtio-serial-pci-msix
        00000000febfe000-00000000febfe01f (prio 0, i/o): msix-table
        00000000febfe800-00000000febfe807 (prio 0, i/o): msix-pba
      00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix
        00000000febff000-00000000febff01f (prio 0, i/o): msix-table
        00000000febff800-00000000febff807 (prio 0, i/o): msix-pba
      00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
    00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 
00000000000a0000-00000000000bffff
    00000000000c0000-00000000000c2fff (prio 1000, i/o): alias kvmvapic-rom 
@pc.ram 00000000000c0000-00000000000c2fff
    00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000c0000-00000000000c3fff [disabled]
    00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000c0000-00000000000c3fff [disabled]
    00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000c0000-00000000000c3fff
    00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pci 
00000000000c0000-00000000000c3fff [disabled]
    00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000c4000-00000000000c7fff [disabled]
    00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000c4000-00000000000c7fff [disabled]
    00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000c4000-00000000000c7fff
    00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pci 
00000000000c4000-00000000000c7fff [disabled]
    00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000c8000-00000000000cbfff [disabled]
    00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000c8000-00000000000cbfff [disabled]
    00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000c8000-00000000000cbfff
    00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pci 
00000000000c8000-00000000000cbfff [disabled]
    00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000cc000-00000000000cffff [disabled]
    00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000cc000-00000000000cffff [disabled]
    00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000cc000-00000000000cffff
    00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pci 
00000000000cc000-00000000000cffff [disabled]
    00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000d0000-00000000000d3fff [disabled]
    00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000d0000-00000000000d3fff [disabled]
    00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000d0000-00000000000d3fff
    00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pci 
00000000000d0000-00000000000d3fff [disabled]
    00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000d4000-00000000000d7fff [disabled]
    00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000d4000-00000000000d7fff [disabled]
    00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000d4000-00000000000d7fff
    00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pci 
00000000000d4000-00000000000d7fff [disabled]
    00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000d8000-00000000000dbfff [disabled]
    00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000d8000-00000000000dbfff [disabled]
    00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000d8000-00000000000dbfff
    00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pci 
00000000000d8000-00000000000dbfff [disabled]
    00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000dc000-00000000000dffff [disabled]
    00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000dc000-00000000000dffff [disabled]
    00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000dc000-00000000000dffff
    00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pci 
00000000000dc000-00000000000dffff [disabled]
    00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000e0000-00000000000e3fff [disabled]
    00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000e0000-00000000000e3fff [disabled]
    00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000e0000-00000000000e3fff
    00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pci 
00000000000e0000-00000000000e3fff [disabled]
    00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000e4000-00000000000e7fff [disabled]
    00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000e4000-00000000000e7fff [disabled]
    00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000e4000-00000000000e7fff
    00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pci 
00000000000e4000-00000000000e7fff [disabled]
    00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000e8000-00000000000ebfff
    00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000e8000-00000000000ebfff [disabled]
    00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000e8000-00000000000ebfff [disabled]
    00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pci 
00000000000e8000-00000000000ebfff [disabled]
    00000000000ec000-00000000000effff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000ec000-00000000000effff
    00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000ec000-00000000000effff [disabled]
    00000000000ec000-00000000000effff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000ec000-00000000000effff [disabled]
    00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pci 
00000000000ec000-00000000000effff [disabled]
    00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-ram @pc.ram 
00000000000f0000-00000000000fffff [disabled]
    00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pc.ram 
00000000000f0000-00000000000fffff [disabled]
    00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-rom @pc.ram 
00000000000f0000-00000000000fffff
    00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pci 
00000000000f0000-00000000000fffff [disabled]
    0000000020000000-0000000020000000 (prio 1, i/o): tseg-blackhole [disabled]
    00000000b0000000-00000000bfffffff (prio 0, i/o): pcie-mmcfg-mmio
    00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic
    00000000fed1c000-00000000fed1ffff (prio 1, i/o): lpc-rcrb-mmio
    00000000feda0000-00000000fedbffff (prio 1, i/o): alias smram-open-high 
@pc.ram 00000000000a0000-00000000000bffff [disabled]
    00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi

address-space: I/O
  0000000000000000-000000000000ffff (prio 0, i/o): io
    0000000000000000-0000000000000007 (prio 0, i/o): dma-chan
    0000000000000008-000000000000000f (prio 0, i/o): dma-cont
    0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic
    0000000000000040-0000000000000043 (prio 0, i/o): kvm-pit
    0000000000000060-0000000000000060 (prio 0, i/o): i8042-data
    0000000000000061-0000000000000061 (prio 0, i/o): pcspk
    0000000000000064-0000000000000064 (prio 0, i/o): i8042-cmd
    0000000000000070-0000000000000071 (prio 0, i/o): rtc
      0000000000000070-0000000000000070 (prio 0, i/o): rtc-index
    000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
    0000000000000080-0000000000000080 (prio 0, i/o): ioport80
    0000000000000081-0000000000000083 (prio 0, i/o): dma-page
    0000000000000087-0000000000000087 (prio 0, i/o): dma-page
    0000000000000089-000000000000008b (prio 0, i/o): dma-page
    000000000000008f-000000000000008f (prio 0, i/o): dma-page
    0000000000000092-0000000000000092 (prio 0, i/o): port92
    00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic
    00000000000000b2-00000000000000b3 (prio 0, i/o): apm-io
    00000000000000c0-00000000000000cf (prio 0, i/o): dma-chan
    00000000000000d0-00000000000000df (prio 0, i/o): dma-cont
    00000000000000f0-00000000000000f0 (prio 0, i/o): ioportF0
    00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr
    00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr
    0000000000000510-0000000000000511 (prio 0, i/o): fwcfg
    0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma
    0000000000000600-000000000000067f (prio 0, i/o): ich9-pm
      0000000000000600-0000000000000603 (prio 0, i/o): acpi-evt
      0000000000000604-0000000000000605 (prio 0, i/o): acpi-cnt
      0000000000000608-000000000000060b (prio 0, i/o): acpi-tmr
      0000000000000620-000000000000062f (prio 0, i/o): acpi-gpe0
      0000000000000630-0000000000000637 (prio 0, i/o): acpi-smi
      0000000000000660-000000000000067f (prio 0, i/o): sm-tco
    0000000000000cd8-0000000000000ce3 (prio 0, i/o): acpi-mem-hotplug
    0000000000000cf8-0000000000000cfb (prio 0, i/o): pci-conf-idx
    0000000000000cf9-0000000000000cf9 (prio 1, i/o): lpc-reset-control
    0000000000000cfc-0000000000000cff (prio 0, i/o): pci-conf-data
    000000000000c000-000000000000c07f (prio 1, i/o): virtio-pci
    000000000000c080-000000000000c0bf (prio 1, i/o): virtio-pci


 -----------
 | microvm |
 -----------

Command line:

./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M microvm -kernel 
/root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k panic=1 
root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev 
pty,id=virtiocon0,server -device virtio-serial-device -device 
virtconsole,chardev=virtiocon0 -drive 
id=test,file=/root/src/images/hello-rootfs.ext4,format=raw,if=none -device 
virtio-blk-device,drive=test

Average boot times after 10 consecutive runs:

 qemu_init_end: 64.043264
 linux_start_kernel: 65.481782 (+1.438518)
 linux_start_user: 114.938353 (+49.456571)

Memory tree:

 address-space: memory
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g 
@microvm.ram 0000000000000000-000000001fffffff
    00000000d0000000-00000000d00001ff (prio 0, i/o): virtio-mmio
    00000000d0000200-00000000d00003ff (prio 0, i/o): virtio-mmio
    00000000d0000400-00000000d00005ff (prio 0, i/o): virtio-mmio
    00000000d0000600-00000000d00007ff (prio 0, i/o): virtio-mmio
    00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic
    00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi

 address-space: I/O
  0000000000000000-000000000000ffff (prio 0, i/o): io
    000000000000007e-000000000000007f (prio 0, i/o): kvmvapic


 --------------
 | Conclusion |
 --------------

The average boot time of microvm is a third of Q35's (115ms vs. 363ms),
and is smaller on all sections (QEMU initialization, firmware overhead
and kernel start-to-user).

Microvm's memory tree is also visibly simpler, significantly reducing
the exposed surface to the guest.

While we can certainly work on making Q35 smaller, I definitely think
it's better (and way safer!) having a specialized machine type for a
specific use case, than a minimal Q35 whose behavior significantly
diverges from a conventional Q35.

Sergio.

[1] https://github.com/stefano-garzarella/qemu-boot-time
[2] https://paste.fedoraproject.org/paste/YZ9Ok-dJtQrc0xxctFm-nw
[3] https://paste.fedoraproject.org/paste/sck0jfioAJdMq51HH6wkmA

signature.asc
Description: PGP signature