[Qemu-devel] qom-test on netbsd can be very slow

[Peter Maydell]

My NetBSD build system recently seems to have taken a nosedive
in how long it takes to finish "make check". This seems to be
because qom-test (and probably other things where the test interacts
with the QEMU process) can run very slowly.

netbsdvm# for i in 1 2 3 4 5 6 7 8 9 10; do
(QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 time
tests/qom-test -p /x86_64/qom/pc-i440fx-2.0); done
/x86_64/qom/pc-i440fx-2.0: OK
        8.49 real         1.18 user         7.34 sys
/x86_64/qom/pc-i440fx-2.0: OK
       10.41 real         1.32 user         9.09 sys
/x86_64/qom/pc-i440fx-2.0: OK
        8.45 real         1.24 user         7.24 sys
/x86_64/qom/pc-i440fx-2.0: OK
        9.88 real         1.10 user         8.31 sys
/x86_64/qom/pc-i440fx-2.0: OK
       11.60 real         1.47 user         9.90 sys
/x86_64/qom/pc-i440fx-2.0: OK
       10.94 real         1.28 user         9.68 sys
/x86_64/qom/pc-i440fx-2.0: OK
       10.06 real         1.32 user         8.76 sys
/x86_64/qom/pc-i440fx-2.0: OK
       13.38 real         1.37 user        12.04 sys
/x86_64/qom/pc-i440fx-2.0: OK
       16.19 real         1.46 user        14.29 sys
/x86_64/qom/pc-i440fx-2.0: OK
        9.70 real         1.17 user         8.51 sys

Admittedly this is running in a (KVM) VM, but still, there seems
something wrong with how long each of these is taking. On Linux
each run is less than a second, so there's an order-of-magnitude
slowdown here. Further, I've occasionally seen a run take 100 seconds!

Does anybody else see this, and any ideas why it might be running slow?
I'm not very familiar with debugging on netbsd; I had a look at
ktrace output, and the QEMU process seems to sometimes spend
quite a lot of time in a poll() loop not actually doing anything.
I couldn't figure out how to get the ktrace logs to annotate
which thread was making which syscall though, so they weren't
easy to interpret. If anybody's more experienced at debugging
things in the BSDs that would also be helpful.

On OpenBSD, for comparison, each test run seems to fairly reliably
take 5 seconds give-or-take, there's much less run-to-run
variation, though it's still much slower than Linux is.
The OpenBSD and FreeBSD build VMs seem to complete more
reasonably quickly than the NetBSD one.

One thing I noticed looking at ktrace output is that we do
all our reading of QMP input and output with a read syscall
per character. I don't think that's the cause of this slowness,
though.

thanks
-- PMM