[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

From: Cleber Rosa
Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU
Date: Wed, 28 Dec 2011 02:01:22 -0300
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20111115 Thunderbird/8.0

On 12/27/2011 11:37 PM, Anthony Liguori wrote:
On 12/27/2011 04:35 PM, Cleber Rosa wrote:
On 12/26/2011 08:00 PM, Dor Laor wrote:
On 12/26/2011 05:12 PM, Anthony Liguori wrote:
Hi Dor,

Merry Christmas Anthony,

On 12/25/2011 09:19 AM, Dor Laor wrote:
On 12/19/2011 07:13 PM, Anthony Liguori wrote:

Well, I'm still not convinced that a new standalone package should
handle these
cases instead of kvm autotest. I'll be happy to integrate the tests to
autotest anyway and the more the merrier but imho it's a duplicate.

I'm sure kvm autotest could be taught to do exactly what qemu-test is
doing. But why does kvm autotest have to do everything? I doubt there
would be much code reuse.

I think it's not a bad thing to have multiple test suites when there
isn't considerable overlap.

I think the main goal of qemu-tests (may be implicit) is to be quick and simple.

qemu-test doesn't have a main goal. My goal is to improve QEMU's quality. qemu-test is just a tool to help achieve that goal.

Maybe I've used the wrong wording. I got the feeling that, besides testing qemu the way you need it, keeping qemu-test simple was really important. If not, I'd imagine you'd try to re-use something that already exists (yes, kvm autotest).

That is indeed great, but if one thinks that's all we'll ever going to need,
that thought is pretty naive.

I don't know who "we" is, but I can tell you that qemu-test is exactly what *I* need. Consider that I spent a good portion of every single day testing QEMU with either my own or other people's patches, making that job easier and more automated is fairly important to me.

"We" is everyone that somehow contributes to QEMU, that is, the QEMU community. If you're only concerned about what *you* need, then you're on the right track. If, besides that, you feel it'd be nice to *try to* concentrate our efforts, then we're all on the same track.

I'm sharing it because I suspect that a lot of other developers have a similar need.

And it may be true that there's room for both test
suites... or that, as busy developers, we're refusing to deal with the added complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I
believe on the later.

One example: kvm-autotest has a complex configuration file format with a steep learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have to be tweaked if somehow the kernel detects the first ethernet interface as em1
(such as recent Fedora systems do). Simple, but not scalable.

I can tell by this comment that you don't actually understand how qemu-test works. Please take a look at it before jumping to conclusions about whether it should or shouldn't be part of kvm-autotest.

Hint: qemu-test always uses the same kernel because it builds it as part of the test suite. The behavior of how a nic test will never change unless someone explicitly changes the kernel.

I can tell you did not get my point: I'm giving some reasons of why current kvm autotest is somehow complex, and how qemu-tests gets away and keeps it simple.

BTW, I did not jumped at any conclusion. I'm just trying to enrich the discussion, which may end up proving that there's no other way to have what qemu-tests does.

1) It builds a custom kernel and initramfs based on busybox. This is
fairly important to ensure that we can run tests with no device

This can be done easily w/ autotest too.

The Python requirement inside the guest is true *if* we want to run regular autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and this accounts for very very little of kvm autotest usage. All other tests interact with the monitor directly and with the guest via ssh/telnet/serial.

qemu-test does not require any specific hardware to be used in the guest which lets it test a wider variety of scenarios in QEMU. So you cannot assume there is ssh/telnet/serial available.

I really thought we could rely on, at least, a serial connection. If not, then indeed the current kvm autotest approach is not compatible with that test environment. That is not to say that kvm autotest couldn't incorporate the qemu-tests approach/functionality.

BTW, I just don't like the idea of having lots of functionalities/tests implemented on two test suites for a single piece of software, unless proven that there's no way around it. To me, this is the whole point of this discussion.

So, I see no reason for not using a more expressive language,

I seriously doubt you can build a useful initramfs that contains python without doing something crazy like livecd-tools does....

You're right. Again, I was thinking we could rely at least on a serial connection. Can we not?

Actually, kvm-autotest has various layers of abstraction in how QEMU
ends up being launched. As you mention below, those layers are there to
allow for things like using libvirt.

Indeed the qemu command line parameters gets generated depending on many
configuration parameters. It'd be *really* simple to add a configuration
parameters that overwrites the qemu command with an static one.

But if you're a QEMU developer, you want to have as much control of the command line as possible. For instance, one of the tests in qemu-test makes sure to test invocations without -device as this triggers a different code path (there was a recent regression in this too). You can't just add arguments to reproduce this behavior.

It goes beyond that, since it also related to the monitor interface as well.

That's desirable when you're doing "virt testing", but not so desirably
when you're trying to write specific unit tests against QEMU.

True, one may not need it at all but it's nice that a test for
migration/stress/hotplug will be tested directly w/ qemu and libvirt w/ the
same effort.

5) The tests execute very quickly, can be run stand alone, and do not
require root privileges

ditto for kvm auotest. It's possible to configure it w/o root too
which is not a
huge issue.

When I say, "run quickly", I mean, they execute very quickly.

/me too

$ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64

real 0m4.385s
user 0m1.460s
sys 0m1.860s

That's impressive but it's more of a function of the guest being used - if instead of running a full Fedora install, you'll chose your busybox image w/
-kernel/initrd you'll get a similar result.

I also think so. Maybe kvm-autotest would take a little more time because of the different approach we take when communicating with the guest, but I bet it'd be

I don't see any reason why everything needs to live in kvm-autotest... but if you really feel that way, please provide patches that demonstrate how this would work.

If it's technically viable, I think that having it as part of kvm autotest, shows that the project is more cohesive, attracts more contributions, and makes better use of our efforts.

We could argue indefinitely about how things could work, it's much better to compare how things actually do work :-)

Sure, that's also what I suggested when I mentioned it'd be a "nice exercise for all of us".

I agree autotest is not perfect but it likes to be such.
If you wish, you can challenge Lucas and Cleber w/ these type of requirements
and we'll all improve as a result.

Yes, I believe it'd be a nice exercise for all of us.

The only thing I ask is that we bear at least with some of the complexity that
kvm-autotest inherently holds...

I think there's something of a knee jerk reaction here. The existence of qemu-test does not take anything away from kvm-autotest. It's just another tool in our arsenal to achieve our joint goal of making QEMU (and KVM) higher quality.

You're right, It does not take anything away from kvm autotest today. But suppose we can prove that kvm autotest can indeed absorve all of qemu-tests functionalities, it'd be itself a reason for doing so. It'd avoid finding ourselves with two evolved test tools that do some of the same things, but are separate implementations.

autotest is made to invoke third party tests so the two tools can co-exist in a complimentary way.

That's a no brainier and everyone so far agrees on that. The delicate issue is whether qemu-tests functionality could live in kvm autotest in a nice way.


Anthony Liguori

the last requests we had was to get rid of all
the complexity, while retaining all the other nice characteristics. Pretty hard,
so I think we failed, or maybe only half-succeeded at it.



Anthony Liguori

6) They are random by nature with the ability to fix the seed in order
to be used in git-bisect.

I think Gerd had been looking at doing something similar with a custom

I've tried to consider other architectures and had hoped that we could commit the vmlinuz and initramfs into git so that it was easy to test
other architectures without having a full build environment.
Unfortunately, busybox doesn't link statically with glibc and I can't see an obvious way to commit binaries while respecting the GPL since we
need to pull glibc into the initramfs.

I know buildroot exists specifically to deal with this but in my
experience, buildroot is very unreliable and extremely heavy weight
since it rebuilds gcc multiple times in order to bootstrap a ulibc

Anyway, the code is available at:


See the README for instructions on how to use it.


Anthony Liguori

reply via email to

[Prev in Thread] Current Thread [Next in Thread]