Re: [Qemu-devel] [PATCH v5 0/3] Add common QEMU control functionality to qemu-iotests

[Kevin Wolf]

Am 06.05.2014 um 10:29 hat Stefan Hajnoczi geschrieben:
> On Mon, May 05, 2014 at 05:32:09PM +0200, Kevin Wolf wrote:
> > Am 05.05.2014 um 17:21 hat Stefan Hajnoczi geschrieben:
> > > On Wed, Apr 30, 2014 at 10:55:07AM -0400, Jeff Cody wrote:
> > > > This adds some common functionality to control QEMU for qemu-iotests.
> > > > 
> > > > Additionally, test 085 is updated to use this new functionality.
> > > > 
> > > > Some minor fixups along the way, to clear up spaced pathname issues, 
> > > > for common.rc, test 019, and test 086.
> > > > 
> > > > 
> > > > Jeff Cody (3):
> > > >   block: qemu-iotests - add common.qemu, for bash-controlled qemu tests
> > > 
> > > Once a test launches QEMU, it soon needs to parse QMP commands or wait
> > > for QMP events.  That doesn't lend itself to the traditional
> > > qemu-iotests shell model.  That is why iotests.py exists.
> > > 
> > > Shell script is a poor language for test cases that go beyond
> > > pre-defined commands whose output is saved for diffing.  The string
> > > manipulation is clumsy, JSON is not supported, tricks with fifos can
> > > easily deadlock or break when a process terminates unexpectedly, etc.
> > > 
> > > If we go further in the direction of this patch series, we'll duplicate
> > > existing iotests.py code and have complex shell tests that are hard to
> > > extend.  I think it's time to draw the line and convert any test cases
> > > that need to complexity to Python.
> > > 
> > > Why not use iotests.py?
> > 
> > Because it's hard to use. The "compare against reference output" thing
> > is the first thing that you lose with iotests.py, and it's the most
> > useful feature in qemu-iotests.
> > 
> > When a Python test case fails, you get into real debugging. When a shell
> > script test case fails, you usually see immediately from the reference
> > output diff what's wrong.
> > 
> > I accept iotest.py for anything that needs to evaluate QMP return
> > values, reluctantly, because we have nothing better. But that's it, I
> > don't actually _like_ using it.
> 
> I agree with what you say but we're arguing about different things.
> 
> I'm saying:
>  * For command output diff tests, use shell.
>  * For tests that interact dynamically with running QEMU, use
>    iotests.py.

That's a false dichotomy to start with. What about test cases that
interact dynamically with running QEMU _and_ we want to use a command
output diff?

But, what this series is aiming at is not really "dynamic" interaction
at all. It is mostly just sending monitor commands and comparing the
return value to reference output. The only time it looks at the return
value is when it needs to wait for completion of something; and grep is
just fine for something like that.

Just have a look at the test cases that run qemu today. These shell
scripts turn out to be really simple. And as you can see in this series,
even things like migration are easy to do.

> You're saying:
>  * Command output diff tests are easy to understand and debug.
>  * Tests that interact dynamically with QEMU are harder to debug.

No. I'm specifically saying that iotests.py tests are harder to debug.
Even if they don't do much dynamic interaction.

> Just because we like the simple shell tests better doesn't mean writing
> complex interaction tests in shell will make them simple!
> 
> These test cases in this patch aren't simple shell tests.  Let's admit
> they are complex and use iotests.py, which is a more appropriate tool
> for managing a running QEMU process and interacting with it.

What's wrong with the tests in this series? I find them quite easy to
read and work with. But yes, I guess you can actually make them complex
by using iotest.py...

Kevin