[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH v5 0/3] Add common QEMU control functionality to
From: |
Stefan Hajnoczi |
Subject: |
Re: [Qemu-devel] [PATCH v5 0/3] Add common QEMU control functionality to qemu-iotests |
Date: |
Tue, 6 May 2014 16:06:28 +0200 |
User-agent: |
Mutt/1.5.21 (2010-09-15) |
On Tue, May 06, 2014 at 10:51:19AM +0200, Kevin Wolf wrote:
> Am 06.05.2014 um 10:29 hat Stefan Hajnoczi geschrieben:
> > On Mon, May 05, 2014 at 05:32:09PM +0200, Kevin Wolf wrote:
> > > Am 05.05.2014 um 17:21 hat Stefan Hajnoczi geschrieben:
> > > > On Wed, Apr 30, 2014 at 10:55:07AM -0400, Jeff Cody wrote:
> > > > > This adds some common functionality to control QEMU for qemu-iotests.
> > > > >
> > > > > Additionally, test 085 is updated to use this new functionality.
> > > > >
> > > > > Some minor fixups along the way, to clear up spaced pathname issues,
> > > > > for common.rc, test 019, and test 086.
> > > > >
> > > > >
> > > > > Jeff Cody (3):
> > > > > block: qemu-iotests - add common.qemu, for bash-controlled qemu
> > > > > tests
> > > >
> > > > Once a test launches QEMU, it soon needs to parse QMP commands or wait
> > > > for QMP events. That doesn't lend itself to the traditional
> > > > qemu-iotests shell model. That is why iotests.py exists.
> > > >
> > > > Shell script is a poor language for test cases that go beyond
> > > > pre-defined commands whose output is saved for diffing. The string
> > > > manipulation is clumsy, JSON is not supported, tricks with fifos can
> > > > easily deadlock or break when a process terminates unexpectedly, etc.
> > > >
> > > > If we go further in the direction of this patch series, we'll duplicate
> > > > existing iotests.py code and have complex shell tests that are hard to
> > > > extend. I think it's time to draw the line and convert any test cases
> > > > that need to complexity to Python.
> > > >
> > > > Why not use iotests.py?
> > >
> > > Because it's hard to use. The "compare against reference output" thing
> > > is the first thing that you lose with iotests.py, and it's the most
> > > useful feature in qemu-iotests.
> > >
> > > When a Python test case fails, you get into real debugging. When a shell
> > > script test case fails, you usually see immediately from the reference
> > > output diff what's wrong.
> > >
> > > I accept iotest.py for anything that needs to evaluate QMP return
> > > values, reluctantly, because we have nothing better. But that's it, I
> > > don't actually _like_ using it.
> >
> > I agree with what you say but we're arguing about different things.
> >
> > I'm saying:
> > * For command output diff tests, use shell.
> > * For tests that interact dynamically with running QEMU, use
> > iotests.py.
>
> That's a false dichotomy to start with. What about test cases that
> interact dynamically with running QEMU _and_ we want to use a command
> output diff?
Use print and ./check will diff the output.
> > You're saying:
> > * Command output diff tests are easy to understand and debug.
> > * Tests that interact dynamically with QEMU are harder to debug.
>
> No. I'm specifically saying that iotests.py tests are harder to debug.
> Even if they don't do much dynamic interaction.
Not true as mentioned above, use print if you want diffing.
> > Just because we like the simple shell tests better doesn't mean writing
> > complex interaction tests in shell will make them simple!
> >
> > These test cases in this patch aren't simple shell tests. Let's admit
> > they are complex and use iotests.py, which is a more appropriate tool
> > for managing a running QEMU process and interacting with it.
>
> What's wrong with the tests in this series? I find them quite easy to
> read and work with. But yes, I guess you can actually make them complex
> by using iotest.py...
I guess we'll only know by converting the tests and comparing.