Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Stefan Hajnoczi]

On Thu, Apr 26, 2018 at 10:57:55AM -0300, Eduardo Habkost wrote:
> (Starting a new thread, for more visibility)
> 
> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> devices with all machines, not only with "none")
> 
> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
> > Thomas Huth <address@hidden> writes:
> > > On 17.04.2018 14:12, Markus Armbruster wrote:
> > >> Thomas Huth <address@hidden> writes:
> > >> 
> > >>> Many device introspection crashes only happen if you are using a
> > >>> certain machine, e.g.:
> > >>>
> > >>> $ ppc-softmmu/qemu-system-ppc -S -M ref405ep,accel=qtest -qmp stdio
> > >>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 11, "major": 2},
> > >>>  "package": "build-all"}, "capabilities": []}}
> > >>> { 'execute': 'qmp_capabilities' }
> > >>> {"return": {}}
> > >>> { 'execute': 'device-list-properties',
> > >>>   'arguments': {'typename': 'macio-newworld'}}
> > >>> Unexpected error in qemu_chr_fe_init() at chardev/char-fe.c:222:
> > >>> Device 'serial0' is in use
> > >>> Aborted (core dumped)
> > >>>
> > >>> To be able to catch these problems, let's extend the device-introspect
> > >>> test to check the devices on all machine types. Since this is a rather
> > >>> slow operation, the test is only run in "SPEED=slow" mode.
> > >> 
> > >> If the device works with one machine type, it has a decent chance to
> > >> work with others, too.  Thus, testing each device with every machine
> > >> type is overkill.  I appreciate having overkill as an option :)
> > >> 
> > >> What I'd like to see for a quick "make check" is testing each device
> > >> once.  That should flush out most bugs.  
> > >
> > > That's already done with the "none" machine.
> > 
> > I was too terse.  We test each device with -machine none for every
> > target.  Fine if that's quick enough.  If not, we might want to reduce
> > redundancy there.
> > 
> > Actually, a worse offender in the "waste everybody's time via redunancy"
> > department could be qom-test.
> > 
> > > Anyway, do you think my patch here is useful and has a chance of getting
> > > included? I.e. shall I re-spin this as a non-RFC patch? Or shall we
> > > rather wait for Eduardo's python-based tests to get included into the
> > > repository?
> > 
> > I don't mind having make check SPEED=slow run more extensive tests.
> > Assuming we actually run them at least once in a while, which seems
> > doubtful.
> 
> We probably don't do that, but we really must be running a more
> extensive (and slower) test set at least once before every
> release.
> 
> Maybe some people are running SPEED=slow tests, or even more
> extensive test suites like avocado-vt once in a while, but we
> need to know who is running them, and when.
> 
> Today, the only test set I know people really run and would
> surely block a release is "make check [SPEED=quick]".
> 
> So, for anybody that runs automated QEMU tests once in a while,
> can we know:
> 
> * What test cases are you running?  Where can we get more
>   information about the tests you run?
> * When do you run them?  What triggers a new test run?

I manually run qemu-iotests on release candidates:

 $ (cd tests/qemu-iotests && ./check && ./check -qcow2)

The goal is to identify test failures that still need to be addressed
before the release is made.

I think Kevin Wolf and John Snow have also been running qemu-iotests to
eliminate regressions during the freeze.

Stefan

signature.asc
Description: PGP signature