Re: [Qemu-devel] [RFC] image-fuzzer: Trivial test runner

[Stefan Hajnoczi]

On Thu, Jun 05, 2014 at 12:49:36PM +0400, M.Kustova wrote:
> On Wed, Jun 4, 2014 at 6:26 PM, Stefan Hajnoczi <address@hidden> wrote:
> > On Sat, May 31, 2014 at 01:56:46PM +0400, Maria Kustova wrote:
> >> +    def execute(self, q_args, seed, size=8*512):
> >> +        """ Execute a test.
> >> +
> >> +        The method creates a test image, runs 'qemu_img' and analyzes its 
> >> exit
> >> +        status. If the application was killed by a signal, the test is 
> >> marked
> >> +        as failed.
> >> +        """
> >> +        os.chdir(self.current_dir)
> >> +        seed = qcow2.create_image('test_image.qcow2', seed, size)
> >
> > The qcow2 module is missing from this patch series.
> 
> As far as the qcow2 module was just a stub and an image format should
> not be hardcoded, it will be sent as a patch as soon as it gets some
> functionality implemented.

Patch emails cannot contain dangling references.  It makes code review
hard when you don't actually have all the code.  And it means the
commits are not bisectable (if you try the code somewhere in the middle
of the commit history it won't work and this prevents git-bisect(1)
usage).

> >> +        multilog("Seed: %s\nCommand: %s\nTest directory: %s\n"\
> >> +                 %(seed, " ".join(q_args), self.current_dir),\
> >> +                 sys.stdout, self.log, self.parent_log)
> >
> > It will probably be useful to dial back the logging for test cases that
> > pass.
> 
> This information is kept for heisenbugs, that can be reproduced only
> in the sequence of several tests. It's significant if all passed tests
> data would be removed.

I think the amount of shared or accumulated state between runes is so
minimal that this type of failure will not happen.

Each test run has a uniquely generated image.  The qemu-img/qemu-io
processes are terminated and started from scratch.

Where is the shared state that allows "independent" runs to affect each
other?  If there is some, I think we need to fix that instead of trying
to log the whole sequence of success cases.

> >> +    work_dir = args[0]
> >> +    # run_log created in 'main', because multiple tests are expected to \
> >> +    # log in it
> >> +    # TODO: Make unique run_log names on every run (for one test per run
> >> +    # this functionality is omitted in favor of usability)
> >
> > I don't understand this TODO comment.  Here's roughly what I expected
> > when I read the --help output:
> >
> >   work_dir/
> >       run.log
> >       failure-01/
> >           core
> >           input.img
> >           cmdline
> >           seed
> >
> 
> You can do several test runs one by one in the same working directory.
> Each run will have a unique name and all tests of the run will write
> their output to the log with this unique name.
> Something like:
> 
> work_dir/
>        run-01.log
>        run-02.log
>        failure-02_01/
>         core
>         input.img
>         test-02_01.log

I don't understand how this works or why it's useful.  If you want to
separate test runs, use different work dirs.

> For now outputs of all runs are appended to the same file with name
> 'run.log' while there is only one test per run.

Yes, that's what I see in the code and I think more than that isn't
necessary.