Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Aleksandar Markovic]

On May 23, 2019 7:27 PM, "Philippe Mathieu-Daudé" <address@hidden> wrote:
>
> On 5/23/19 7:11 PM, Aleksandar Markovic wrote:
> > On May 23, 2019 3:45 PM, "Cleber Rosa" <address@hidden> wrote:
> >>> From: "Aleksandar Markovic" <address@hidden>
> >>> On May 22, 2019 11:46 PM, "Cleber Rosa" <address@hidden> wrote:
> >>>>> From: "Eduardo Habkost" <address@hidden>
> >>>>>
> >>>>> Until we actually have a mechanism to exclude the test case on
> >>>>> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> >>>>> please don't merge patch 4/4 yet or it will break travis-ci.
> >>>>>
> >>>>> Cleber, Wainer, is it already possible to make "avocado run" skip
> >>>>> tests tagged with "slow"?
> >>>>>
> >>>>
> >>>> The mechanism exists, but we haven't tagged any test so far as slow.
> >>>>
> >>>
> >>> Cleber,
> >>>
> >>> For the test from patch 4/4, there is no dilemma - it should be in the
> >>> “slow” group, as Philippe envisioned and said, so that it is not
> > humpered
> >>> with stricter requirements for “fast” (default) group. Could you
> > explain us
> >>> how to do it, so that we can hopefully finally proceed?
> >>>
> >>
> >> Hi Aleksandar,
> >>
> >> The point is that there's no "group" definition at this point.  This is
> > the
> >> core of the discussion.
> >>
> >> I think we're close to converging to something simple and effective.
> > Please
> >> let us know what you think of the proposals given.
> >>
> >> Thanks!
> >> - Cleber.
> >>
> >
> > Cleber, hi.
> >
> > Thanks for responding.
> >
> > My views are very similar to Philippe's, but I will provide you with
more
> > details of our (mips) perspective.
> >
> > As far as black/whitelist issues that is a moot point for us - we only
want
> > to be able to have a way to tag a test within the test itself (so,
without
> > updating some common files, external lists,etc.)
> >
> > In general, we would like to have a test environment where we would be
able
> > to test what WE deem suitable for us, without feeling that we bother
you or
> > anybody else, or that we are bothered by others.
> >
> > Let me give you a little extreme example: Let's say we want a complex
test
> > that downloads components from let's say fifty internet location,
executes
> > zillion test cases, and last two days. I wouldn't like anybody to ask me
> > “Why would you that?” or tell me “You can't do this.” or say “No, we did
> > not anticipate such tests, patch rejected.” I (we, people from mips)
should
> > be able to define what I (we) need.
>
> Maybe we can use subdirectory like we do for the TCG tests (Aleksandar
> maintains tests/tcg/mips/). We should try to keep contribution upstream,
> so good idea/pattern can be reused by others.
>
> What I'd like to have with those tests is, at least:
>
> 1/ we don't need to run all the tests (but there is a set of 'quick'
> tests we can run on daily basis)
>
> 2/ maintainers can run their default tests easily (using a combination
> of Avocado tags)
>
> 3/ if a developer working on the PCI subsystem has to modify the MIPS
> subsystem (for example), he should be able to run the MIPS tests before
> sending his series.
>

Exactly! Excellent ideas and examples!

> > Having such test would be a big deal for me, not only that I could run
it
> > manually or automatically every weekend, but I could ask submitters of
> > critical changes: “Did you run this test that we have in Avocado dir?”,
> > without specifying test details, procedures, etc. All this is a BIG deal
> > for me.
> >
> > On the other hand, I agree that certain group of tests (envisioned for
> > daily or so Travis CI) should have some stricter limitations and
structure.
> > But right now I feel humpered by it, and this is counterproductive.
> >
> > So, we want freedom, responsibility and ownersheep of our tests. Please
> > give us the opportunity to get down on business and start writing tests
and
> > start testing.
> >
> > Yours,
> > Aleksandar