Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Aleksandar Markovic]

On May 23, 2019 3:45 PM, "Cleber Rosa" <address@hidden> wrote:
>
>
>
> ----- Original Message -----
> > From: "Aleksandar Markovic" <address@hidden>
> > To: "Cleber Rosa" <address@hidden>
> > Cc: "Wainer dos Santos Moschetta" <address@hidden>, "Aleksandar
Markovic" <address@hidden>,
> > address@hidden, "Aleksandar Rikalo" <address@hidden>,
"Eduardo Habkost" <address@hidden>,
> > "Aurelien Jarno" <address@hidden>, "Philippe Mathieu-Daudé" <
address@hidden>
> > Sent: Wednesday, May 22, 2019 6:43:54 PM
> > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> >
> > On May 22, 2019 11:46 PM, "Cleber Rosa" <address@hidden> wrote:
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Eduardo Habkost" <address@hidden>
> > > > To: "Philippe Mathieu-Daudé" <address@hidden>
> > > > Cc: address@hidden, "Aleksandar Rikalo" <address@hidden
>,
> > "Aleksandar Markovic"
> > > > <address@hidden>, "Aleksandar Markovic" <
> > address@hidden>, "Cleber Rosa" <address@hidden>,
> > > > "Aurelien Jarno" <address@hidden>, "Wainer dos Santos
Moschetta" <
> > address@hidden>
> > > > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > > >
> > > > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé
wrote:
> > > > > Hi,
> > > > >
> > > > > It was a rainy week-end here, so I invested it to automatize some
> > > > > of my MIPS tests.
> > > > >
> > > > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > > > meant to run on a CI system but rather on a workstation previous
> > > > > to post a pull request.
> > > > > It can surely be improved, but it is a good starting point.
> > > >
> > > > Until we actually have a mechanism to exclude the test case on
> > > > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > > > please don't merge patch 4/4 yet or it will break travis-ci.
> > > >
> > > > Cleber, Wainer, is it already possible to make "avocado run" skip
> > > > tests tagged with "slow"?
> > > >
> > >
> > > The mechanism exists, but we haven't tagged any test so far as slow.
> > >
> >
> > Cleber,
> >
> > For the test from patch 4/4, there is no dilemma - it should be in the
> > “slow” group, as Philippe envisioned and said, so that it is not
humpered
> > with stricter requirements for “fast” (default) group. Could you
explain us
> > how to do it, so that we can hopefully finally proceed?
> >
>
> Hi Aleksandar,
>
> The point is that there's no "group" definition at this point.  This is
the
> core of the discussion.
>
> I think we're close to converging to something simple and effective.
Please
> let us know what you think of the proposals given.
>
> Thanks!
> - Cleber.
>

Cleber, hi.

Thanks for responding.

My views are very similar to Philippe's, but I will provide you with more
details of our (mips) perspective.

As far as black/whitelist issues that is a moot point for us - we only want
to be able to have a way to tag a test within the test itself (so, without
updating some common files, external lists,etc.)

In general, we would like to have a test environment where we would be able
to test what WE deem suitable for us, without feeling that we bother you or
anybody else, or that we are bothered by others.

Let me give you a little extreme example: Let's say we want a complex test
that downloads components from let's say fifty internet location, executes
zillion test cases, and last two days. I wouldn't like anybody to ask me
“Why would you that?” or tell me “You can't do this.” or say “No, we did
not anticipate such tests, patch rejected.” I (we, people from mips) should
be able to define what I (we) need.

Having such test would be a big deal for me, not only that I could run it
manually or automatically every weekend, but I could ask submitters of
critical changes: “Did you run this test that we have in Avocado dir?”,
without specifying test details, procedures, etc. All this is a BIG deal
for me.

On the other hand, I agree that certain group of tests (envisioned for
daily or so Travis CI) should have some stricter limitations and structure.
But right now I feel humpered by it, and this is counterproductive.

So, we want freedom, responsibility and ownersheep of our tests. Please
give us the opportunity to get down on business and start writing tests and
start testing.

Yours,
Aleksandar

> > Gratefully,
> > Aleksandar
> >
> > > Should we define/document a criteria for a test to be slow?  Given
> > > that this is highly subjective, we have to think of:
> > >
> > >  * Will we consider the average or maximum run time (the timeout
> > >    definition)?
> > >
> > >  * For a single test, what is "slow"? Some rough numbers from Travis
> > >    CI[1] to help us with guidelines:
> > >    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS
(6.04 s)
> > >    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS
(2.91 s)
> > >    -
> >
linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> > PASS (18.14 s)
> > >    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> > >
> > >  * Do we want to set a maximum job timeout?  This way we can skip
> > >    tests after a given amount of time has passed.  Currently we
interrupt
> > >    the test running when the job timeout is reached, but it's possible
> > >    to add a option so that no new tests will be started, but currently
> > >    running ones will be waited on.
> > >
> > > Regards,
> > > - Cleber.
> > >
> > > [1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518
> > >
> > > > --
> > > > Eduardo
> > > >
> >