[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
From: |
Eduardo Habkost |
Subject: |
Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests |
Date: |
Wed, 22 May 2019 20:07:05 -0300 |
User-agent: |
Mutt/1.10.1 (2018-07-13) |
On Wed, May 22, 2019 at 05:46:06PM -0400, Cleber Rosa wrote:
>
>
> ----- Original Message -----
> > From: "Eduardo Habkost" <address@hidden>
> > To: "Philippe Mathieu-Daudé" <address@hidden>
> > Cc: address@hidden, "Aleksandar Rikalo" <address@hidden>, "Aleksandar
> > Markovic"
> > <address@hidden>, "Aleksandar Markovic" <address@hidden>, "Cleber Rosa"
> > <address@hidden>,
> > "Aurelien Jarno" <address@hidden>, "Wainer dos Santos Moschetta"
> > <address@hidden>
> > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> >
> > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > > Hi,
> > >
> > > It was a rainy week-end here, so I invested it to automatize some
> > > of my MIPS tests.
> > >
> > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > meant to run on a CI system but rather on a workstation previous
> > > to post a pull request.
> > > It can surely be improved, but it is a good starting point.
> >
> > Until we actually have a mechanism to exclude the test case on
> > travis-ci, I will remove patch 4/4 from the queue. Aleksandar,
> > please don't merge patch 4/4 yet or it will break travis-ci.
> >
> > Cleber, Wainer, is it already possible to make "avocado run" skip
> > tests tagged with "slow"?
> >
>
> The mechanism exists, but we haven't tagged any test so far as slow.
>
> Should we define/document a criteria for a test to be slow? Given
> that this is highly subjective, we have to think of:
>
> * Will we consider the average or maximum run time (the timeout
> definition)?
>
> * For a single test, what is "slow"? Some rough numbers from Travis
> CI[1] to help us with guidelines:
> - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc: PASS (6.04 s)
> - boot_linux_console.py:BootLinuxConsole.test_arm_virt: PASS (2.91 s)
> -
> linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> PASS (18.14 s)
> - boot_linux.py:BootLinuxAarch64.test_virt: PASS (396.88 s)
I don't think we need to overthink this. Whatever objective
criteria we choose, I'm sure we'll have to adapt them later due
to real world problems.
e.g.: is 396 seconds too slow? I don't know, it depends: does it
break Travis and other CI systems often because of timeouts? If
yes, then we should probably tag it as slow.
If having subjective criteria is really a problem (I don't think
it is), then we can call the tag "skip_travis", and stop worrying
about defining what exactly is "slow".
>
> * Do we want to set a maximum job timeout? This way we can skip
> tests after a given amount of time has passed. Currently we interrupt
> the test running when the job timeout is reached, but it's possible
> to add a option so that no new tests will be started, but currently
> running ones will be waited on.
I'm not sure I understand the suggestion to skip tests. If we
skip tests after a timeout, how would we differentiate a test
being expectedly slow from a QEMU hang?
--
Eduardo
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, (continued)
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Cleber Rosa, 2019/05/22
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Aleksandar Markovic, 2019/05/22
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Aleksandar Markovic, 2019/05/22
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Cleber Rosa, 2019/05/23
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Aleksandar Markovic, 2019/05/23
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Philippe Mathieu-Daudé, 2019/05/23
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Aleksandar Markovic, 2019/05/23
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Eduardo Habkost, 2019/05/24
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Aleksandar Markovic, 2019/05/24
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Eduardo Habkost, 2019/05/24
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests,
Eduardo Habkost <=
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Cleber Rosa, 2019/05/22
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Eduardo Habkost, 2019/05/22
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Philippe Mathieu-Daudé, 2019/05/23
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Cleber Rosa, 2019/05/23
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Eduardo Habkost, 2019/05/23
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Aleksandar Markovic, 2019/05/24
- Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests, Eduardo Habkost, 2019/05/24