Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

On Fri, May 21, 2021 at 2:14 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 21/05/2021 16.29, Peter Maydell wrote:
> > On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4bug@amsat.org> 
> > wrote:
> >> If you think these tests belong to tests/tcg/, I am OK to put
> >> them they, but I don't think adding the Avocado buildsys
> >> machinery to the already-complex tests/tcg/ Makefiles is going
> >> to help us...
> >
> > This does raise the question of what we're actually trying
> > to distinguish. It seems to me somewhat that what tests/acceptance/
> > actually contains that makes it interestingly different from other
> > tests/ stuff is that it's specifically "tests using the Avocado
> > framework". On that theory we might name it tests/avocado/.
>
> I think there are two aspects:
>
> 1) These tests are using the avocado framework
>
> 2) These tests are downloading other stuff from the internet (unlike the
> other tests that we have)
>

After Peter's reply, I noticed QEMU does not organize tests under the
tests folder by software engineering test category but by the
mechanism/machinery the tests run on. This makes me think that we may
need to handle the folders name and the CI jobs name differently:

1 - Change the current "test/acceptance" folder name to "test/(avocado
or avoqado)." Change the "make check-acceptance" to "make
check-validation," and the GitLab CI job names to "validation,"
meaning that, in a promising future, other tests running on a
different framework and acting like validation tests would run in the
same make command and same GitLab CI job.

2 - Change the current "test/acceptance" folder name to "test/(avocado
or avoqado)." Change the "make check-acceptance" to "make
check-(avocado or avoqaco)" and the GitLab CI job names to "(avocado
or avoqado)," meaning that, in a promising future, we can categorize
validation jobs inside the CI and run each of the different validation
tests supported by a framework on its own GitLab CI job.

Personally, I prefer option 2 as it gives more flexibility to decide
how to set a GitLab CI job or run it when testing locally.

> > Or we could just leave it as it is -- is the current naming
> > actually confusing anybody? :-)
>
> Yes, I think "acceptance" is rather confusing. So far they haven't been part
> of your PR acceptance tests (well, now they are part of the gitlab-CI,
> though), and it's also not about tests that have been set up by customers,
> which is what you normally think of when hearing "acceptance tests". So a
> different name would be adequate.
>
> I think I'd vote for either "avocado", "avoqado" or "validation".
>

Even laughing every time I read "avoqado" (and thanks for that), I
liked the idea as there is supplementary code added inside
"tests/acceptance/avocado_qemu" to support the tests, meaning they are
not "pure" avocado.

>   Thomas
>