Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them

[Philippe Mathieu-Daudé]

On 9/21/18 12:06 AM, Eduardo Habkost wrote:
> On Thu, Sep 20, 2018 at 11:18:46PM +0200, Philippe Mathieu-Daudé wrote:
>> On 9/20/18 10:14 PM, Eduardo Habkost wrote:
>>> On Thu, Sep 20, 2018 at 04:00:27PM -0400, Cleber Rosa wrote:
>>>> On 9/20/18 2:58 PM, Eduardo Habkost wrote:
>>>>> On Thu, Sep 20, 2018 at 11:19:56AM -0400, Cleber Rosa wrote:
>>>>>> The acceptance (aka functional, aka Avocado-based) tests are
>>>>>> Python files located in "tests/acceptance" that need to be run
>>>>>> with the Avocado libs and test runner.
>>>>>>
>>>>>> Let's provide a convenient way for QEMU developers to run them,
>>>>>> by making use of the tests-venv with the required setup.
>>>>>>
>>>>>> Also, while the Avocado test runner will take care of creating a
>>>>>> location to save test results to, it was understood that it's better
>>>>>> if the results are kept within the build tree.
>>>>>>
>>>>>> Signed-off-by: Cleber Rosa <address@hidden>
>>>>>> ---
>>>>>>  docs/devel/testing.rst      | 28 +++++++++++++++++++++++-----
>>>>>>  tests/Makefile.include      | 17 +++++++++++++++--
>>>>>>  tests/venv-requirements.txt |  1 +
>>>>>>  3 files changed, 39 insertions(+), 7 deletions(-)
>>>>>>
>>>>>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
>>>>>> index 727c4019b5..0fbf0d0aac 100644
>>>>>> --- a/docs/devel/testing.rst
>>>>>> +++ b/docs/devel/testing.rst
>>>>>> @@ -545,10 +545,24 @@ Tests based on ``avocado_qemu.Test`` can easily:
>>>>>>     - 
>>>>>> http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
>>>>>>     - 
>>>>>> http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
>>>>>>  
>>>>>> -Installation
>>>>>> -------------
>>>>>> +Running tests
>>>>>> +-------------
>>>>>>  
>>>>>> -To install Avocado and its dependencies, run:
>>>>>> +You can run the acceptance tests simply by executing:
>>>>>> +
>>>>>> +.. code::
>>>>>> +
>>>>>> +  make check-acceptance
>>>>>> +
>>>>>> +This involves the automatic creation of Python virtual environment
>>>>>> +within the build tree (at ``tests/venv``) which will have all the
>>>>>> +right dependencies, and will save tests results also within the
>>>>>> +build tree (at ``tests/results``).
>>>>>> +
>>>>>> +Manual Installation
>>>>>> +-------------------
>>>>>> +
>>>>>> +To manually install Avocado and its dependencies, run:
>>>>>>  
>>>>>>  .. code::
>>>>>>  
>>>>>> @@ -689,11 +703,15 @@ The exact QEMU binary to be used on QEMUMachine.
>>>>>>  Uninstalling Avocado
>>>>>>  --------------------
>>>>>>  
>>>>>> -If you've followed the installation instructions above, you can easily
>>>>>> -uninstall Avocado.  Start by listing the packages you have installed::
>>>>>> +If you've followed the manual installation instructions above, you can
>>>>>> +easily uninstall Avocado.  Start by listing the packages you have
>>>>>> +installed::
>>>>>>  
>>>>>>    pip list --user
>>>>>>  
>>>>>>  And remove any package you want with::
>>>>>>  
>>>>>>    pip uninstall <package_name>
>>>>>> +
>>>>>> +If you've used ``make check-acceptance``, the Python virtual 
>>>>>> environment where
>>>>>> +Avocado is installed will be cleaned up as part of ``make check-clean``.
>>>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>>>> index 9bb90a83d4..8cef694954 100644
>>>>>> --- a/tests/Makefile.include
>>>>>> +++ b/tests/Makefile.include
>>>>>> @@ -11,6 +11,7 @@ check-help:
>>>>>>          @echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
>>>>>>          @echo " $(MAKE) check-block          Run block tests"
>>>>>>          @echo " $(MAKE) check-tcg            Run TCG tests"
>>>>>> +        @echo " $(MAKE) check-acceptance     Run all acceptance 
>>>>>> (functional) tests"
>>>>>>          @echo " $(MAKE) check-report.html    Generates an HTML test 
>>>>>> report"
>>>>>>          @echo " $(MAKE) check-venv           Creates a Python venv for 
>>>>>> tests"
>>>>>>          @echo " $(MAKE) check-clean          Clean the tests"
>>>>>> @@ -1002,10 +1003,11 @@ check-decodetree:
>>>>>>  
>>>>>>  # Python venv for running tests
>>>>>>  
>>>>>> -.PHONY: check-venv
>>>>>> +.PHONY: check-venv check-acceptance
>>>>>>  
>>>>>>  TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>>>>  TESTS_VENV_REQ=$(BUILD_DIR)/tests/venv-requirements.txt
>>>>>> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>>>>>  
>>>>>>  $(TESTS_VENV_DIR):
>>>>>>          $(call quiet-command, \
>>>>>> @@ -1015,8 +1017,19 @@ $(TESTS_VENV_DIR):
>>>>>>              $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
>>>>>>              PIP, $(TESTS_VENV_REQ))
>>>>>>  
>>>>>> +$(TESTS_RESULTS_DIR):
>>>>>> +        $(call quiet-command, mkdir -p $@, \
>>>>>> +            MKDIR, $@)
>>>>>> +
>>>>>>  check-venv: $(TESTS_VENV_DIR)
>>>>>>  
>>>>>> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
>>>>>> +        $(call quiet-command, \
>>>>>> +            $(TESTS_VENV_DIR)/bin/avocado \
>>>>>> +            --show=none run --job-results-dir=$(TESTS_RESULTS_DIR) 
>>>>>> --failfast=on \
>>>>>> +            $(SRC_PATH)/tests/acceptance, \
>>>>>> +            "AVOCADO", "tests/acceptance")
>>>>>
>>>>> I think we should provide something easy to use for people who
>>>>> already have the right Avocado version installed in their system
>>>>> and want to avoid re-downloading Avocado every time.
>>>>>
>>>>
>>>> Right now, people using their own Avocado installation is actually the
>>>> documented way.  The difference from the currently documented way is
>>>> that instead of doing `make check-acceptance`, people will run:
>>>>
>>>>  $ avocado run tests/acceptance
>>>>
>>>> IMO, for these users, a `alias check-acceptance='avocado run
>>>> tests/acceptance'` brings almost the same value.
>>>>
>>>> About re-downloading: pip caches files by default, so while Avocado will
>>>> be installed every time a new venv is created, it should be downloaded
>>>> only once.  And I should mention that, given the fact that one of the
>>>> packaged formats of Avocado is a "Python wheel", the installation is
>>>> basically a "tar xf" of sorts.
>>>
>>> Fair enough.  Note that I'm just guessing what other developers
>>> would expect here.  Maybe most people won't mind having "pip
>>> install" running implicitly when they run acceptance tests and
>>> this is a non-issue.
>>>
>>> I'm hoping we can get the attention of more people on this thread
>>> so we can get feedback from actual users.  If we don't get any
>>> feedback about this, I won't mind if we include only the rule you
>>> suggested, and improve the system later.
>>
>> I'm experiencing the 2 cases:
>>
>> - As a QEMU developer working on a feature, you plan to add some Avocado
>> tests. You likely works with bleeding edge QEMU. If system packaged
>> Avocado is not updated enough for you, you'll use the venv+pip setup.
>> [This is my particular use.]
>>
>> - As a QEMU maintainer you would run (upstream merged) Avocado tests to
>> avoid regressions. The system packages should be sufficient.
>> [This is how I'm using it on Travis-CI.]
> 
> By "system packages" you mean packages provided by the OS
> distribution, or just that you want to run "pip install" before
> the build is executed, as described at
>   https://docs.travis-ci.com/user/languages/python/#dependency-management
> ?

Good point :)

Avocado's team provides a Fedora package, so I'm expecting maintainers
using Fedora to simply give "dnf install python-avocado" a try.

Travis-CI is Ubuntu based, so we have to bootstrap from the python3-pip
package, installing python dependencies in the user environment.

I'm currently using Debian, and feel much safer installing/testing
bleeding edge in trashable virtual environments (venv).

3 cases so far :)