qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running


From: Cleber Rosa
Subject: Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them
Date: Fri, 21 Sep 2018 15:19:06 -0400
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0


On 9/20/18 7:07 PM, Philippe Mathieu-Daudé wrote:
> On 9/21/18 12:06 AM, Eduardo Habkost wrote:
>> On Thu, Sep 20, 2018 at 11:18:46PM +0200, Philippe Mathieu-Daudé wrote:
>>> On 9/20/18 10:14 PM, Eduardo Habkost wrote:
>>>> On Thu, Sep 20, 2018 at 04:00:27PM -0400, Cleber Rosa wrote:
>>>>> On 9/20/18 2:58 PM, Eduardo Habkost wrote:
>>>>>> On Thu, Sep 20, 2018 at 11:19:56AM -0400, Cleber Rosa wrote:
>>>>>>> The acceptance (aka functional, aka Avocado-based) tests are
>>>>>>> Python files located in "tests/acceptance" that need to be run
>>>>>>> with the Avocado libs and test runner.
>>>>>>>
>>>>>>> Let's provide a convenient way for QEMU developers to run them,
>>>>>>> by making use of the tests-venv with the required setup.
>>>>>>>
>>>>>>> Also, while the Avocado test runner will take care of creating a
>>>>>>> location to save test results to, it was understood that it's better
>>>>>>> if the results are kept within the build tree.
>>>>>>>
>>>>>>> Signed-off-by: Cleber Rosa <address@hidden>
>>>>>>> ---
>>>>>>>  docs/devel/testing.rst      | 28 +++++++++++++++++++++++-----
>>>>>>>  tests/Makefile.include      | 17 +++++++++++++++--
>>>>>>>  tests/venv-requirements.txt |  1 +
>>>>>>>  3 files changed, 39 insertions(+), 7 deletions(-)
>>>>>>>
>>>>>>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
>>>>>>> index 727c4019b5..0fbf0d0aac 100644
>>>>>>> --- a/docs/devel/testing.rst
>>>>>>> +++ b/docs/devel/testing.rst
>>>>>>> @@ -545,10 +545,24 @@ Tests based on ``avocado_qemu.Test`` can easily:
>>>>>>>     - 
>>>>>>> http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
>>>>>>>     - 
>>>>>>> http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
>>>>>>>  
>>>>>>> -Installation
>>>>>>> -------------
>>>>>>> +Running tests
>>>>>>> +-------------
>>>>>>>  
>>>>>>> -To install Avocado and its dependencies, run:
>>>>>>> +You can run the acceptance tests simply by executing:
>>>>>>> +
>>>>>>> +.. code::
>>>>>>> +
>>>>>>> +  make check-acceptance
>>>>>>> +
>>>>>>> +This involves the automatic creation of Python virtual environment
>>>>>>> +within the build tree (at ``tests/venv``) which will have all the
>>>>>>> +right dependencies, and will save tests results also within the
>>>>>>> +build tree (at ``tests/results``).
>>>>>>> +
>>>>>>> +Manual Installation
>>>>>>> +-------------------
>>>>>>> +
>>>>>>> +To manually install Avocado and its dependencies, run:
>>>>>>>  
>>>>>>>  .. code::
>>>>>>>  
>>>>>>> @@ -689,11 +703,15 @@ The exact QEMU binary to be used on QEMUMachine.
>>>>>>>  Uninstalling Avocado
>>>>>>>  --------------------
>>>>>>>  
>>>>>>> -If you've followed the installation instructions above, you can easily
>>>>>>> -uninstall Avocado.  Start by listing the packages you have installed::
>>>>>>> +If you've followed the manual installation instructions above, you can
>>>>>>> +easily uninstall Avocado.  Start by listing the packages you have
>>>>>>> +installed::
>>>>>>>  
>>>>>>>    pip list --user
>>>>>>>  
>>>>>>>  And remove any package you want with::
>>>>>>>  
>>>>>>>    pip uninstall <package_name>
>>>>>>> +
>>>>>>> +If you've used ``make check-acceptance``, the Python virtual 
>>>>>>> environment where
>>>>>>> +Avocado is installed will be cleaned up as part of ``make 
>>>>>>> check-clean``.
>>>>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>>>>> index 9bb90a83d4..8cef694954 100644
>>>>>>> --- a/tests/Makefile.include
>>>>>>> +++ b/tests/Makefile.include
>>>>>>> @@ -11,6 +11,7 @@ check-help:
>>>>>>>         @echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
>>>>>>>         @echo " $(MAKE) check-block          Run block tests"
>>>>>>>         @echo " $(MAKE) check-tcg            Run TCG tests"
>>>>>>> +       @echo " $(MAKE) check-acceptance     Run all acceptance 
>>>>>>> (functional) tests"
>>>>>>>         @echo " $(MAKE) check-report.html    Generates an HTML test 
>>>>>>> report"
>>>>>>>         @echo " $(MAKE) check-venv           Creates a Python venv for 
>>>>>>> tests"
>>>>>>>         @echo " $(MAKE) check-clean          Clean the tests"
>>>>>>> @@ -1002,10 +1003,11 @@ check-decodetree:
>>>>>>>  
>>>>>>>  # Python venv for running tests
>>>>>>>  
>>>>>>> -.PHONY: check-venv
>>>>>>> +.PHONY: check-venv check-acceptance
>>>>>>>  
>>>>>>>  TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>>>>>>>  TESTS_VENV_REQ=$(BUILD_DIR)/tests/venv-requirements.txt
>>>>>>> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>>>>>>  
>>>>>>>  $(TESTS_VENV_DIR):
>>>>>>>         $(call quiet-command, \
>>>>>>> @@ -1015,8 +1017,19 @@ $(TESTS_VENV_DIR):
>>>>>>>              $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), 
>>>>>>> \
>>>>>>>              PIP, $(TESTS_VENV_REQ))
>>>>>>>  
>>>>>>> +$(TESTS_RESULTS_DIR):
>>>>>>> +       $(call quiet-command, mkdir -p $@, \
>>>>>>> +            MKDIR, $@)
>>>>>>> +
>>>>>>>  check-venv: $(TESTS_VENV_DIR)
>>>>>>>  
>>>>>>> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
>>>>>>> +       $(call quiet-command, \
>>>>>>> +            $(TESTS_VENV_DIR)/bin/avocado \
>>>>>>> +            --show=none run --job-results-dir=$(TESTS_RESULTS_DIR) 
>>>>>>> --failfast=on \
>>>>>>> +            $(SRC_PATH)/tests/acceptance, \
>>>>>>> +            "AVOCADO", "tests/acceptance")
>>>>>>
>>>>>> I think we should provide something easy to use for people who
>>>>>> already have the right Avocado version installed in their system
>>>>>> and want to avoid re-downloading Avocado every time.
>>>>>>
>>>>>
>>>>> Right now, people using their own Avocado installation is actually the
>>>>> documented way.  The difference from the currently documented way is
>>>>> that instead of doing `make check-acceptance`, people will run:
>>>>>
>>>>>  $ avocado run tests/acceptance
>>>>>
>>>>> IMO, for these users, a `alias check-acceptance='avocado run
>>>>> tests/acceptance'` brings almost the same value.
>>>>>
>>>>> About re-downloading: pip caches files by default, so while Avocado will
>>>>> be installed every time a new venv is created, it should be downloaded
>>>>> only once.  And I should mention that, given the fact that one of the
>>>>> packaged formats of Avocado is a "Python wheel", the installation is
>>>>> basically a "tar xf" of sorts.
>>>>
>>>> Fair enough.  Note that I'm just guessing what other developers
>>>> would expect here.  Maybe most people won't mind having "pip
>>>> install" running implicitly when they run acceptance tests and
>>>> this is a non-issue.
>>>>
>>>> I'm hoping we can get the attention of more people on this thread
>>>> so we can get feedback from actual users.  If we don't get any
>>>> feedback about this, I won't mind if we include only the rule you
>>>> suggested, and improve the system later.
>>>
>>> I'm experiencing the 2 cases:
>>>
>>> - As a QEMU developer working on a feature, you plan to add some Avocado
>>> tests. You likely works with bleeding edge QEMU. If system packaged
>>> Avocado is not updated enough for you, you'll use the venv+pip setup.
>>> [This is my particular use.]
>>>
>>> - As a QEMU maintainer you would run (upstream merged) Avocado tests to
>>> avoid regressions. The system packages should be sufficient.
>>> [This is how I'm using it on Travis-CI.]
>>
>> By "system packages" you mean packages provided by the OS
>> distribution, or just that you want to run "pip install" before
>> the build is executed, as described at
>>   https://docs.travis-ci.com/user/languages/python/#dependency-management
>> ?
> 
> Good point :)
> 
> Avocado's team provides a Fedora package, so I'm expecting maintainers
> using Fedora to simply give "dnf install python-avocado" a try.
> 
> Travis-CI is Ubuntu based, so we have to bootstrap from the python3-pip
> package, installing python dependencies in the user environment.
> 

I have a feeling that if you do something like:

install:
    - pip install -r tests/env-requirements.txt

On your .travis.yaml, then the deps should be fulfilled ahead of time.
pip here, should be the *right* one -- Python 3 based if your travis
environment is configured with a Python version:

python:
    - 3.6

Then, with the --system-site-packages option mentioned before, the
"tests-venv" should reuse those modules.

> I'm currently using Debian, and feel much safer installing/testing
> bleeding edge in trashable virtual environments (venv).
> 

Agreed.

- Cleber.

> 3 cases so far :)
> 



reply via email to

[Prev in Thread] Current Thread [Next in Thread]