[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [RFC PATCH] configure: Enable out-of-tree acceptance te
From: |
Philippe Mathieu-Daudé |
Subject: |
Re: [Qemu-devel] [RFC PATCH] configure: Enable out-of-tree acceptance tests |
Date: |
Tue, 5 Jun 2018 11:45:03 -0300 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 |
On 05/30/2018 10:06 PM, Cleber Rosa wrote:
> On 05/30/2018 04:03 PM, Philippe Mathieu-Daudé wrote:
>> In order to allow out-of-tree acceptance tests with Avocado,
>> create a symlink in the build tree.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <address@hidden>
>> ---
>> Based-on: address@hidden
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-05/msg06877.html
>>
>> Before:
>>
>> build_dir$ avocado run
>> /full/path/to/sources/qemu/tests/acceptance/boot_linux_console.py
>>
>> After
>>
>> build_dir$ avocado run tests/acceptance/boot_linux_console.py
>>
>> I first wanted to try something like:
>>
>> echo "[datadir.paths]" > .avocado.conf
>> echo "test_dir = $source_path/tests/acceptance" >> .avocado.conf
>>
>
> FIY, "$HOME/.config/avocado/avocado.conf" is parsed by default. Take a
> look at the "avocado config" output for the details.
I followed the virtualenv installation from the link you added in
testing.rst:
http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado
So my config is:
Config files read (in order):
/builds/qemu/venv2/local/lib/python2.7/site-packages/avocado/etc/avocado/avocado.conf
/builds/qemu/venv2/local/lib/python2.7/site-packages/avocado/etc/avocado/conf.d/result_upload.conf
/builds/qemu/venv2/local/lib/python2.7/site-packages/avocado/etc/avocado/conf.d/resultsdb.conf
/builds/qemu/venv2/local/lib/python2.7/site-packages/avocado/etc/avocado/conf.d/jobscripts.conf
/builds/qemu/venv2/local/lib/python2.7/site-packages/avocado/etc/avocado/conf.d/gdb.conf
Section.Key Value
datadir.paths.base_dir /var/lib/avocado
datadir.paths.test_dir /usr/share/avocado/tests
datadir.paths.data_dir /var/lib/avocado/data
datadir.paths.logs_dir ~/avocado/job-results
I find useful to be able to append/overwrite the user 'global' config
with a current directory one, like we have with .gdbinit, because I use
out-of-tree build dir to test different build options/features with the
same codebase.
So personally I'd like this parse order:
- $VENV/lib/python2.7/site-packages/avocado/etc/avocado/avocado.conf
- $HOME/.config/avocado/avocado.conf
- .avocado.conf
>
>> to run:
>>
>> build_dir$ avocado run -t console
>>
>
> I see your point, you'd like the command above to behave similarly to:
>
> avocado run -t console $avocado_datadir_paths_test_dir
>
> Right? So, the first compromise solution with current Avocado is:
>
> echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
> echo "test_dir = $source_path/tests/acceptance" >>
> ~/.config/avocado/avocado.conf
>
> With "$source_path" being something literal, and not a variable. It
> allows individual tests to be found at that location:
>
> cd /some/random/path
> avocado run boot_linux_console.py
'echo' is not that friendly :S
$ avocado run boot_linux_console.py
Avocado crashed unexpectedly: Value 'logs_dir' not found in section
'datadir.paths'
You can find details in
/var/tmp/avocado-traceback-2018-06-05_11:32:01-MUn_kE.log
>
> But the following still won't work:
>
> avocado run -t console
> No test references provided nor any other arguments resolved into
> tests. Please double check the executed command.
>
> I do think the current Avocado behavior is somewhat inconsistent,
> because list and run should ideally be symmetrical. With the
> configuration above, "list" behaves like this:
>
> avocado list
> INSTRUMENTED
> /home/cleber/src/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test
> INSTRUMENTED
> /home/cleber/src/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version
> INSTRUMENTED /home/cleber/src/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc
> INSTRUMENTED
> /home/cleber/src/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password
> INSTRUMENTED
> /home/cleber/src/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password
> INSTRUMENTED
> /home/cleber/src/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password
>
> But at the same, there are security implications: `list` won't
> load/execute any test code (different from, say, standard Python
> unittests), while "run" obviously will. So "avocado run" may end up
> running what users don't want if a malicious user controls
> "$avocado_datadir_paths_test_dir".
>
> What if Avocado introduces an option that would enable the use of
> Avocado's test dir when no test reference is given? This could be
> enabled only once (but would be disabled by default):
>
> echo "[loader]" > ~/.config/avocado/avocado.conf
> echo "use_test_dir_when_no_references_given = True" >> >
> ~/.config/avocado/avocado.conf
>
> echo "[datadir.paths]" >> ~/.config/avocado/avocado.conf
> echo "test_dir = $source_path/tests/acceptance" >>
> ~/.config/avocado/avocado.conf
>
> And after that, the following would run all "console" tests:
>
> avocado run -t console
>
> How does this sound?
For my use cases this doesn't worry me, I'll let Eduardo/Fam opine about
use_test_dir_when_no_references_given.
>
> - Cleber.
>
>> but this doesn't work this way, I'd have to use
>>
>> build_dir$ avocado --config .avocado.conf boot_linux_console.py -t console
>>
>> which isn't a win.
>> ---
>> configure | 8 +++++---
>> 1 file changed, 5 insertions(+), 3 deletions(-)
>>
>> diff --git a/configure b/configure
>> index a6a4616c3e..0763ebe5ba 100755
>> --- a/configure
>> +++ b/configure
>> @@ -7213,9 +7213,11 @@ for rom in seabios vgabios ; do
>> done
>>
>> # set up tests data directory
>> -if [ ! -e tests/data ]; then
>> - symlink "$source_path/tests/data" tests/data
>> -fi
>> +for tests_subdir in acceptance data; do
>> + if [ ! -e tests/$tests_subdir ]; then
>> + symlink "$source_path/tests/$tests_subdir" tests/$tests_subdir
>> + fi
>> +done
>>
>> # set up qemu-iotests in this build directory
>> iotests_common_env="tests/qemu-iotests/common.env"
>>
- Re: [Qemu-devel] [RFC PATCH] configure: Enable out-of-tree acceptance tests,
Philippe Mathieu-Daudé <=