Re: [Qemu-devel] [RFC PATCH] configure: Enable out-of-tree acceptance tests

[Cleber Rosa]

On 05/30/2018 04:03 PM, Philippe Mathieu-Daudé wrote:
> In order to allow out-of-tree acceptance tests with Avocado,
> create a symlink in the build tree.
> 
> Signed-off-by: Philippe Mathieu-Daudé <address@hidden>
> ---
> Based-on: address@hidden
> http://lists.nongnu.org/archive/html/qemu-devel/2018-05/msg06877.html
> 
> Before:
> 
>   build_dir$ avocado run 
> /full/path/to/sources/qemu/tests/acceptance/boot_linux_console.py
> 
> After
> 
>   build_dir$ avocado run tests/acceptance/boot_linux_console.py
> 
> I first wanted to try something like:
> 
> echo "[datadir.paths]" > .avocado.conf
> echo "test_dir = $source_path/tests/acceptance" >> .avocado.conf
> 

FIY, "$HOME/.config/avocado/avocado.conf" is parsed by default.  Take a
look at the "avocado config" output for the details.

> to run:
> 
>   build_dir$ avocado run -t console
> 

I see your point, you'd like the command above to behave similarly to:

  avocado run -t console $avocado_datadir_paths_test_dir

Right?  So, the first compromise solution with current Avocado is:

  echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
  echo "test_dir = $source_path/tests/acceptance" >>
~/.config/avocado/avocado.conf

With "$source_path" being something literal, and not a variable.  It
allows individual tests to be found at that location:

  cd /some/random/path
  avocado run boot_linux_console.py

But the following still won't work:

  avocado run -t console
  No test references provided nor any other arguments resolved into
tests. Please double check the executed command.

I do think the current Avocado behavior is somewhat inconsistent,
because list and run should ideally be symmetrical.  With the
configuration above, "list" behaves like this:

   avocado list
INSTRUMENTED
/home/cleber/src/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test
INSTRUMENTED
/home/cleber/src/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version
INSTRUMENTED /home/cleber/src/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc
INSTRUMENTED
/home/cleber/src/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password
INSTRUMENTED
/home/cleber/src/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password
INSTRUMENTED
/home/cleber/src/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password

But at the same, there are security implications: `list` won't
load/execute any test code (different from, say, standard Python
unittests), while "run" obviously will.  So "avocado run" may end up
running what users don't want if a malicious user controls
"$avocado_datadir_paths_test_dir".

What if Avocado introduces an option that would enable the use of
Avocado's test dir when no test reference is given?  This could be
enabled only once (but would be disabled by default):

  echo "[loader]" > ~/.config/avocado/avocado.conf
  echo "use_test_dir_when_no_references_given = True" >> >
~/.config/avocado/avocado.conf

  echo "[datadir.paths]" >> ~/.config/avocado/avocado.conf
  echo "test_dir = $source_path/tests/acceptance" >>
~/.config/avocado/avocado.conf

And after that, the following would run all "console" tests:

  avocado run -t console

How does this sound?

- Cleber.

> but this doesn't work this way, I'd have to use
> 
>   build_dir$ avocado --config .avocado.conf boot_linux_console.py -t console
> 
> which isn't a win.
> ---
>  configure | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/configure b/configure
> index a6a4616c3e..0763ebe5ba 100755
> --- a/configure
> +++ b/configure
> @@ -7213,9 +7213,11 @@ for rom in seabios vgabios ; do
>  done
>  
>  # set up tests data directory
> -if [ ! -e tests/data ]; then
> -    symlink "$source_path/tests/data" tests/data
> -fi
> +for tests_subdir in acceptance data; do
> +    if [ ! -e tests/$tests_subdir ]; then
> +        symlink "$source_path/tests/$tests_subdir" tests/$tests_subdir
> +    fi
> +done
>  
>  # set up qemu-iotests in this build directory
>  iotests_common_env="tests/qemu-iotests/common.env"
>