Re: [PATCH] tests/avocado: using several workers while testing

[Daniel P . Berrangé]

On Wed, Nov 16, 2022 at 04:48:10PM +0300, Pavel Dovgalyuk wrote:
> From: bakulinm <bakulinm@gmail.com>
> 
> make check-avocado takes a lot of time, and avocado since version 91 has
> multithreaded mode for running several tests simultaneously.
> This patch allows to run "make check-avocado -j" to use all cores or,
> for example, "make check-avocado -j4" to select number of workers to use.
> By default ("make check-avocado") only one worker is used.
> 
> Changes:
> 1) Version of avocado in requirements.txt upgraded from 88.1 to <93
>  (LTS version is used, as mentioned here
>   https://avocado-framework.readthedocs.io/en/latest/releases/lts/92_0.html )
> 2) Makefile 4.1 (used in e.g. Ubuntu 18.04) doesn't provide number of jobs
>  in $MAKEFLAGS, so python script from here
>  https://stackoverflow.com/a/67247743/5936122 is used.


> diff --git a/tests/jobs.py b/tests/jobs.py
> new file mode 100644
> index 0000000000..a339192d97
> --- /dev/null
> +++ b/tests/jobs.py
> @@ -0,0 +1,42 @@

No license information or attribution put on this code that
you've said was directly copied from stackoverflow. AFAICT,
all content on stackoverflow is placed under the creative
commons license. This is not something we would generally
want to be applied to code in QEMU as that's generally
considered as a content license.

Unless the copied code is trivial (this case is not), then
stackoverflow should really only be used a learning resource,
and then code written from scratch without copying, so it
can be placed under the project's usual license.

> +import argparse, os
> +import sys
> +
> +def safe_int(s):
> +    try:
> +        return int(s)
> +    except:
> +        return -1
> +
> +class JobserverArgs:
> +    known_names = ["jobserver-fds","jobserver-auth"]
> +    def __init__(self):
> +        self.fds = "-1,-1"
> +
> +    @staticmethod
> +    def from_argv():
> +        ja = JobserverArgs()
> +        parser = argparse.ArgumentParser()
> +        for name in JobserverArgs.known_names:
> +            parser.add_argument('--'+name, dest="fds")
> +        parser.parse_known_args(namespace=ja)
> +        return ja
> +
> +    def get_fds(self):
> +        return tuple([safe_int(fd) for fd in (self.fds+",").split(",")][:2])
> +
> +fd_in, fd_out = JobserverArgs.from_argv().get_fds()
> +
> +if fd_in == -1 or fd_out == -1:
> +# if no jobserver is used, but -j is present, use total number of cpu cores
> +    if '-j' in sys.argv:
> +        print(os.cpu_count())
> +# use single thread
> +    else:
> +        print(1)
> +else:
> +    os.set_blocking(fd_in, False)
> +
> +    tokens = os.read(fd_in, 1024)
> +    os.write(fd_out, tokens)
> +
> +    print(len(tokens)+1)
> \ No newline at end of file
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 0ba561b6bd..3b8c4d4706 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -2,5 +2,5 @@
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>  # Note that qemu.git/python/ is always implicitly installed.
> -avocado-framework==88.1
> +avocado-framework<93
>  pycdlib==1.11.0
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|