Re: [Qemu-devel] make check speed

[Cleber Rosa]

On 08/23/2017 08:13 AM, Lukáš Doktor wrote:
> Dne 23.8.2017 v 14:01 Thomas Huth napsal(a):
>> On 23.08.2017 13:51, Lukáš Doktor wrote:
>>> Dne 23.8.2017 v 10:35 Thomas Huth napsal(a):
>>>> On 23.08.2017 10:01, Paolo Bonzini wrote:
>>>>> On 23/08/2017 09:49, Thomas Huth wrote:
>>>>>> While we're at it: I'd like to have a "make check-fast", too. Sometimes
>>>>>> the normal "make check" is already too slow, e.g. while developing new
>>>>>> patches, I sometimes just want to do a very quick sanity test to see
>>>>>> whether I broke some basic things or not, and only do the "make check"
>>>>>> before I submit my patches.
>>>>>> So we would get three stages:
>>>>>>
>>>>>> - make check-fast => For very, very quick sanity tests only
>>>>>>
>>>>>> - make check => E.g. has to be run before submitting patches
>>>>>>
>>>>>> - make check-harder => might run a very long time, so best suited for
>>>>>>                        nightly regression tests etc.?
>>>>>>
>>>>>> Does that sound reasonable? And the crucial question: Who is going to
>>>>>> implement the basic framework for this?
>>>>>
>>>>> There's already make check-unit or make check-qtest-x86_64 depending on
>>>>> what you're working on.
>>>>
>>>> True. And I just learned that you can also already set the SPEED
>>>> variable to either "quick" or "slow" and that we're already using
>>>> g_test_quick() and g_test_slow() in a couple of places to check this. So
>>>> the framework for running quick vs. thorough tests is already there ...
>>>> we just might want to add this to some more tests, I guess...
>>>>
>>>> Question for the maintainers and the test automation folks: Is anybody
>>>> already running "make check SPEED=slow" or is this just rather an
>>>> unheard-of way of running the tests?
>>>>
>>>>> If you have a many-core machine, of course, there's no simpler solution
>>>>> than throwing more CPUs at it. :)
>>>>
>>>> Is it safe nowadays to run "make check -j4" for example? Last time I
>>>> tried (maybe 1 or 2 years ago), there were still issues since some tests
>>>> were using hard-coded temporary file names, so the parallel tests were
>>>> disturbing each other, for example...
>>>>
>>>
>>> Actually the `.travis.yml` defines `MAKEFLAGS="-j3"`, which results in 
>>> `make check` being executed with 3 threads...
>>>
>>> I was actually looking at the increasing number of failed travis builds and 
>>> it seems to be related to the fluctuating performance. Running `nice -n 20 
>>> make check -j 12` with `nice -n 5 stress -c 20` in background results in 
>>> the same kind of failures:
>>>
>>>     File '/tmp/qemu/include/qapi/qmp/qobject.h'
>>>     Lines executed:0.00% of 9
>>>     **
>>>     ERROR:tests/prom-env-test.c:42:check_guest_memory: assertion failed 
>>> (signature == MAGIC): (0x7c7f1b78 == 0xcafec0de)
>>
>> I think you're simply running into timeout issues here since you're
>> likely overloading the host, I guess. Or does your host have that many CPUs?
>> If the timeouts are really an issue here, we simply might have to
>> increase the timeout values a bit again...
>>
> 
> The reason I did this is that the `make check -j 12` runs for ~ 4 minutes on 
> my machine (8 cores) while in travis it sometimes runs even more than 40 
> minutes. I wanted to get closer to see why is it failing and this might be 
> the reason and yes, it's most certainly timeout issue. The problem with 
> Travis is it gives quite decent power, but sometimes it's slowed for couple 
> of seconds, or even minutes. This affects many of our (Avocado) selftests so 
> we usually have timeouts between 10-60s for operations that usually take less 
> than a second.

And we learned that even huge timeouts are not a valid solution, in fact
they can become a counter solution.  On many environments it's not wise
to even run tests are time sensitive.  Travis is one, Fedora package
build hosts are another example.

IMO, this signals that test metadata (and categorization on top of it)
can be of great help in many situations.  Instead of writing a multitude
of "make check-(you-characteristics-here)" targets, something like "make
EXCLUDE=time_senstitive check" is far more flexible.

Using Avocado as an example, it'd mean running something like:

 $ avocado run -t '-time_sensitive,-superuser' tests/*

- Cleber.

> 
> Lukáš
> 
> PS: Usually the `make check -j 12` works well on my machine...
> 
>>  Thomas
>>
> 
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

signature.asc
Description: OpenPGP digital signature