Re: [Qemu-devel] make check speed

[Lukáš Doktor]

Dne 23.8.2017 v 14:01 Thomas Huth napsal(a):
> On 23.08.2017 13:51, Lukáš Doktor wrote:
>> Dne 23.8.2017 v 10:35 Thomas Huth napsal(a):
>>> On 23.08.2017 10:01, Paolo Bonzini wrote:
>>>> On 23/08/2017 09:49, Thomas Huth wrote:
>>>>> While we're at it: I'd like to have a "make check-fast", too. Sometimes
>>>>> the normal "make check" is already too slow, e.g. while developing new
>>>>> patches, I sometimes just want to do a very quick sanity test to see
>>>>> whether I broke some basic things or not, and only do the "make check"
>>>>> before I submit my patches.
>>>>> So we would get three stages:
>>>>>
>>>>> - make check-fast => For very, very quick sanity tests only
>>>>>
>>>>> - make check => E.g. has to be run before submitting patches
>>>>>
>>>>> - make check-harder => might run a very long time, so best suited for
>>>>>                        nightly regression tests etc.?
>>>>>
>>>>> Does that sound reasonable? And the crucial question: Who is going to
>>>>> implement the basic framework for this?
>>>>
>>>> There's already make check-unit or make check-qtest-x86_64 depending on
>>>> what you're working on.
>>>
>>> True. And I just learned that you can also already set the SPEED
>>> variable to either "quick" or "slow" and that we're already using
>>> g_test_quick() and g_test_slow() in a couple of places to check this. So
>>> the framework for running quick vs. thorough tests is already there ...
>>> we just might want to add this to some more tests, I guess...
>>>
>>> Question for the maintainers and the test automation folks: Is anybody
>>> already running "make check SPEED=slow" or is this just rather an
>>> unheard-of way of running the tests?
>>>
>>>> If you have a many-core machine, of course, there's no simpler solution
>>>> than throwing more CPUs at it. :)
>>>
>>> Is it safe nowadays to run "make check -j4" for example? Last time I
>>> tried (maybe 1 or 2 years ago), there were still issues since some tests
>>> were using hard-coded temporary file names, so the parallel tests were
>>> disturbing each other, for example...
>>>
>>
>> Actually the `.travis.yml` defines `MAKEFLAGS="-j3"`, which results in `make 
>> check` being executed with 3 threads...
>>
>> I was actually looking at the increasing number of failed travis builds and 
>> it seems to be related to the fluctuating performance. Running `nice -n 20 
>> make check -j 12` with `nice -n 5 stress -c 20` in background results in the 
>> same kind of failures:
>>
>>     File '/tmp/qemu/include/qapi/qmp/qobject.h'
>>     Lines executed:0.00% of 9
>>     **
>>     ERROR:tests/prom-env-test.c:42:check_guest_memory: assertion failed 
>> (signature == MAGIC): (0x7c7f1b78 == 0xcafec0de)
> 
> I think you're simply running into timeout issues here since you're
> likely overloading the host, I guess. Or does your host have that many CPUs?
> If the timeouts are really an issue here, we simply might have to
> increase the timeout values a bit again...
> 

The reason I did this is that the `make check -j 12` runs for ~ 4 minutes on my 
machine (8 cores) while in travis it sometimes runs even more than 40 minutes. 
I wanted to get closer to see why is it failing and this might be the reason 
and yes, it's most certainly timeout issue. The problem with Travis is it gives 
quite decent power, but sometimes it's slowed for couple of seconds, or even 
minutes. This affects many of our (Avocado) selftests so we usually have 
timeouts between 10-60s for operations that usually take less than a second.

Lukáš

PS: Usually the `make check -j 12` works well on my machine...

>  Thomas
>

signature.asc
Description: OpenPGP digital signature