Re: Proposal for a regular upstream performance testing

[Lukáš Doktor]

Dne 28. 03. 22 v 11:57 Stefan Hajnoczi napsal(a):
> On Mon, Mar 28, 2022 at 08:18:43AM +0200, Lukáš Doktor wrote:
>> Hello Stefan, folks,
>>
>> I seem to have another hit, an improvement actually and it seems to be 
>> bisected all the way to you, Stefan. Let me use this as another example of 
>> how such process could look like and we can use this to hammer-out the 
>> details like via what means to submit the request, whom to notify and how to 
>> proceed further.
>>
>> ---
>>
>> Last week I noticed an improvement in 
>> TunedLibvirt/fio-rot-Aj-8i/0000:./write-4KiB/throughput/iops_sec.mean 
>> (<driver name="qemu" type="raw" io="native" cache="none"/>, fio, rotationary 
>> disk, raw file on host xfs partition, jobs=#cpus, iodepth=8, 4k writes) 
>> check and bisected it to:
>>
>> commit fc8796465c6cd4091efe6a2f8b353f07324f49c7
>> Author: Stefan Hajnoczi <stefanha@redhat.com>
>> Date:   Wed Feb 23 15:57:03 2022 +0000
>>
>>     aio-posix: fix spurious ->poll_ready() callbacks in main loop
>>
>> Could you please confirm that it does make sense and that it is expected? 
>> (looks like it from the description).
>>
>> Note that this commit was pin pointed using 2 out of 3 commits result, there 
>> were actually some little differences between commits fc8 and cc5. The fc8 
>> and 202 results scored similarly to both, good and bad commits with 2 being 
>> closer to the bad one. Since cc5 they seem to stabilize further scoring 
>> slightly lower than the median fc8 result. Anyway I don't have enough data 
>> to declare anything. I can bisect it further if needed.
> 
> Yes, I can imagine that commit fc8796465c6c might improve non-IOThread
> performance!
> 

Hello Stefan and thank you for this confirmation as well as questions. Let me 
explain that a bit and perhaps you can help me to improve the report as it was 
designed for CI team and can be I can see where the confusion might come from.

> I don't know how to read the report:
> - What is the difference between "Group stats" and "Failures"?
> - Why are there 3 different means in "Group stats"?

The group stats combine (average) multiple individual failures with some common 
aspect and stricter thresholds are then used based on the number of matches. 
The group names contain asterisk (*) character which works the same way linux 
globs work and all matching tests are included. In this report it doesn't 
really makes sense as only single test variant was performed, but when various 
tests/scenarios are being used it can indicate change across all fio-write 
jobs, or all tests on various profiles...

> - Why are there 3 "fc8" columns in "Failures"?

To avoid bisect wandering away I'm usually using 3-5 samples per run. Still 
this wasn't reliable enough in some cases and bisect ended up on random commits 
so I added a 2 out of 3 feature. Every commit is checked twice and when they 
don't match it runs the third one. In the end all of the attempts are included 
in the report to better visualize how stable the results are.

> 
> I don't feel confident searching git-log(1) output with 3-character
> commit IDs. git itself uses 12 characters for short commit IDs with a
> reasonably low chance of collisions.
> 

I chose 3 letters only as usually the bisection window is quite small and there 
is the bisection log with all the full commits attached. Using longer hashes 
results in very wide table, which I wanted to avoid. Perhaps I can modify the 
links which currently point to the jenkins results to point to the qemu sha 
instead, what do you think?

Regards,
Lukáš

> Stefan

OpenPGP_0x26B362E47FCF22C1.asc
Description: OpenPGP public key

OpenPGP_signature
Description: OpenPGP digital signature