Re: Proposal for a regular upstream performance testing

[Lukáš Doktor]

Dne 30. 11. 20 v 14:23 Stefan Hajnoczi napsal(a):
> On Thu, Nov 26, 2020 at 09:43:38AM +0000, Daniel P. Berrangé wrote:
> > On Thu, Nov 26, 2020 at 09:10:14AM +0100, Lukáš Doktor wrote:
> > > Ideally the community should have a way to also issue their custom builds
> > > in order to verify their patches so they can debug and address issues
> > > better than just commit to qemu-master.
> >
> > Allowing community builds certainly adds an extra dimension of complexity
> > to the problem, as you need some kind of permissions control, as you can't
> > allow any arbitrary user on the web to trigger jobs with arbitrary code,
> > as that is a significant security risk to your infra.
>
> syzkaller and other upstream CI/fuzzing systems do this, so it may be
> hard but not impossible.

Sure, not impossible, but could not be offered by me at this point. I can't 
promise anything but maybe in the future this can change, or in solution 2 
someone else might resolve the perm issues and I can only assist with the setup 
(if needed).

> > I think I'd just suggest providing a mechanism for the user to easily spin
> > up performance test jobs on their own hardware. This could be as simple
> > as providing a docker container recipe that users can deploy on some
> > arbitrary machine of their choosing that contains the test rig. All they
> > should need do is provide a git ref, and then launching the container and
> > running jobs should be a single command. They can simply run the tests
> > twice, with and without the patch series in question.
>
> As soon as developers need to recreate an environment it becomes
> time-consuming and there is a risk that the issue won't be reproduced.
> That doesn't mean the system is useless - big regressions will still be
> tackled - but I think it's too much friction and we should aim to run
> community builds.

I do understand but unfortunately at this point I can not serve.

> > > The problem with those is that we can not simply use travis/gitlab/...
> > > machines for running those tests, because we are measuring in-guest
> > > actual performance.
> >
> > As mentioned above - distinguish between the CI framework, and the
> > actual test runner.
>
> Does the CI framework or the test runner handle detecting regressions
> and providing historical data? I ask because I'm not sure if GitLab CI
> provides any of this functionality or whether we'd need to write a
> custom CI tool to track and report regressions.

Currently I am using Jenkins which allows to publish result (number of failures 
and total checks) and store artifacts. I am storing the pbench json results 
with metadata (few MBs) and html report (also few MBs). Each html report 
contains a timeline of usually 14 previous builds using them as a reference.

Provided GitLab can do that similarly we should be able to see the number of 
tests run/failed somewhere and then browse the builds html reports. Last but 
not least we can fetch the pbench json results and issue another comparison 
cherry-picking individual results (internally I have a pipeline to do that for 
me, I could add a helper to do that via cmdline/container for others as well).

Regards,
Lukáš

> Stefan