Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 80 minutes

[Philippe Mathieu-Daudé]

On 11/16/21 17:49, Daniel P. Berrangé wrote:
> On Tue, Nov 16, 2021 at 05:33:09PM +0100, Thomas Huth wrote:
>> The jobs on Cirrus-CI sometimes get delayed quite a bit, waiting to
>> be scheduled, so while the build test itself finishes within 60 minutes,
>> the total run time of the jobs can be longer due to this waiting time.
>> Thus let's increase the timeout on the gitlab side a little bit, so
>> that these jobs are not marked as failing just because of the delay.
> 
> On a successful pipeline I see
> 
>  freebsd-11  - 28 minutes
>  freebsd-12  - 57 minutes
>  macos       - 30 minutes
> 
> We know cirrus allows 2 concurrent jobs, so from that I infer
> that the freebsd-12 job was queued for ~30 minutes waiting for
> either the freebsd-11 or macos job to finish, and then it
> ran in 30 minutes, giving the ~60 minute total.
> 
> That's too close to the 60 minute gitlab default job timeout
> for comfort - it can easily slip over 60 minutes by just a
> small amount.
> 
> 80 minutes will certainly help in the case where we
> randomly take a little longer than 30 minutes to build,
> and have 1 of the 3 jobs queued.
> 
> When we're running jobs on both master + staging, we can
> have 2 jobs running and 4 more queued - 2 of those queued
> might just finish in time, but 2 will definitely fail.
> My patch will cut these extra jobs on master, so in common
> case we only ever get 1 queued, which should work well in
> combo with your patch here. That should be good enough
> for the qemu-project namespace, unless someone is triggering
> pipelines for stable branch staging at the same time as
> the master branch staging.
> 
> If we do want to worry about more than 2 queued jobs
> again for that reason, we might consider putting
> it upto 100 minutes. That would give us enough slack to
> have 4 queued jobs behind two running jobs and have
> them all succeed
> 
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  .gitlab-ci.d/cirrus.yml | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
>> index e7b25e7427..22d42585e4 100644
>> --- a/.gitlab-ci.d/cirrus.yml
>> +++ b/.gitlab-ci.d/cirrus.yml
>> @@ -14,6 +14,7 @@
>>    stage: build
>>    image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
>>    needs: []
>> +  timeout: 80m
>>    allow_failure: true
>>    script:
>>      - source .gitlab-ci.d/cirrus/$NAME.vars
> 
> Whether 80 or 100 minute, consider it
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

This pipeline took 1h51m09s:
https://gitlab.com/qemu-project/qemu/-/pipelines/409666733/builds
But Richard restarted unstable jobs, which probably added time
to the total.

IIRC from a maintainer perspective 1h15 is the upper limit.
80m fits, 100m is over. Up to the project maintainers
(personally I don't have any objection, in particular if
this reduces the failures rate).

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>