Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 80 minutes

[Daniel P . Berrangé]

On Tue, Nov 16, 2021 at 06:36:50PM +0100, Richard Henderson wrote:
> On 11/16/21 6:22 PM, Thomas Huth wrote:
> > On 16/11/2021 18.09, Philippe Mathieu-Daudé wrote:
> > > On 11/16/21 17:49, Daniel P. Berrangé wrote:
> > > > On Tue, Nov 16, 2021 at 05:33:09PM +0100, Thomas Huth wrote:
> > > > > The jobs on Cirrus-CI sometimes get delayed quite a bit, waiting to
> > > > > be scheduled, so while the build test itself finishes within 60 
> > > > > minutes,
> > > > > the total run time of the jobs can be longer due to this waiting time.
> > > > > Thus let's increase the timeout on the gitlab side a little bit, so
> > > > > that these jobs are not marked as failing just because of the delay.
> > ...>>> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > > > > index e7b25e7427..22d42585e4 100644
> > > > > --- a/.gitlab-ci.d/cirrus.yml
> > > > > +++ b/.gitlab-ci.d/cirrus.yml
> > > > > @@ -14,6 +14,7 @@
> > > > >     stage: build
> > > > >     image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> > > > >     needs: []
> > > > > +  timeout: 80m
> > > > >     allow_failure: true
> > > > >     script:
> > > > >       - source .gitlab-ci.d/cirrus/$NAME.vars
> > > > 
> > > > Whether 80 or 100 minute, consider it
> > > > 
> > > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> > > 
> > > This pipeline took 1h51m09s:
> > > https://gitlab.com/qemu-project/qemu/-/pipelines/409666733/builds
> > > But Richard restarted unstable jobs, which probably added time
> > > to the total.
> > > 
> > > IIRC from a maintainer perspective 1h15 is the upper limit.
> > > 80m fits, 100m is over.
> > 
> > I think I agree ... I normally don't want to wait more than a little bit
> > more than one hour, so 100 minutes feels too long already. We already
> > have some 70m timeouts in other jobs, and one 80 minute timeout in
> > .gitlab-ci.d/crossbuild-template.yml, so I'd say 80 minutes are really
> > the upper boundary that we should use.
> 
> We are also talking apples and oranges:
> Gitlab timeouts are on the amount of time the job runs.
> Cirrus timeouts appear to be on the amount of time the job is queued.
> 
> If cirrus would just not start accounting until the thing runs we'd be fine.

Unfortunately it isn't that easy. Our cirrus CI jobs are launched using
the cirrus-run tool, from a gitlab job. The timeouts we're usually
hitting are from the gitlab job which is sitting around waiting for
the cirrus job it launched to finish, so it can report back stats.

Cirrus CI does itself have a job timeout, but I'm not aware of us
hitting that typically, unless i'm misinterpreting something.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|