Re: how long do we need to retain gitlab CI job stdout logs?

[Daniel P . Berrangé]

On Tue, Aug 09, 2022 at 11:44:52AM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Mon, Aug 08, 2022 at 08:42:28PM +0200, Thomas Huth wrote:
> >> On 08/08/2022 19.47, Peter Maydell wrote:
> >> > Hi; I just reduced QEMU's storage usage on gitlab by 130GB (no typo!)
> >> > using https://gitlab.com/eskultety/gitlab_cleaner, which Dan helpfully
> >> > pointed me at. This script removes old pipelines, which take up a
> >> > lot of storage space for QEMU because they include the stdout logs
> >> > for all the CI jobs in the pipeline. (Gitlab doesn't expire these,
> >> > either by default or configurably -- you have to either manually delete
> >> > the pipeline in the UI or else use the API, as this script does.)
> >> > 
> >> > I somewhat conservatively only blew away pipelines from before the
> >> > 1st January 2022. I feel like we don't really even need 6 months worth
> >> > of CI job logs, though -- any views on whether we should be pruning
> >> > them more aggressively ?
> >> 
> >> I'd say we should at least keep the logs of the last 4 to 5 months, i.e. 
> >> the
> >> logs for one release cycle, so we can check these logs in case we 
> >> introduced
> >> a new bug in the current release cycle.
> >
> > Have we ever actually done this in practice ?  I don't think I've ever
> > looked at a pipeline older than 1-2 weeks in any project I've worked
> > with on gitlab.
> >
> > Note that we currently use 165 GB, over an 8 month period (not sure on
> > the split between container registry and pipeline). I'd guess 4-5 months
> > might knock another 30-40 GB off our usage, still leaving it huge.
> 
> 100GiB is a lot even in 2022.

BTW, frequent users of gitlab CI should check their forks too

  https://gitlab.com/$USERNAME/qemu/-/usage_quotas

I'm a bit of an extreme case since I run sooooo many pipelines when
working on CI configs, but I was using about 450 GB in my fork !

I can recommend Erik's cleaner script linked above, works fine for
forks too.

> > Personally I would suggest 1 month is sufficent for 99% of our needs.
> 
> Makes sense to me.
> 
> If we really need more, maybe look into storing suitable deltas?

We don't really have control over how stuff is stored. GitLab just
captures stdout/err from the jobs and presents that. Our options are
keep it, or delete it. For anything else, we would have to download
it and store it oursdide of gitlab, which doesn't look like its a
good use of time.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|