On Thu, Jun 25, 2020 at 12:26:53PM +0100, Daniel P. Berrangé wrote:
On Thu, Jun 25, 2020 at 01:15:52PM +0200, Thomas Huth wrote:
On 22/06/2020 17.33, Daniel P. Berrangé wrote:
The current gitlab CI jobs are quite inefficient because they
use the generic distro images and then apt-get/dnf install
extra packages every time.
The other downside is that the container environment used is
only defined in thte .gitlab-ci.yml file, so it tedious to
reproduce locally.
We already have containers defined in tests/docker for use by
developers building locally. We can use these for CI systems
too if we just had a way to build them....
...GitLab CI offers such a way. We can use docker-in-docker
to build the images at the start of the CI cycle, and use
the built images in later jobs.
These later jobs are now faster because they're not having
to install any software.
Did you see any speed-up? I had a look at some pipelines, and it seems to me
that they rather got slower now? For example, this is the system1 pipeline
before your change:
https://gitlab.com/huth/qemu/-/jobs/610924897
and after your change:
https://gitlab.com/huth/qemu/-/jobs/611069374
Duration went up from 35 minutes to 42 minutes.
Seems also to happen in your builds, before the change:
https://gitlab.com/berrange/qemu/-/jobs/582995084
and after the change:
https://gitlab.com/berrange/qemu/-/jobs/606175927
... went from 36 minutes up to 42 minutes.
Could be a coincidence due to the load on the shared runners, but it looks
at least a little bit suspicious...
I think the difference is because we're building more features now. The
dockerfiles have provided more build pre-requisites that the old gitlab
recipe did.
If you compare the configure summary, I see the new build now covers
SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs
and a few more. So we've saved time by not intsallling many packages
each time, but consumed a greater amount of time by compiling more
features.
Oh a missed a lot more actually - there's also spice, opengl, libiscsi,
libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev
too.