Re: Experiment in generating multi-layer Docker images with guix pack

[Ludovic Courtès]

Hello Chris,

Christopher Baines <address@hidden> skribis:

> These patches are very rough, and not ready, but do at least work in some
> limited capacity. I've been testing with the following commands:
>
>   guix pack --format=docker guile@2.2.6
>   guix pack --format=docker guile@2.2.7
>
> With the previous Docker image generation implementation, two different ~130MB
> images would be generated. These patches mean that each .tar.gz file generated
> by guix pack contains a ~53MB layer which contains the profile and directly
> referenced store items, and then a ~77MB layer with all the other store items
> which is identical for both the 2.2.6 and 2.2.7 pack file.

Nice!

> I think it could be useful to support multiple different strategies for
> generating layers for Docker images, with different trade-offs. This approach
> using two layers should make the resulting images more efficient to use in the
> case where like the guile example above, where the packages you run guix pack
> with have exactly matching inputs.

Did you read <https://grahamc.com/blog/nix-and-layered-docker-images>?
They came up with a pretty smart algorithm that would be worth copying.

> This could often be the case if you're developing an application, packaging it
> with Guix, then using guix pack to generate a Docker image which you
> deploy. With the single layer approach, if you change the application code,
> you'll get an entirely different image. I haven't tried this out, but my hope
> is that by generating a common base layer, if you change the application code
> only the top layer of the Docker image will change, meaning you'll only have
> to deploy that, rather than having to deploy the entire image. If you're
> deploying the images across a network, having less data to send around can
> save time, and reduce the amount of space required to store the images.

Definitely.

> As well as these behaviour changes, these patches also modify the
> implementation. Rather than having some build side code that's used in the
> pack and vm module gexpressions, these patches introduce two new record types:
> <docker-image-layer> and <docker-image>. This at least structures the
> derivations so that each layer is represented by a derivation, and then
> there's a derivation for the image itself, which is a little more efficient in
> terms of computation.

Nice.

I think a layering algorithm like Graham Christensen’s above requires
knowledge of the reference graph, meaning that layering can only be
computed on the build side, using #:references-graphs.  In that case, it
could be that you can’t have a host-side <docker-image-layer> record.

> What do people think about generating multi-layer images, and using record
> types to represent the layers and image?

I think multi-layering is something we should definitely have, and
record for at least the image are a good idea.  :-)

Thanks for looking into this!

Ludo’.