[bug#36029] [PATCH 0/2] 'publish' and 'substitute' support several compression methods

[Ludovic Courtès]

Hello Guix!

This is a followup to <https://issues.guix.gnu.org/issue/35880>.

One idea we discussed there was to allow clients to pass an
‘X-Guix-Accepted-Encoding’ header in HTTP requests, and the server
would return an lzip narinfo or a gzip narinfo depending on that.
However, I thought that this was not very flexible, and that we
were bound to mess up with caching.

This patch implements a different solution: ‘guix publish’ can
be passed multiple ‘-C’ options, in which case it compresses
substitutes with all these compression methods.  The corresponding
narinfo looks like this:

--8<---------------cut here---------------start------------->8---
StorePath: /gnu/store/9czlz7ss3187l2vi1hvrlkwlgrggdg5p-inkscape-0.92.4
URL: nar/gzip/9czlz7ss3187l2vi1hvrlkwlgrggdg5p-inkscape-0.92.4
Compression: gzip
FileSize: 40308611
URL: nar/lzip/9czlz7ss3187l2vi1hvrlkwlgrggdg5p-inkscape-0.92.4
Compression: lzip
FileSize: 19867767
NarHash: sha256:1jv4nkq68a7zwqhi9inrnh340a4jxcpb91wq7d25hgw0nk8isbbk
NarSize: 136499024
References: …
--8<---------------cut here---------------end--------------->8---

IOW, it’s like before, except that there are multiple
URL/Compression/FileSize fields instead of just one of each.

The trick is that old clients take the first occurrence of each
of these fields and ignore subsequent occurrences.  In the example
above, they’d just take gzip and ignore the rest.

The new ‘guix substitute’ (second patch) “sees” all these fields
and is able to choose the most appropriate compression method (i.e.,
the best one among those it supports.)

This adds a bit of complexity that is useless beyond the transitioning
period from gzip to lzip, but I think that’s OK; plus there might be
an lzip to super-lzip transition in the future, who knows.

Thoughts?

When we deploy that, we’ll obviously more use storage and more CPU on
the build farm, but that seems unavoidable.  OTOH, we’ll progressively
end up sending less data over the wire (and paying less for the CDN!),
given that lzip compresses better.

Ludo’.

Ludovic Courtès (2):
  publish: '--compression' can be repeated.
  substitute: Select the best compression methods.

 doc/guix.texi               |   5 +
 guix/scripts/challenge.scm  |   4 +-
 guix/scripts/publish.scm    | 204 ++++++++++++++++++++++--------------
 guix/scripts/substitute.scm | 141 ++++++++++++++++++-------
 guix/scripts/weather.scm    |   5 +-
 tests/publish.scm           |  89 ++++++++++++++--
 tests/substitute.scm        |  51 ++++++++-
 7 files changed, 370 insertions(+), 129 deletions(-)

-- 
2.21.0