[bug#28324] [PATCH 0/2] Allow substitute downloads from unauthorized servers when they are undistinguishable

[Ludovic Courtès]

Hello Guix!

‘guix publish’ is really helpful, but it’s still inconvenient to share
binaries among machines: one has to authorize the other machine’s key,
which is tedious and is an all-or-nothing kind of decision.

Ideally we’d like to support zero-configuration binary sharing on a
local network, for example, and that means it should Just Work without
having to fiddle with keys/ACLs and without having to give blanket
permission to your colleague’s machine next-door.

This patch allows substitutes to be downloaded from an unauthorized
server (one whose public key is not in /etc/guix/acl), or from a server
that does not even sign substitutes, *provided* that server advertises
and serves the exact same content as one of the authorized servers.

Concretely, here’s the metadata substitute servers provide:

--8<---------------cut here---------------start------------->8---
$ wget -q -O - 
https://mirror.hydra.gnu.org/sfx1wh27i6gsrk21p87rdyikc64v7d51.narinfo
StorePath: /gnu/store/sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11
URL: guix/nar/gzip/sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11
Compression: gzip
NarHash: sha256:0ac82i3kn10lgb64d3mn0n062rj291bwjpgv7asn32ja4phri463
NarSize: 384240
References: 3x53yv4v144c9xp02rs64z7j597kkqax-gcc-5.4.0-lib 
n6nvxlk2j8ysffjh3jphn1k5silnakh6-glibc-2.25 
sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11
FileSize: 154865
System: x86_64-linux
Deriver: bpxa7iypl2q2fxzf1dgq9k4daa3p3s39-zlib-1.2.11.drv
Signature: 1;20121227-hydra.gnu.org;KHNpZ25…
--8<---------------cut here---------------end--------------->8---

Let’s look at what another server provides for the same item:

--8<---------------cut here---------------start------------->8---
$ wget -q -O - 
https://berlin.guixsd.org/sfx1wh27i6gsrk21p87rdyikc64v7d51.narinfo
StorePath: /gnu/store/sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11
URL: nar/gzip/sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11
Compression: gzip
NarHash: sha256:0ac82i3kn10lgb64d3mn0n062rj291bwjpgv7asn32ja4phri463
NarSize: 384240
References: 3x53yv4v144c9xp02rs64z7j597kkqax-gcc-5.4.0-lib 
n6nvxlk2j8ysffjh3jphn1k5silnakh6-glibc-2.25 
sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11
FileSize: 154511
System: x86_64-linux
Deriver: 512mdjkpbyp2kggijxcxw0j0xw0scvn4-zlib-1.2.11.drv
Signature: 1;berlin.guixsd.org;KHNpZ25…
--8<---------------cut here---------------end--------------->8---

Apart from the signature, compressed size (“FileSize”), deriver, and
URL, everything is the same.  Thus, even if we only authorized
hydra.gnu.org, it is safe to download from berlin.guixsd.org (in fact,
it’d be enough to have the same “NarHash”) and more generally to
trust the metadata it provides for this store item.

With this patch, one can use:

  --substitute-urls="https://foo.example.org https://hydra.gnu.org";

If foo.example.org serves narinfos equivalent to those of hydra (modulo
signature, URL, compression, etc.), then Guix will download substitutes
from foo.example.org.

Thanks to reproducible builds, that means we’ll effectively be able to
share binaries with pretty much any machine out there.  Pretty neat no?

>From there there are several things we can do:

  1. Add optional service discovery support in ‘guix substitute’ and
     service advertisement in ‘guix publish’ via Guile-Avahi.  Imagine
     going to a GHM or FOSDEM and getting binaries from your neighbor’s
     laptop.  I think that’d be pretty cool.  :-)

  2. Change narinfos such that the signature is computed only over the
     relevant parts: StorePath, NarHash, and References (currently it’s
     computed over all the fields up to “Signature”.)  Not strictly
     necessary, but would make sense.

  3. Change ‘guix publish’ to use content-addressed URLs for nars,
     similar to what is done for /file URLs.

  4. Maybe replace --substitute-urls with two options, say
     --metadata-urls and --nar-urls.  You could, say, fetch
     narinfos only from hydra.gnu.org and fetch nars only from
     foo.example.org (using the content-addressed URLs to get nars from
     foo.example.org.)

  5. Add support to download those content-addressed nars over
     Bittorrent, IPFS, and whatnot.  We can keep downloading narinfos
     themselves over HTTPS.

Sky is the limit.

Until then, feedback welcome!  I’d particularly like for people to read
the patch and the associated unit tests; more eyeballs is a good thing.

Ludo’.

Ludovic Courtès (2):
  substitute: Make substitute URLs a SRFI-39 parameter.
  substitute: Download from unauthorized sources that provide the right
    content.

 doc/guix.texi               |  28 ++++++-
 guix/scripts/substitute.scm | 152 ++++++++++++++++++++++------------
 tests/substitute.scm        | 193 ++++++++++++++++++++++++++++++++++++++++----
 3 files changed, 303 insertions(+), 70 deletions(-)

-- 
2.14.1