bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:

  1. Fetching substitute info about the things being built so that it
     can determine its references, which in turns allows it to determine
     whether they need to be grafted.

  2. Building stuff, as a last resort, so that it can determine its
     references.

Case #1 is hopefully going to be the most common.

The problem with #1 is that when building a profile, we do one
‘package-derivation’ call for each package in the profile, which
translates in one ‘graft-derivation’ call for each relevant package¹,
which translates into one ‘references/substitutes’ call for each.

Concretely, what this means is this:

  $ guix package -u
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 
100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 
100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 
100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 
100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 
100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 
100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 
100.0%
  […]
  The following files would be downloaded:

Each of the initial “updating list” message corresponds to an HTTP
request for a single narinfo file, which can take around 1 second.

Instead, the ideal thing would be to fetch the narinfo files for all the
relevant packages at once; that way, we’d spawn ‘guix substitute’ only
once, and it would benefit from HTTP pipelining (one round-trip instead
of N.)

To achieve this, I’m thinking of extending gexp code such that gexp
compilers can return a list of applicable grafts.  The ‘package’
compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
‘graft-derivation’.

I’ll give it a try and report back.

Ludo’.

¹ A package is “relevant” if ‘package-grafts’ returns a non-empty list.