--- Begin Message ---
|
Subject: |
Grafts leads to inefficient substitute info retrieval |
|
Date: |
Fri, 11 Mar 2016 17:52:59 +0100 |
|
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) |
As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
1. Fetching substitute info about the things being built so that it
can determine its references, which in turns allows it to determine
whether they need to be grafted.
2. Building stuff, as a last resort, so that it can determine its
references.
Case #1 is hopefully going to be the most common.
The problem with #1 is that when building a profile, we do one
‘package-derivation’ call for each package in the profile, which
translates in one ‘graft-derivation’ call for each relevant package¹,
which translates into one ‘references/substitutes’ call for each.
Concretely, what this means is this:
$ guix package -u
substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
100.0%
substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
100.0%
[…]
The following files would be downloaded:
Each of the initial “updating list” message corresponds to an HTTP
request for a single narinfo file, which can take around 1 second.
Instead, the ideal thing would be to fetch the narinfo files for all the
relevant packages at once; that way, we’d spawn ‘guix substitute’ only
once, and it would benefit from HTTP pipelining (one round-trip instead
of N.)
To achieve this, I’m thinking of extending gexp code such that gexp
compilers can return a list of applicable grafts. The ‘package’
compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
‘graft-derivation’.
I’ll give it a try and report back.
Ludo’.
¹ A package is “relevant” if ‘package-grafts’ returns a non-empty list.
--- End Message ---
--- Begin Message ---
|
Subject: |
Re: bug#22990: Grafts leads to inefficient substitute info retrieval |
|
Date: |
Sun, 29 Mar 2020 15:41:24 +0200 |
|
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) |
Hi younger self!
address@hidden (Ludovic Courtès) skribis:
> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>
> 1. Fetching substitute info about the things being built so that it
> can determine its references, which in turns allows it to determine
> whether they need to be grafted.
>
> 2. Building stuff, as a last resort, so that it can determine its
> references.
>
> Case #1 is hopefully going to be the most common.
>
> The problem with #1 is that when building a profile, we do one
> ‘package-derivation’ call for each package in the profile, which
> translates in one ‘graft-derivation’ call for each relevant package¹,
> which translates into one ‘references/substitutes’ call for each.
>
> Concretely, what this means is this:
>
> $ guix package -u
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
> 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
> 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
> 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
> 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
> 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
> 100.0%
> substitute: updating list of substitutes from 'http://mirror.guixsd.org'...
> 100.0%
> […]
> The following files would be downloaded:
>
> Each of the initial “updating list” message corresponds to an HTTP
> request for a single narinfo file, which can take around 1 second.
This is finally fixed with commit
710854304b1ab29332edcb76f3de532e0724c197, as discussed earlier this
week¹. \o/
The solution is simpler and more orthogonal than what was envisioned
earlier in this thread, which is nice.
I also potentially addresses other uses of “dynamic dependencies”
elsewhere in the code base, such as ‘remote-eval’ calls in ‘guix
deploy’.
Feedback welcome!
Ludo’.
¹ https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00337.html
--- End Message ---