bug#26201: No notification of cache misses when downloading substitutes

[dian_cecht]

On Tue, 21 Mar 2017 04:57:09 +0100
Tobias Geerinckx-Rice <address@hidden> wrote:

> Ahoy,
> 
> On 21/03/17 03:52, address@hidden wrote:
> > The URL displayed during the download was mirror.hydra.gnu.org.
> > [...] It was a binary download, not source.  
> 
> Oh, OK. I'm not an expert on how Hydra's set up these days, but will
> assume it's not too different from my own (a fast nginx proxy_cache,
> mirror.hydra.gnu.org, in front of a slower build farm, hydra.gnu.org).
> 
> Whenever you're the first to request a substitute,
> mirror.hydra.gnu.org transparently forwards the request to
> hydra.gnu.org.
> 
> The latter has to compress the response on the fly, leading to much
> slower transfer speeds. It slowly sends it back to the mirror, which
> slowly sends it on to you while also saving it on disc so all
> subsequent downloads will be fast — by Hydra standards – and not
> involve hydra.gnu.org.
> 
> Maybe you knew all this, but it's also the reason that...

I'm not familiar with the implementation details, nor how hydra is
currently setup.

> > On 21/03/17 02:44, address@hidden wrote:
> > It would be nice if there was some notification that a cache miss
> > happened and the download will likely be slow, otherwise a user
> > might wonder what problem there is with their connection.  
> 
> ...I'm afraid this makes no sense from guix's point of view.
> 
> The term ‘cache miss’ here is an implementation detail of our current
> Hydra set-up, not something guix can or IMO should care about. There
> are hundreds of reasons why your connection might be slow at any
> given time. Guix should just tell you so (it does), not guess why. Or
> worse: know.

I'm not suggesting having Guix tell me why my network is slow, only if
the download might be slow because it's having to pull from
hydra.gnu.org. Having Guix automagically troubleshoot networking
problems is well beyond the scope of a package manager, even one that
goes as far beyond simple package management as Guix does.

> 
> (But if others disagree, we'll have to extend the Hydra API to somehow
> relay this information to the client, in the spirit of the modern
> Web.)

AFAIK, Guix devs are working on a replacement for the current build
system, so the sane option wouldn't be extending the current hydra
system to handle a new API call, but to try and work this type of
feature into the next system. Unless, of course, something like this
could be done in hydra reasonably easily, in which case why not.

Another option would be to have the mirrors automatically cache the
files as soon as they are available to try. I'd hope this would be how
things are handled already, but one never knows.