[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug#47336] Disarchive as a fallback for downloads
From: |
Ludovic Courtès |
Subject: |
[bug#47336] Disarchive as a fallback for downloads |
Date: |
Fri, 14 May 2021 23:36:21 +0200 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) |
Hi!
Timothy Sample <samplet@ngyro.com> skribis:
> This patch series adds Disarchive assembly (backed by SWH lookup) as a
> fallback for downloads.
>
> To try it, make sure you are running the daemon in an environment with
> Disarchive available:
>
> $ ./pre-inst-env guix environment --ad-hoc guile disarchive
> # ./pre-inst-env guix-daemon --build-users-group=guixbuild
>
> Don’t forget to stop your existing Guix Daemon. :)
>
> You also need to make sure that regular downloads are unavailable. I do
> this by adjusting the “try” loop at the end of “url-fetch” in
> “guix/build/download.scm”. I replace the usual list of URLs with ‘()’:
>
> (let try ((uri (append uri content-addressed-uris)))
> (match '() ; uri
> ...))
>
> Now you can ask Guix for a recent .tar.gz source package:
>
> $ ./pre-inst-env guix build --no-substitutes -S python-httpretty
>
> You should see:
>
> Trying to use Disarchive to assemble
> /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty-1.0.5.tar.gz
> Assembling the directory httpretty-1.0.5
> Downloading from Software Heritage...
> 7903d608efc89c14afb4d692a3721156e31a43e2/
> 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/
> 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING
> [...]
> Checking httpretty-1.0.5 digest... ok
> Assembling the tarball httpretty-1.0.5.tar
> Checking httpretty-1.0.5.tar digest... ok
> Assembling the Gzip file httpretty-1.0.5.tar.gz
> Checking httpretty-1.0.5.tar.gz digest... ok
> Copying result to
> /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty-1.0.5.tar.gz
> successfully built
> /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpretty-1.0.5.tar.gz.drv
Commits 67bf61255414115ffae0141df9dd3623bc742bff and
0b1f70d1a792af40aa0d13b3d227fde88f02d061 add the dependency on
Disarchive, so this fallback path is now enabled!
> There’s lots to talk about though....
>
> First, it looks up the metadata on my server. This is fine for a demo,
> but not what we want forever. The patch series supports adding several
> mirrors for looking up the metadata. In the past, we talked about
> putting everything on one or a few of the big Git hosting platforms like
> GitHub or Gitlab. That way, it would be easily picked up by SWH and
> archived “forever”. Right now, I have Cuirass set up to build the
> metadata, and a little script that moves it from the build server to my
> Web server. It would be simple enough to adjust that script to push it
> to a remote Git repo. (Of course, the next step is to move this setup
> to Guix infrastructure.) Thoughts?
We should talk to SWH, giving them the figures you gave earlier in this
thread. But yeah, a Git repo looks best to me (it would be useful to
keep track of changes, for example if we eventually update metadata to a
new format) and it simplifies archival to SWH.
Second thing we need to figure out if where to create this database. If
you have a Cuirass job already, we should run it on ci.guix. WDYT?
> On the code level, there were two things I couldn’t figure out for
> myself.
>
> I made the mirror list just simple strings. AIUI, the client and the
> daemon have to agree about the format of the mirror list. Given that
> running old daemons is common, changing the format is difficult. Is it
> worth it to copy the more flexible interface used by the content
> addressed mirrors? If yes, do I have to do the same ‘module-autoload!’
> dance to use ‘bytevector->base16-string’? :) (I probably would have
> just copied it, but that part confused me a bit.)
I had overlooked this suggestion of yours. Yes, I think it’s best to
copy the SWH scheme. Don’t worry about ‘module-autoload!’: nowadays we
can safely assume (guix base16) is available.
When we change from list-of-strings to list-of-procedures, we’ll have to
adjust the (guix build download) code so that it can deal with both.
> I imported some modules from “guix/build/download.scm” (well, just
> “base16” and “swh”). It feels weird to use a bunch of host-side modules
> from what’s nominally a “guix/build” module. This is okay because
> “guix/build/download.scm” is not /really/ build-side code. It’s more
> like daemon (-ish) code that just happens to live in “guix/build”, which
> is why importing host-side modules is OK... right?
Yup. :-) In the end, the whole point is to reuse code on both sides,
and that’s what’s being done here.
Thanks,
Ludo’.
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- [bug#47336] Disarchive as a fallback for downloads,
Ludovic Courtès <=