bug#39885: Bioconductor tarballs are not archived

[Simon Tournier]

Hi,

On Fri, 22 Dec 2023 at 14:40, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>> guix build: error: build of 
>> `/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv' 
>> failed

First thing first, please note that we are speaking about tag 1.4.1 and
not 1.20.0.  And this 1.4.1 is gone from “our” infra since… ?? That’s
one of the things I do not like with Guix: I never know what to expect
from the infra.  Anyway, I have my list of TODOs for improving the
annoyances (I and maybe others have :-)); stay tuned. ;-)

Considering the state of “our” infra and how Bioconductor manages the
tarballs, many tarballs are lost forever, sadly.  Although the content
is still around, I guess.


> I was wondering whether we’re now doing better for Bioconductor
> tarballs.  The answer, based on small sample, seems to be “not quite”:

Thanks for diving into this.


> <https://guix.gnu.org/sources.json> has entries like:
>
> --8<---------------cut here---------------start------------->8---
>     {
>       "type": "url",
>       "urls": [
>         
> "https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz";,
>         
> "https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz";,
>         
> "https://bordeaux.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas";,
>         
> "https://ci.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas";,
>         
> "https://tarballs.nixos.org/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas";
>       ],
>       "integrity": "sha256-WlUXSI41dP7xSTx81ibFsrIsACW5jmzaX5I/lxJ4vCg=",
>       "outputHashAlgo": "sha256",
>       "outputHashMode": "flat"
>     },
> --8<---------------cut here---------------end--------------->8---

Please note that Bioconductor 3.18 released BiocNeighbors v1.20.0 but
then updated to v1.20.1 still under Bioconductor 3.18 and Ricardo did
this update with 5673484cbc2ed74c61ae81d623646fa7829fbc32.  On a side
note, between the Bioconductor update and the update on our side, there
is a mismatch where the source of r-biocneighbors is unreachable.

Other said, post-update on our side,

--8<---------------cut here---------------start------------->8---
$ zcat sources.json | jq | grep BiocNeighbors | grep bioconductor | sed 
's/"//g' | sed 's/,//g'
        
https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.1.tar.gz
        
https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.1.tar.gz

$ for url in $(zcat sources.json | jq | grep BiocNeighbors | grep bioconductor 
| sed 's/"//g' | sed 's/,//g'); \
     do guix download $url ;done

Starting download of /tmp/guix-file.STc9fQ
>From 
>https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.1.tar.gz...
 …_1.20.1.tar.gz  1015KiB                                                       
                                  60.3MiB/s 00:00 ▕██████████████████▏ 100.0%
/gnu/store/nxab1pskh9zcjspczph6jcs5fk79pb7k-BiocNeighbors_1.20.1.tar.gz
0w7hd6w0lmj1jaaq9zd5gwnnpkzcr0byqm5q584wjg4xgvsb981j

Starting download of /tmp/guix-file.aZFRLv
>From 
>https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.1.tar.gz...
 …_1.20.1.tar.gz  1015KiB                                                       
                                  63.5MiB/s 00:00 ▕██████████████████▏ 100.0%
/gnu/store/nxab1pskh9zcjspczph6jcs5fk79pb7k-BiocNeighbors_1.20.1.tar.gz
0w7hd6w0lmj1jaaq9zd5gwnnpkzcr0byqm5q584wjg4xgvsb981j
--8<---------------cut here---------------end--------------->8---

but, now the past reads,

--8<---------------cut here---------------start------------->8---
$ for url in 
https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz
 \
             
https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz
 ;  \
      do guix download $url ;done

> > 
Starting download of /tmp/guix-file.MUB3ow
>From 
>https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz...
download failed 
"https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz";
 404 "Not Found"

Starting download of /tmp/guix-file.MUB3ow
>From 
>https://web.archive.org/web/20240102105016/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz...
download failed 
"https://web.archive.org/web/20240102105016/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz";
 404 "NOT FOUND"
Trying to use Disarchive to assemble /tmp/guix-file.MUB3ow...
could not find its Disarchive specification
failed to download "/tmp/guix-file.MUB3ow" from 
"https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz";
guix download: error: 
https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz:
 download failed

Starting download of /tmp/guix-file.ZO9N08
>From 
>https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz...
download failed 
"https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz";
 404 "Not Found"

Starting download of /tmp/guix-file.ZO9N08
>From 
>https://web.archive.org/web/20240102105018/https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz...
download failed 
"https://web.archive.org/web/20240102105018/https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz";
 404 "NOT FOUND"
Trying to use Disarchive to assemble /tmp/guix-file.ZO9N08...
could not find its Disarchive specification
failed to download "/tmp/guix-file.ZO9N08" from 
"https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz";
guix download: error: 
https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz:
 download failed
--8<---------------cut here---------------end--------------->8---

As explained in [1], Bioconductor removes v1.20.0 from their URI scheme
and despite the fact Bioconductor v3.18 had released v1.20.0, arf!  And
I do not even know if the tarball of v1.20.0 is kept on Bioconductor
infra.  Hum?

Hence the discussion we had: switch from url-fetch to git-fetch.
However, after some investigations, it does not seem straightforward:
The main issue being the almost automatic current updater.  See for
details [2].


1: bug#39885: Bioconductor URI, fallback and time-machine
zimoun <zimon.toutoune@gmail.com>
Mon, 18 Jul 2022 18:03:04 +0200
id:87lesqmmrr.fsf@gmail.com
https://issues.guix.gnu.org/39885
https://issues.guix.gnu.org/msgid/87lesqmmrr.fsf@gmail.com
https://yhetil.org/guix/87lesqmmrr.fsf@gmail.com

2: bug#39885: Bioconductor URI, fallback and time-machine
Ricardo Wurmus <rekado@elephly.net>
Wed, 10 Aug 2022 20:25:00 +0200
id:878rnwuemq.fsf@elephly.net
https://issues.guix.gnu.org/39885
https://issues.guix.gnu.org/msgid/878rnwuemq.fsf@elephly.net
https://yhetil.org/guix/878rnwuemq.fsf@elephly.net


> Any volunteer to write a script that’d generate a list of Bioconductor
> content-addressed URLs (the bordeaux.guix.gnu.org/file ones) for say the
> past couple of years?

I did stuff past week about that.  I will report this week what I did.

Cheers,
simon