bug#54447: cuirass: missing derivation error

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> Mathieu Othacehe <othacehe@gnu.org> skribis:
>
>> A lot of builds, among them ~20 system tests[1], are failing with:
>> "cannot build missing derivation
>> ?/gnu/store/hs6kp1lqgymhyp3jndc0dsp0pn4psgv0-gui-installed-desktop-os-encrypted.drv?"
>> errors.
>>
>> Those derivations are present on the CI head node. This means that the
>> errors occur during substitution. This is most likely caused by some
>> issue with the publish server, because:
>>
>> - The publish server serves a 404 error. We should get rid once and for
>>   all of this 404 thing, pushing something like:
>>   https://issues.guix.gnu.org/50040.
>>
>> or
>>
>> - The publish server is not fast enough and hits an Nginx timeout that
>>   closes the communication.
>
> Also being discussed at <https://issues.guix.gnu.org/48468#12>.

I got confirmation that the cache-bypass-threshold hypothesis holds, at
least for system tests.

Namely, looking at <https://ci.guix.gnu.org/build/2258097/details>,
which ends like this:

--8<---------------cut here---------------start------------->8---
@ substituter-succeeded /gnu/store/qh2876i5l1wvxgwhg9fbl9zmb3px3n2m-gc-roots.drv
fetching path 
`/gnu/store/fh9dnmrfsz429pwqmvsjnk0snlm959kc-xdg-mime-database-builder'...
@ substituter-started 
/gnu/store/fh9dnmrfsz429pwqmvsjnk0snlm959kc-xdg-mime-database-builder substitute
Downloading 
http://141.80.167.131/nar/lzip/fh9dnmrfsz429pwqmvsjnk0snlm959kc-xdg-mime-database-builder...
. xdg-mime-database-builder                    3.6MiB/s 00:00 | 3KiB 
transferred. xdg-mime-database-builder                    1.9MiB/s 00:00 | 
3KiB transferred

@ substituter-succeeded 
/gnu/store/fh9dnmrfsz429pwqmvsjnk0snlm959kc-xdg-mime-database-builder
cannot build missing derivation 
‘/gnu/store/4r1wij3bzj9zv75ds82a93jl7bcman2x-installed-extlinux-os.drv’
--8<---------------cut here---------------end--------------->8---

Looking at the nginx and ‘guix publish’ logs, I found that the missing
substitute is not that of 4r1wij3bzj9zv75ds82a93jl7bcman2x (the .drv
itself) but rather that of a dependency of that .drv:

  [14/Oct/2023:23:22:09 +0200] "GET /wqqzcxrhbnv0nzg64iiiqd5grr4vk2zg.narinfo 
HTTP/1.1" 404 58 "-" "GNU Guile"

That item’s size is above the cache bypass threshold of 100 MiB as
currently configured on berlin:

--8<---------------cut here---------------start------------->8---
$ du -hs /gnu/store/wqqzcxrhbnv0nzg64iiiqd5grr4vk2zg-guix-5a6b1a5
124M    /gnu/store/wqqzcxrhbnv0nzg64iiiqd5grr4vk2zg-guix-5a6b1a5
--8<---------------cut here---------------end--------------->8---

The immediate fix/workaround is to raise that threshold.

A better solution would be for system tests to depend on a fixed-output
derivation for the Guix source instead of the “source” above (I use
“source” as it is used in the context of <derivation>).

Thanks,
Ludo’.