bug#67575: guix-daemon assertion failure upon substitution failure

[Ludovic Courtès]

On a build machine running ‘cuirass remote-server’ with
/gnu/store/5g4c9kii8l50lw23s7yb2a4bvc7qzw5d-guix-1.4.0-10.4dfdd82/bin/guix-daemon
(from Guix commit 2ab5e449246f98b049888dde3c310f5b4a0a64a2), I see these
‘cuirass remote-worker’ errors:

--8<---------------cut here---------------start------------->8---
2023-11-27 06:49:54 CwQfwEE3: building derivation 
`/gnu/store/rivlxrqr10g3kj9knyz7i41aq1vzxn9z-r-ecoregime-0.1.3.drv' (system: 
x86_64-linux)
2023-11-27 06:50:11 error: CwQfwEE3: unexpected error while building 
'/gnu/store/rivlxrqr10g3kj9knyz7i41aq1vzxn9z-r-ecoregime-0.1.3.drv': 
#<&nar-error file: #f port: #<input-output: socket 52>>
[…]
2023-11-27 07:00:58 CwQfwEE3: building derivation 
`/gnu/store/b5qrpwpx5sbmlj5f7nx6wknnkczadjcn-r-cpp-0.1.0.drv' (system: 
x86_64-linux)
2023-11-27 07:01:20 error: CwQfwEE3: unexpected error while building 
'/gnu/store/b5qrpwpx5sbmlj5f7nx6wknnkczadjcn-r-cpp-0.1.0.drv': #<&nar-error 
file: #f port: #<input-output: socket 52>>
--8<---------------cut here---------------end--------------->8---

… corresponding to these guix-daemon messages:

--8<---------------cut here---------------start------------->8---
2023-11-27 06:49:54 accepted connection from pid 14007, user root
2023-11-27 06:50:11 guix-daemon: nix/libstore/build.cc:3322: void 
nix::Worker::childTerminated(pid_t, bool): Assertion `i != children.end()' 
failed.
[…]
2023-11-27 07:00:58 accepted connection from pid 14007, user root
2023-11-27 07:01:20 guix-daemon: nix/libstore/build.cc:3322: void 
nix::Worker::childTerminated(pid_t, bool): Assertion `i != children.end()' 
failed.
--8<---------------cut here---------------end--------------->8---

The corresponding build log (sent by ‘cuirass remote-worker’ to the
server):

--8<---------------cut here---------------start------------->8---
$ zcat /var/cache/cuirass/remote/rivlxrqr10g3kj9knyz7i41aq1vzxn9z.log.gz |tail
@ substituter-started 
/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz substitute
guix substitute: warning: while fetching 
https://ci.guix.gnu.org/nar/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz:
 server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
retrying download of 
'/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz' with other 
substitute URLs...
guix substitute: warning: while fetching 
https://ci.guix.gnu.org/nar/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz:
 server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
guix substitute: error: failed to find alternative substitute for 
'/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz'
fetching path 
`/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz' (empty 
status: '')
@ substituter-failed 
/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz  fetching 
path `/gnu/store/gymm7n27rfhyz83zx3lw0bwg4m6alpzw-ecoregime_0.1.3.tar.gz' 
(empty status: '')
fetching path `/gnu/store/yiqzanx7vb8s0sj9h15p2llxwrkv0xzh-r-descriptio-1.2'...

$ zcat /var/cache/cuirass/remote/b5qrpwpx5sbmlj5f7nx6wknnkczadjcn.log.gz |tail
substitute: updating substitutes from 'http://194.199.1.1:5557'... 100.0%
guix substitute: warning: while fetching 
https://ci.guix.gnu.org/nar/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz: 
server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
retrying download of 
'/gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz' with other 
substitute URLs...
guix substitute: warning: while fetching 
https://ci.guix.gnu.org/nar/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz: 
server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
guix substitute: error: failed to find alternative substitute for 
'/gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz'
fetching path `/gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz' 
(empty status: '')
@ substituter-failed 
/gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz  fetching path 
`/gnu/store/iqi6s452hqy3k3n335c5v9nb6wq8h9nn-CPP_0.1.0.tar.gz' (empty status: 
'')
fetching path `/gnu/store/b7rnd5y4zsc1995h0a8w294pzdgbmhr9-mc2d_0.2.0.tar.gz'...
--8<---------------cut here---------------end--------------->8---

In both cases, “failed to find alternative substitute” leads ‘guix
substitute’ to exit, which guix-daemon doesn’t really cope with.

Ludo’.