bug#57391: "error: connect*: Connection timed out" when using 6+ jobs to fetch substitutes

[Maxim Cournoyer]

Hello,

On a machine configured with 20 builders:

--8<---------------cut here---------------start------------->8---
        (guix-service-type config =>
                           (guix-configuration
                            (inherit config)
                            (build-accounts 20)
                            (discover? #t)
                            (log-compression 'none)
                            (extra-options '("--max-jobs=20"))))
--8<---------------cut here---------------end--------------->8---


Running 'guix package -m manifest.scm' with hundreds of packages/MiB to
fetch from the substitute servers, it fails like:

--8<---------------cut here---------------start------------->8---
$ guix package -m ~/stow/guix/manifest.scm -M7
The following packages will be installed:
   acpi                      1.7
   adb                       7.1.2_r36
   adwaita-icon-theme        40.1.1
   alsa-utils                1.2.4
   ansible                   4.7.0
   anthy                     9100h
   arc-icon-theme            20161122
   arc-theme                 20210412
[...]
   xournal                   0.4.8.2016
   xrandr                    1.5.1
   xrdb                      1.2.1
   xsetroot                  1.1.2
   yelp                      41.0

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
924.3 MB will be downloaded

substitution of /gnu/store/1w54x244cykra3p2bmmxv9jn12q92l0r-gimp-2.10.32-doc 
complete

[...]

substitution of /gnu/store/adq43ck5b12sx0n9hy4hgay4gk7spy5b-libxml++-2.40.1 
complete

substitution of /gnu/store/i379f1vadva2lnk1jh60c03s26m6rj7b-ibus-1.5.24-doc 
complete

substitution of 
/gnu/store/9j1hfzgpllgshy6qn82pl8zkkgn4058w-hackneyed-x11-cursors-0.8.2 complete

substitution of 
/gnu/store/bl9y57k0wg3aqxb8xiwddagj3i4dk036-perl-b-hooks-endofscope-0.24 
complete

substitution of /gnu/store/qpc614b2hbzcsjhac38m9av3cqnrwjqi-perl-datetime-1.54 
complete

substitution of /gnu/store/r07xvd187my02gg5smlw4pis90x7m3ah-opensp-1.5.2 
complete
guix substitute: warning: while fetching 
https://ci.guix.gnu.org/nar/lzip/q7hiikl3q3dk1r46rlyyzd90yaddydrd-gnucash-docs-4.10:
 server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
guix substitute: error: connect*: Connection timed out
substitution of /gnu/store/q7hiikl3q3dk1r46rlyyzd90yaddydrd-gnucash-docs-4.10 
failed
guix package: error: corrupt input while restoring archive from socket
--8<---------------cut here---------------end--------------->8---

It fails like the above when the number of jobs is 6 or higher, but
proceeds without error when the number of jobs is 5 or less.

I do not think the network to be at caused, as everything is cabled and
the link is healthy and fast.

Thanks,

Maxim