bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds (was: New linux-libre failed to build on armhf on Berlin)

[Mark H Weaver]

retitle 36754 SSH connections to hydra-slave{1,2,3} fail during builds
thanks

Hi,

I've added Ludovic to the CC list, since he recently added
hydra-slave{1,2,3} to Berlin.

Marius wrote:
> I tried building 5.2.2 'interactively' on Berlin, and got an SSH error:
> 
>   CC [M]  net/openvswitch/vport-geneve.o
>   CC [M]  net/openvswitch/vport-gre.o
>   LD [M]  net/openvswitch/openvswitch.o
> ;;; [2019/07/23 05:14:53.501502, 0] read_from_channel_port: [GSSH ERROR] 
> Error reading from the channel: #<input-output: channel (closed) 14c0e60>
> Backtrace:
>           16 (apply-smob/1 #<catch-closure b79640>)
> In ice-9/boot-9.scm:
>     705:2 15 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
> In ice-9/eval.scm:
>     619:8 14 (_ #(#(#<directory (guile-user) bfb140>)))
> In guix/ui.scm:
>   1747:12 13 (run-guix-command _ . _)
> In guix/scripts/offload.scm:
>    781:22 12 (guix-offload . _)
> In ice-9/boot-9.scm:
>     829:9 11 (catch _ _ #<procedure 7f576678d910 at guix/ui.scm:703…> …)
>     829:9 10 (catch _ _ #<procedure 7f576678d928 at guix/ui.scm:826…> …)
> In guix/scripts/offload.scm:
>    580:19  9 (process-request _ _ _ _ #:print-build-trace? _ # _ # _)
>     531:6  8 (call-with-timeout _ _ _)
>     361:2  7 (transfer-and-offload #<derivation /gnu/store/yfns7ga4…> …)
> In ice-9/boot-9.scm:
>     829:9  6 (catch _ _ #<procedure dbdab0 at guix/scripts/offload.…> …)
> In guix/scripts/offload.scm:
>     385:6  5 (_)
> In guix/store.scm:
>   1203:15  4 (_ #<store-connection 256.99 19a0ba0> _ _)
>    692:11  3 (process-stderr #<store-connection 256.99 19a0ba0> _)
> In guix/serialization.scm:
>     87:11  2 (read-int _)
>     73:12  1 (get-bytevector-n* #<input-output: channel (closed) 14…> …)
> In unknown file:
>            0 (get-bytevector-n #<input-output: channel (closed) 14c…> …)
> 
> ERROR: In procedure get-bytevector-n:
> Throw to key `guile-ssh-error' with args `("read_from_channel_port" "Error 
> reading from the channel" #<input-output: channel (closed) 14c0e60> #f)'.
> guix build: error: build of 
> `/gnu/store/yfns7ga468vmv9jn72snk79b16p8mhfa-linux-libre-5.2.2.drv' failed
> 
> real    637m24.906s
> user    0m6.661s
> sys     0m0.897s

Thank you, this is helpful.

> Unfortunately I failed to record which machine was used and don't know a
> way to find out after the fact.

I believe it was hydra-slave2, one of the three armhf machines that I
host which were formerly part of hydra.gnu.org's build farm and were
recently added to Berlin by Ludovic.  I checked hydra-slave{1,2,3} for
build log files corresponding to the derivation above, and found that
all three of them have been attempted recently:

hydra-slave2 attempted to build it on July 23 08:07 UTC.
hydra-slave3 attempted to build it on July 22 16:40 UTC.
hydra-slave1 attempted to build it on July 22 04:44 UTC.

To be precise, each of those dates correspond to the end of the build
attempt.  All three build logs are truncated on the build machine as
well, with no error message at the end.

I now believe that these failures are related to the newly added armhf
build slaves, and that they have nothing to do with the recent changes
to our linux-libre packages.

Well, except for the silence timeout that sometimes happens on slower
machines while deblobbing linux-libre.  That's a separate issue.

      Thanks,
        Mark