[bug#32980] [PATCH 0/2] Multiplexed build output from the daemon

[Ludovic Courtès]

Hello Guix!

This patch set is partly a response to the problem Ricardo raised at:

  https://lists.gnu.org/archive/html/guix-devel/2018-09/msg00322.html

and partly a natural followup to the addition of (guix status):

  https://issues.guix.info/issue/32837

So far the daemon would send all its output (messages it writes as well as
messages any of its build processes writes) directly as a single stream
to its clients.  This had several shortcomings:

  1. Clients could not distinguish messages coming from the daemon (such
     as build traces or ‘guix substitute’ messages) from messages coming
     from build processes.  (Notably build processes could “forge” build
     traces such as “@ build-started”.)

  2. When max-jobs > 1, clients would receive intermingled output from all
     the build processes without any way to disentangle it.

  3. Build traces written by the daemon were expected to start on a new
     line but a builder could write output not terminated by a newline
     (e.g., “checking for …” messages from ./configure) and consequently
     build traces would not start on a new line and would go unnoticed
     by (guix status) and co.

With this change clients can optionally ask for “multiplexed build
output”.  When it’s enabled, build output is prefixed by a special
trace, like this:

  @ build-output 1234 21
  checking for fcntl...

where 1234 is the PID of the build process speaking and 21 is the number
of bytes in the following build output fragment.  The PID is first given
in the corresponding “@ build-started” trace.

On the client side, (guix status) is adjusted to produce events like:

  (build-log #f MESSAGE)

for a message coming from the daemon, and:

  (build-log PID MESSAGE)

for a message coming from PID.

The downside of the protocol is that it creates quite some overhead.  For
example, when extracting a tarball, we see things like:

  read(13, "gmlo\0\0\0\0", 8)             = 8
  read(13, "5\0\0\0\0\0\0\0", 8)          = 8
  read(13, "@ build-output 25935 29\ncoreutils-8.29/m4/fseterr.m4\n", 53) = 53
  read(13, "\0\0\0", 3)                   = 3

That is, a 29-byte message with a 24-byte header (plus the
8 + 8 + 3 = 19 bytes of the underlying protocol; see ‘process-stderr’.)

Another option would be to incorporate multiplexing in the lower-level
binary protocol.  However, the client side would need potentially bigger
changes: the single ‘current-build-output-port’ sink would no longer
be a good match.

Thoughts?

Ludo’.

Ludovic Courtès (2):
  daemon: Support multiplexed build output.
  status: Build upon multiplexed build output.

 guix/scripts/build.scm          |   3 +
 guix/scripts/environment.scm    |   1 +
 guix/scripts/pack.scm           |   1 +
 guix/scripts/package.scm        |   3 +-
 guix/scripts/pull.scm           |   1 +
 guix/scripts/system.scm         |   1 +
 guix/status.scm                 | 165 ++++++++++++++++++++++++--------
 guix/store.scm                  |  15 ++-
 nix/libstore/build.cc           |  25 +++--
 nix/libstore/globals.cc         |   2 +
 nix/libstore/globals.hh         |   9 +-
 nix/libstore/worker-protocol.hh |   2 +-
 nix/nix-daemon/nix-daemon.cc    |   2 +-
 tests/status.scm                |  47 ++++++++-
 tests/store.scm                 |  63 ++++++++++++
 15 files changed, 289 insertions(+), 51 deletions(-)

-- 
2.19.0