RPC performance

[Ludovic Courtès]

Hello,

Ricardo Wurmus <address@hidden> skribis:

> Performance differs a lot for different packages.  “guix build bwa” (~ 7
> seconds) is a lot faster than “guix build python2-numpy” (> 40 seconds),
> for example.

My guess is that this is mostly due to RPC performance:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=rpc time ./pre-inst-env guix build bwa -n
Remote procedure call summary: 266 RPCs
  built-in-builders              ...     1
  query-substitutable-path-infos ...     1
  valid-path?                    ...     3
  add-text-to-store              ...   261
0.88user 0.04system 0:00.83elapsed 110%CPU (0avgtext+0avgdata 
497792maxresident)k
0inputs+0outputs (0major+11771minor)pagefaults 0swaps
$ GUIX_PROFILING=rpc time ./pre-inst-env guix build python2-numpy -n
Remote procedure call summary: 668 RPCs
  built-in-builders              ...     1
  query-substitutable-path-infos ...     1
  valid-path?                    ...     3
  add-text-to-store              ...   663
1.22user 0.06system 0:01.33elapsed 95%CPU (0avgtext+0avgdata 508288maxresident)k
1432inputs+0outputs (0major+10752minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---

There are several ways we can improve it, and over time we should try to
implement all of them:

  1. Buffer writes to the server socket (currently it’s terrible if you
     look at the ‘write’ calls in ‘strace’).

  2. Make fewer ‘add-text-to-store’ calls.  This can be done by, for
     instance, always using the same ‘-guile-builder’ script for
     everything that uses ‘gnu-build-system’, and parameterize the
     script through environment variables or similar.

  3. Replace N ‘add-text-to-store’ calls with a single ‘import-paths’
     call.  It’s not possible yet because ‘import-paths’ requires a
     signature, which is pointless for content-address items.

  4. Alternatively, change the protocol to allow for RPC pipelining
     (send N requests, then read N responses).  Currently this isn’t
     possible because all the RPCs call ‘process-stderr’ (which makes
     one or more round trips) because reading their response.

I’ll look into #1 soon and we can investigate #2 on the next
‘core-updates’ cycle.

Thanks,
Ludo’.