bug#22608: Module system thread unsafety and .go compilation

[Ludovic Courtès]

address@hidden (Taylan Ulrich "Bayırlı/Kammer") skribis:

> Sadly that assumption isn't met when autoloads are involved.
> Minimal-ish test-case:
>
> - Check out 0889321.
>
> - Build it.
>
> - Edit gnu/build/activation.scm and gnu/build/linux-boot.scm to contain
>   merely the following expressions, respectively:
>
> (define-module (gnu build activation)
>   #:use-module (gnu build linux-boot))
>
> (define-module (gnu build linux-boot)
>   #:autoload   (system base compile) (compile-file))
>
> - Run make again.
>
> If you're on a multi-core system, you will probably get an error saying
> something weird like "no such language scheme".

Do you have a clear explanation of why this happens?  I would expect
(system base compile) to already be loaded for instance, so it’s not
clear to me what’s going on.  Or is it just the mutation of (gnu build
linux-boot) that’s causing problems?

> Solution proposals:
>
> 1. s/par-for-each/for-each/.  Will make compilation slower on multi-core
>    machines.  We would do the same for guix pull, which is a bit sad
>    because it's so fast right now.  Very simple solution though.
>
> 2. We find out some partitioning of the Scheme modules such that there
>    is minimal overlap in total loaded modules when the modules in one
>    subset are each loaded by one Guile process.  Then each Guile process
>    loads & compiles the modules in its given subset serially, but these
>    Guile processes run in parallel.  This could speed things up even
>    more than now because the module-loading phases of the processes
>    would be parallel too.  It also has the side-effect that less memory
>    is consumed the fewer cores you have (because less Scheme modules
>    loaded into memory at once).  If someone (Ludo?)  has a good general
>    overview of Guix's module graph then maybe they can come up with a
>    sensible partitioning of the modules, say into 4 subsets (maxing out
>    benefits at quad-core), such that loading all modules in one subset
>    loads a minimal amount of modules that are outside that subset.  That
>    should be the only challenging part of this solution.
>
> 3. We do nothing for now since this bug triggers rarely, and can be
>    worked around by simply re-running make.  (We just have to hope that
>    it doesn't trigger on guix pull or on clean builds after some commit;
>    there's no "just rerun make" in guix pull or an automated build of
>    Guix.)  AFAIU Wingo expressed motivation to make Guile's module
>    system thread safe, so this problem would then truly disappear.

Short-term, I’d do #1 or #3; probably #1 though, because random failures
are no fun, and we know they can happen.

Longer-term, I’m not convinced by #2.  I think I would instead build
packages in reverse topological order, probably serially at first, which
would address <http://bugs.gnu.org/15602> (with the caveat that the (gnu
packages …) modules cannot be topologically-sorted, but OTOH they
typically don’t use macros, so we’re fine.)

That would require a tool to extract and the ‘define-module’ forms and
build a graph from there.

But really, we must fix <http://bugs.gnu.org/15602>, an in particular,
‘compile-file’ should not mutate the global module name space.  I think
we could do something like:

  (define (compile-file* …)
    (let ((root the-root-module)
          (compile-root (copy-module the-root-module)))
      (dynamic-wind
        (lambda ()
          (set! the-root-module compile-root)
          ;; ditto with the-scm-module
          )
        (lambda ()
          (compile-file …))
        (lambda ()
          (set! the-root-module root)
          ;; …
          ))))

It’s unclear how costly ‘copy-module’ would be, and the whole strategy
depends on it.

Eventually it seems clear that Guile proper needs to address this use
case, and needs to provide thread-safe modules.

Ludo’.