bug#42009: package.cache not deterministic

[Ludovic Courtès]

Hi Simon,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sun, 28 Jun 2020 at 22:29, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Most likely the problem with non-reproducible .go files is that the fix
>> for <https://bugs.gnu.org/20272> was incomplete.  In particular, I think
>> that gensyms are not reproducible when building things in parallel,
>> because the gensym depends on what’s loaded vs. interpreted.
>
> Thank you for the pointer.
>
> How can I test the "hypothesis" of "building things in parallel"?
>
> guix describe -f channels > /tmp/chan.scm
> guix pull -C /tmp/chan.scm --cores=1 -p /tmp/repull1
>
> guix build --check --no-grafts --cores=1 \
>      $(guix gc --derivers \
>             $(readlink -f /tmp/repull1/lib/guix/package.cache))
> The following profile hook will be built:
>    /gnu/store/qbrgxbnx0hi13xm36a6a0zijzc1rcz22-guix-package-cache.drv

I realize I was a bit off-topic: I was commenting on the more general
issue of .go non-reproducibility.

The problem with the package cache seems to be different.  Sorry for the
confusion!

After --check, we can compare both caches like this:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (define a (load-compiled 
"/gnu/store/0yd3kaar87zyxhbrjqjypp5rar3zj4gb-guix-package-cache/lib/guix/package.cache"))
scheme@(guile-user)> (define b (load-compiled 
"/gnu/store/0yd3kaar87zyxhbrjqjypp5rar3zj4gb-guix-package-cache-check/lib/guix/package.cache"))
scheme@(guile-user)> (length a)
$2 = 13949
scheme@(guile-user)> (length b)
$3 = 13949
scheme@(guile-user)> ,use(srfi srfi-1)
scheme@(guile-user)> (lset= equal? a b)
$4 = #f
scheme@(guile-user)> (car (lset-difference equal? a b))
$5 = #("python2" "2.7.17" (gnu packages python) python-2 ("out" "tk") #t #f 
"gnu/packages/python.scm" 107 2)
--8<---------------cut here---------------end--------------->8---

So, surprisingly, it’s not just an ordering issue: the caches do contain
different pieces of information.

This patch solves the ordering issue:

diff --git a/gnu/packages.scm b/gnu/packages.scm
index d22c992bb1..5154d73deb 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -407,13 +407,25 @@ reducing the memory footprint."
       (_
        result+seen)))
 
+  (define entry-key
+    (match-lambda
+      (#(name version module symbol outputs supported? deprecated?
+              file line column)
+       (string-append name version (or file "")
+                      (if line (number->string line) "")))))
+
+  (define (entry<? a b)
+    (string<? (entry-key a) (entry-key b)))
+
   (define exp
-    (first
-     (fold-module-public-variables* expand-cache
-                                    (cons '() vlist-null)
-                                    (all-modules (%package-module-path)
-                                                 #:warn
-                                                 warn-about-load-error))))
+    (sort
+     (first
+      (fold-module-public-variables* expand-cache
+                                     (cons '() vlist-null)
+                                     (all-modules (%package-module-path)
+                                                  #:warn
+                                                  warn-about-load-error)))
+     entry<?))
 
   (mkdir-p (dirname cache-file))
   (call-with-output-file cache-file

But it’s not quite right because the order in which variables are
traversed is still non-deterministic, so between two runs of
‘generate-package-cache’, caches differ like this:

--- a   2020-06-29 14:27:32.291028456 +0200
+++ b   2020-06-29 14:27:37.930993059 +0200
@@ -8581,7 +8581,7 @@
  #("clang-runtime"
    "9.0.1"
    (gnu packages llvm)
-   clang-runtime
+   clang-runtime-9
    ("out")
    #t
    #f
@@ -26511,7 +26511,7 @@
  #("gcc-objc++"
    "7.5.0"
    (gnu packages gcc)
-   gcc-objc++-7
+   gcc-objc++
    ("out" "lib" "debug")
    #t
    #f
@@ -26641,7 +26641,7 @@
  #("gcc-toolchain"
    "7.5.0"
    (gnu packages commencement)
-   gcc-toolchain
+   gcc-toolchain-7
    ("out" "debug" "static")
    #t
    #f
@@ -33311,7 +33311,7 @@
  #("ghc"
    "8.6.5"
    (gnu packages haskell)
-   ghc-8.6
+   ghc-8
    ("out" "doc")
    #t
    #f
@@ -41876,7 +41876,7 @@
  #("icedtea"
    "3.7.0"
    (gnu packages java)
-   icedtea-8
+   icedtea
    ("out" "jdk" "doc")
    #t
    #f
@@ -54376,7 +54376,7 @@
  #("linux-libre-headers"
    "5.4.20"
    (gnu packages linux)
-   linux-libre-headers-5.4.20
+   linux-libre-headers
    ("out")
    #t
    #f
@@ -54636,7 +54636,7 @@
  #("llvm"
    "9.0.1"
    (gnu packages llvm)
-   llvm-9
+   llvm
    ("out" "opt-viewer")
    #t
    #f
@@ -61826,7 +61826,7 @@
  #("ocaml"
    "4.09.0"
    (gnu packages ocaml)
-   ocaml
+   ocaml-4.09
    ("out")
    #t
    #f
@@ -62256,7 +62256,7 @@
  #("opencl-headers"
    "2.2.0-0.e986688"
    (gnu packages opencl)
-   opencl-headers
+   opencl-headers-2.2
    ("out")
    #t
    #f
@@ -92636,7 +92636,7 @@
  #("python2"
    "2.7.17"
    (gnu packages python)
-   python-2
+   python-2.7
    ("out" "tk")
    #t
    #f
@@ -92646,7 +92646,7 @@
  #("python"
    "3.8.2"
    (gnu packages python)
-   python-3
+   python
    ("out" "tk")
    #t
    #f
@@ -123676,7 +123676,7 @@
  #("rust"
    "1.39.0"
    (gnu packages rust)
-   rust-1.39
+   rust
    ("out" "doc" "cargo")
    #t
    #f

The problem has to do with aliases: we don’t always see the same
variable first.  So we have to sort before calling ‘expand-cache’.

To be continued…

Ludo’.