bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible

[Chris Marusich]

Hi everyone,

Efraim Flashner <efraim@flashner.co.il> writes:

> Is this a file we actually need during the bootstrap process? Can we
> "work around it" by just deleting it?

That's a good idea.  I tried building %gcc-static with the
--disable-libstdcxx configure flag (see attached patch), and that caused
the build of %gcc-static itself to become reproducible on Debian and
Fedora when using exactly the same inputs.  To clarify, I have recently
run the following experiments:

+------+-------------+---------------------+--------------+----------------+---------------+
| Case |    Build    |      libstdcxx      | substitutes? |     Inputs     |    
Result     |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Built fresh on |    
Output     |
|  1   | %gcc-static |       enabled       |     yes      | Debian, copied |   
differs:    |
|      |             |                     |              |   to Fedora    |  
libstdc++.a  |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Re-used inputs |  
Toy binary   |
|  2   | toy program |         n/a         |     yes      |   from above   |   
does not    |
|      |             |                     |              |                |    
differ     |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Built fresh on |  
Output does  |
|  3   | %gcc-static |      disabled       |     yes      | Debian, copied |  
not differ   |
|      |             |                     |              |   to Fedora    |    
           |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Built fresh on |    
Output     |
|  4   | %gcc-static |      disabled       |     yes      | Debian, fresh  |   
differs:    |
|      |             |                     |              |   on Fedora    | 
various files |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Inputs failed  | 
Build failed  |
|  5   | %gcc-static |      disabled       |      no      |  to build on   |    
on both    |
|      |             |                     |              |  both systems  |    
systems    |
+------+-------------+---------------------+--------------+----------------+---------------+

The "toy program" in case 2 was just this:

  #include <stdio.h>
  int main() {
    printf("Hello");
    return 0;
  }

When I say I "copied" the inputs from Debian to Fedora, I mean just
that.  I copied stuff like /gnu and /var/guix from Debian to Fedora
(while guix-daemon was stopped, of course), GC'd just %gcc-static on
Fedora, and then rebuilt %gcc-static on Fedora so it would use exactly
the same inputs as were used on Debian.

The most notable new findings are cases 3 and 4.  Case 5 made me pretty
sad because I spent almost an entire weekend trying to build Guix from
source using various binary Guix releases, and in none of the attempts
was I successful in running "guix pull" or even just "guix environment
--pure guix" without substitutes, which really surprised me.

Case 3 confirms Efraim's suggestion: we can fix the libstdc++.a
reproducibility problem by simply not building libstdc++.a in the first
place. It also confirms that, when built with --disable-libstdcxx,
%gcc-static can be built reproducibly on different systems as long as
exactly the same inputs are used.  Whether or not %gcc-static can be
used to successfully bootstrap packages on powerpc64-linux when built in
this way remains to be seen.

Case 4, unfortunately, demonstrates that there are still other
reproducibility issues that we have not yet resolved.  Since the only
difference between cases 3 and 4 is how the inputs were realized, the
differing %gcc-static output must be caused by the inputs somehow.

In case 4, the %gcc-static output differed in the following files (the
hashes on the left are MD5 hashes):

--8<---------------cut here---------------start------------->8---
--- /dev/fd/63  2020-09-25 20:35:33.386554595 -0700
+++ /dev/fd/62  2020-09-25 20:35:33.387554604 -0700
@@ -1,28 +1,28 @@
-c9b0dfcbad566c0b8b88df94bb993312  ./bin/c++
-092823145dc96b9eb81111362f7b4ced  ./bin/cpp
-c9b0dfcbad566c0b8b88df94bb993312  ./bin/g++
-e4cc43b7790dcd25f31419bad606b36e  ./bin/gcc
+8f02302b55643f1c711e472a42fea8bd  ./bin/c++
+9f1fd993e4f2b796fcc56f0b2f8a47d2  ./bin/cpp
+8f02302b55643f1c711e472a42fea8bd  ./bin/g++
+583d1b011a7ba009d7385117dd7a33c8  ./bin/gcc
 f9d94f4bb61f70d14ea4b2ce73c9be9d  ./bin/gcc-ar
 01fc2184f99c558771aa8f2fe30b373d  ./bin/gcc-nm
 da5356ee09ccda4ca06758d056370f7e  ./bin/gcc-ranlib
-98645f7b00ba185e713915099853fd37  ./bin/gcov
-37dd62589454703ae7f2eaac1668b66e  ./bin/gcov-dump
-f3dbc7e0c84a40194af3aa2429444e87  ./bin/gcov-tool
-c9b0dfcbad566c0b8b88df94bb993312  ./bin/powerpc64-linux-gnu-c++
-c9b0dfcbad566c0b8b88df94bb993312  ./bin/powerpc64-linux-gnu-g++
-e4cc43b7790dcd25f31419bad606b36e  ./bin/powerpc64-linux-gnu-gcc
-e4cc43b7790dcd25f31419bad606b36e  ./bin/powerpc64-linux-gnu-gcc-5.5.0
+a208bedbfca9c7bd6c27d0d42f7096fe  ./bin/gcov
+43330e8ae00976b4b3427d2f83b0725e  ./bin/gcov-dump
+9f37da5e96f147d733eb7195350ae5d2  ./bin/gcov-tool
+8f02302b55643f1c711e472a42fea8bd  ./bin/powerpc64-linux-gnu-c++
+8f02302b55643f1c711e472a42fea8bd  ./bin/powerpc64-linux-gnu-g++
+583d1b011a7ba009d7385117dd7a33c8  ./bin/powerpc64-linux-gnu-gcc
+583d1b011a7ba009d7385117dd7a33c8  ./bin/powerpc64-linux-gnu-gcc-5.5.0
 f9d94f4bb61f70d14ea4b2ce73c9be9d  ./bin/powerpc64-linux-gnu-gcc-ar
 01fc2184f99c558771aa8f2fe30b373d  ./bin/powerpc64-linux-gnu-gcc-nm
 da5356ee09ccda4ca06758d056370f7e  ./bin/powerpc64-linux-gnu-gcc-ranlib
-6ed530d13e65c3500b7e7b7cc863afdc  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1
-24a83af179ca8849da8c64aa854ec8ed  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus
-0c05b45bb926a06c2b09acdb1db9aad0  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2
+22b72247a5706f090505341263ca1fc2  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1
+3be618d184038dd30011d6aa8198c0be  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus
+2f31e84c01cc087318d0c7f15b6e3f47  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2
 417a5b42a26275b2c912db16b9abf73a  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/fixincl
 fd6f80ec9089ddf51f9cac26299e45af  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/fixinc.sh
 a585abbd6a9cdc474564b54fc72e4efa  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/mkheaders
 5071acceb24c0c0e8a423286205ed54c  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/mkinstalldirs
-4e77b773ac45ce8f82a4d21a34063920  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper
+5267311e0ed8bb5358e5556dfa205ca6  
./libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper
 a90ab86f837913280f72beb5310714bc  
./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbegin.o
 0b38aa831c40b6bc4fb22248746d60d8  
./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbeginS.o
 8f62d8795bebd8e87caa3640522ff93b  
./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbeginT.o
--8<---------------cut here---------------end--------------->8---

These appear to be the same diffs that I reported before - with the
notable exception that libstdc++.a now is missing from the list of
differing files (hooray!).

Going forward, I'm not sure how best to investigate the inputs to find
out what's causing the differences.  I just know I really, really,
really don't want to rebuild everything multiple times, since it takes
hours/days.  If you have any creative ideas for how to speed up the
investigation, I'm all ears.

-- 
Chris

0001-gnu-Disable-libstdc-in-bootstrap-GCC.patch
Description: Text Data

signature.asc
Description: PGP signature