Re: Bootstrappable bitcoin release builds with Guix

[Carl Dong]

Hi Ludovic!

> OK. You could store the output of ‘guix describe -f channels’ in
> addition to the manifest, and then do ‘guix pull -C’ of this file. That
> way, you’d be pinning a specific package set.

Ah... things like this make me more fall in love with Guix everyday.

> > 1.  libstdc++ would not link statically even with "-static-libstdc++". The 
> > hack
> >     was to remove the .la file under $LIBRARY_PATH.
> >
>
> Weird. Could you send a small test case for this address@hidden

I realized afterwards that this is actually another problem. The real problem 
is that configure insists on adding `-lstdc++` to `config.status` and `libtool` 
even when `-static-libstdc++` is in `LDFLAGS`. I will do some more digging but 
it seems to only happen in Guix environments. I would appreciate any insight.

> > 2.  Upon inspection of the binaries produced at the end of this process, 
> > they all
> >     had rpaths. The hack was to use patchelf --remove-rpath on them.
> >
>
> Yes, ‘ld-wrapper’ and our ‘gcc’ packages add those on purpose; they’re
> required for dynamically-linked binaries. But you’re producing
> statically-linked executables in the end, right?

We're producing a executable that's dynamically linked in the end, we perform a 
few checks to make sure that only the libraries we want to dynamically link to 
are dynamically linked to: 
https://github.com/bitcoin/bitcoin/blob/40a720acb8472f6e8afdf8b8d1897f35d58daf1f/contrib/devtools/symbol-check.py#L57.
 I have resorted to modifying the default gcc package to omit the 
`GNU_USER_TARGET_LIB_SPEC` substitute which is how the rpaths sneak in. I 
suspect that using a specfile would have also worked?

> > 3.  Upon inspection of the binaries produced at the end of this process, 
> > their
> >     interpreters all had a `/gnu/store/blahblah-glibc-2.28' prefix. The 
> > hack was
> >     to use patchelf --set-interpreter on them.
> >
>
> To use /lib/ld-linux-x86-64.so.2 instead?
>
> You can do that, but there’s a risk: this is assuming that the loader
> and libc on the user’s machine are compatible with those you built
> against.

Right. This is why we check our libc version is one that is reasonably old to 
work on old systems: 
https://github.com/bitcoin/bitcoin/blob/40a720acb8472f6e8afdf8b8d1897f35d58daf1f/contrib/devtools/symbol-check.py#L38.
 I'm not familiar with how the loader might be incompatible?

> Can you achieve this by simply taking the ouptut of “guix build
> bitcoin-core-alt” (where ‘bitcoin-core-alt’ is a variant that statically
> links against libstdc++ etc.), patching the ELF interpreter, and then
> copying it in a tarball?
>
> You’d also get cross-compilation by adding ‘--target’.

Yes, I think you're right that that would be a little easier. I might 
transition to this method once I'm ready, and I don't think patching the ELF 
interpreter is that bad.

Cheers,
Carl Dong
address@hidden
"I fight for the users"

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, April 17, 2019 4:58 PM, Ludovic Courtès <address@hidden> wrote:

> Hi Carl,
>
> Carl Dong address@hidden skribis:
>
> > I've been on a quest to use Guix for Bitcoin Core's reproducible builds as I
> > believe that Guix's focus on bootstrappability, and Guile's simplicity and
> > flexibility are very desirable qualities in building an auditable, secure, 
> > and
> > reliable build process. My pull request (very short thanks to Guix's
> > infrastructure) can be found here:
> > https://github.com/bitcoin/bitcoin/pull/15277/files
>
> I’m late to the party, but this is awesome! The goals you’re pursuing
> are in line with those we’re after in Guix, Reproducible Builds, and
> Bootstrappable, so it’s great to see that Guix can help others here.
>
> > I've submitted patches for the Guix bitcoin-core package to make it
> > reproducible, which seems to work fine. However, for easier acceptance into 
> > the
> > bitcoin core process, I need to produce tarballs like the ones we have on 
> > our
> > servers today: https://bitcoincore.org/bin/bitcoin-core-0.17.1/
> > For some context, we have a "mini-guix" of sorts seen in our "depends tree"
> > here: https://github.com/bitcoin/bitcoin/tree/master/depends. This builds 
> > all
> > the dependencies for bitcoin just the way we want them, in preparation for
> > getting linked into bitcoin itself.
> > My current approach for the build process is to produce a Guix container in
> > which I execute a build of our "depends tree" followed by a build of bitcoin
> > itself. See the Guix manifest and scripts here:
> > https://github.com/bitcoin/bitcoin/pull/15277/files
>
> OK. You could store the output of ‘guix describe -f channels’ in
> addition to the manifest, and then do ‘guix pull -C’ of this file. That
> way, you’d be pinning a specific package set.
>
> > However, there were three hiccups that I had to hack my way around:
> >
> > 1.  libstdc++ would not link statically even with "-static-libstdc++". The 
> > hack
> >     was to remove the .la file under $LIBRARY_PATH.
> >
>
> Weird. Could you send a small test case for this address@hidden
>
> > 2.  Upon inspection of the binaries produced at the end of this process, 
> > they all
> >     had rpaths. The hack was to use patchelf --remove-rpath on them.
> >
>
> Yes, ‘ld-wrapper’ and our ‘gcc’ packages add those on purpose; they’re
> required for dynamically-linked binaries. But you’re producing
> statically-linked executables in the end, right?
>
> > 3.  Upon inspection of the binaries produced at the end of this process, 
> > their
> >     interpreters all had a `/gnu/store/blahblah-glibc-2.28' prefix. The 
> > hack was
> >     to use patchelf --set-interpreter on them.
> >
>
> To use /lib/ld-linux-x86-64.so.2 instead?
>
> You can do that, but there’s a risk: this is assuming that the loader
> and libc on the user’s machine are compatible with those you built
> against.
>
> > My questions are:
> >
> > 1.  Is there a way to avoid the hacks that I listed above? I understand 
> > that it
> >     might mean writing custom gcc packages and I'm 100% okay with that.
> >
> > 2.  Is there an easier way of achieving the same thing?
> >
>
> I think the difficulty here lies in producing non-self-contained binary
> tarballs, paradoxically. :-)
>
> By that I mean that the tarball at
> https://bitcoincore.org/bin/bitcoin-core-0.17.1/bitcoin-0.17.1-x86_64-linux-gnu.tar.gz
> makes assumptions on the user’s machine:
>
> --8<---------------cut here---------------start------------->8---
> $ file /tmp/bitcoin-0.17.1/bin/bitcoin-cli
> /tmp/bitcoin-0.17.1/bin/bitcoin-cli: ELF 64-bit LSB pie executable x86-64, 
> version 1 (GNU/Linux), dynamically linked, interpreter 
> /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, 
> BuildID[sha1]=aced8cd0e28adb7a9e0c15f2e794aae7150d348e, stripped
> $ objdump -x /tmp/bitcoin-0.17.1/bin/bitcoin-cli |grep NEED
> NEEDED librt.so.1
> NEEDED libm.so.6
> NEEDED libgcc_s.so.1
> NEEDED libpthread.so.0
> NEEDED libc.so.6
> NEEDED ld-linux-x86-64.so.2
> VERNEED 0x0000000000002850
> VERNEEDNUM 0x0000000000000006
> --8<---------------cut here---------------end--------------->8---
>
> The binary assumes that /lib64/ld-linux-x86-64.so.2 exists, that
> libgcc_s.so & co. will be found in the “standard locations”, etc. These
> assumptions do not hold on a standalone Guix system or on NixOS or
> Gobolinux. They are probably reasonable for FHS distros, though.
>
> Can you achieve this by simply taking the ouptut of “guix build
> bitcoin-core-alt” (where ‘bitcoin-core-alt’ is a variant that statically
> links against libstdc++ etc.), patching the ELF interpreter, and then
> copying it in a tarball?
>
> You’d also get cross-compilation by adding ‘--target’.
>
> Thanks,
> Ludo’.