[bug#43591] [PATCH core-updates] gnu: glibc-final: Catch all cases of a glibc user not requesting 64-bit offsets and then using readdir.

[Danny Milosavljevic]

Hi Ludo,

On Wed, 30 Sep 2020 11:32:58 +0200
Ludovic Courtès <ludo@gnu.org> wrote:

> Dropping emulated builds, or at least 32-bit emulated builds.  We just
> need to remove build machines from the file above.

Oh.

Do we have real armhf machines?  (as in not aarch64)

*Looks at guix-maintenance*  We do.  Awesome.  Then sure.

But just to be clear, WE MUST NOT USE aarch64 to build armhf as long as this
problem isn't fixed.

This problem has nothing to do with emulation.

> A change in gnu-build-system would change all the derivations.  I don’t
> think the Data Service can help us here.

I want to know what actually changes in the final binaries.  Surely that works
somehow--guix data services or not.

Basically, for each package in Guix,

  diff -r `~/broken-guix/pre-inst-env guix build $package` 
`~/fixed-guix/pre-inst-env guix build $package` || echo "affected: $package"

but after replacing references by deduplicated content addressed references
(for example if derivation A refers to files in derivation B, but derivation B
only changed the directory name it's in and not the content of the derivation,
then that should not count as a diff in A.  That should happen recursively).

Basically ignore the directory names in /gnu/store and make new directory
names that are the hash of each directory's (recursive) CONTENTS (after
fixing references in that content, too, obviously)--as opposed to sources.
Then diff those.

If necessary, I can run that on my laptop--it will just take several weeks
and miss derivations I don't have in the first place.

> I have mixed feelings: fixing packages one by one doesn’t sound great,
> but OTOH setting the ‘CFLAGS’ environment variable globally can have
> unexpected side effects in some cases (overriding package-specific
> CFLAGS) and zero effects in other cases (for non-Autoconf packages or
> badly-written ‘configure.ac’ files), both of which would be hard to
> detect.

The latter is easy to detect since I patched dirent.h in glibc exactly for that
reason.  That way, glibc WON'T let you use it wrong (except if you explicitly
ask for it).  On Guix systems, there is no legitimate reason to use it wrong
in the first place.

In my opinion, not having an automated way to tell us when a package is using
it wrong would be not doing our due diligence--how would you know we had
actually fixed the problem for good?  You wouldn't know.
And you wouldn't have fixed the problem for good--I can tell you that much now.

There already was an essential package, rhash, which didn't pick up the
CFLAGS--and that's how I found it.  It's easy.

About the unexpected side effects--yes, that's right.  That's why we should get
a list of diff results (see above for the command) and then manually look at
the source code of those packages and their dependencies.

> If we take a step back: what’s the problem?

It means we have no trustworthy i686 packages, which means we do not have a
trustworthy full source bootstrap using Mes (since that uses i686 executable
to bootstrap).

In practice, this problem is not so bad since the kernel on i686 has a compat
layer that hasn't been telling us the truth for d_off, so we should be "good".
But philosophically, we are doing it dead wrong.

Also, this won't work on armhf or any other 32 bit architecture--so there,
we would be both philosophically and practically wrong.

Also, the "not telling us the truth for d_off on i686" is a leaky compat layer.
It totally DOES wind up telling us the truth sometimes (see my earlier test
table)--and then we have a problem.

>  Then we have packages that do not
> support large files; it’s not great but evidently we can live with it.
> :-)  Ideally, we’d report it upstream when we encounter it.

I really don't care about large file support.  That's mostly a bonus we get
while fixing this whole ordeal the right way.  That said, maybe users
care they actually can store a 5 GiB file on their 4000 GiB drive on armhf :P

> So to me that hints at targeted fixes: fixing key packages like CMake
> (roughly what you already did) where lack of large file support can be
> problematic.

As long as we patch glibc's dirent.h so it tells us when we are doing stupid
stuff, we can't go wrong much.  So sure.

As I said, the main problem is FINDING the affected packages.  It's not like
they'll fail building (in general).  They'll just do weird stuff at runtime
instead.  Case in point: cmake DID NOT fail building.  And it totally is
broken.

Everything after that is easy.

pgp7RhiIthmgk.pgp
Description: OpenPGP digital signature