bug#41607: Deleted store items are not actually deleted

[Ludovic Courtès]

Hi,

Chris Marusich <cmmarusich@gmail.com> skribis:

> The reason Guix is failing to GC dead items in the Docker container is
> because those dead items are not on the "top layer", so Docker returns
> an EXDEV error:
>
> https://docs.docker.com/storage/storagedriver/overlayfs-driver/
>
> "Renaming directories: Calling rename(2) for a directory is allowed only
> when both the source and the destination path are on the top
> layer. Otherwise, it returns EXDEV error ('cross-device link not
> permitted'). Your application needs to be designed to handle EXDEV and
> fall back to a 'copy and unlink' strategy."
>
> You can observe this by running guix-daemon with strace in the
> container, and watching what happens when you try to delete one of the
> offending store items (make sure it is a directory).  For example:
>
> 685   rename("/gnu/store/xib50iqk3w1gw9l770mad59m9bi3bcpc-manual-database", 
> "/gnu/store/trash/xib50iqk3w1gw9l770mad59m9bi3bcpc-manual-database") = -1 
> EXDEV (Invalid cross-device link)
>
> In most cases, when guix-daemon GC's a dead directory, it does this
> (see: nix/libstore/gc.cc):
>
> - Create a trash directory (usually /gnu/store/trash)
> - Move dead directories into the trash directory.
> - Delete the trash directory.
>
> The trash directory is on the "top layer" because it gets created in the
> running container.  However, in practice many store items from lower
> layers are made dead when Stephen's script runs "guix pull" and deletes
> the old profiles.  If any of those store items were directories,
> guix-daemon will fail to GC them because of an XDEV error.  If this is
> confusing to you, I suggest you experiment with Docker a little bit, and
> look closely at the steps that Stephen's script is running.  I outlined
> this in the email I accidentally deleted, but I'm a little too tired to
> reproduce it all a second time.  I hope you'll understand.

Interesting, thanks for the analysis!

> Should Guix do anything about this?  We could change guix-daemon to take
> correct action in the face of an XDEV error.  We could also improve the
> logging, since currently it silently swallows the XDEV error.

I guess we could delete recursively right away upon EXDEV.  It should be
just two lines of code, right?

> To work around the issue, Stephen can build the images from the same
> base image, rather than daisy-chaining new images from old ones.  That
> way, they would not accumulate layers without bound.

Maybe that too.

Thanks,
Ludo’.