[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug#45253] [PATCH 0/6] Pipeline substitute integrity check, deduplicati
From: |
Ludovic Courtès |
Subject: |
[bug#45253] [PATCH 0/6] Pipeline substitute integrity check, deduplication, and canonicalization |
Date: |
Tue, 15 Dec 2020 10:38:30 +0100 |
Hello Guix!
This is a followup to <https://issues.guix.gnu.org/45018>. It is
meant to be applied on top of <https://issues.guix.gnu.org/44760>.
Until now, guix-daemon would check the hash of store items just
substituted, reset timestamps/permissions, and deduplicate. This
would lead to extra I/O: the whole set of files is traversed three
times by the daemon and read two times.
This patch series is about delegating that work to ‘guix substitute’,
which it can do directly as it restores file, thereby reducing I/O
to the minimum necessary.
I tested with substitutes that contain many files:
guix build pipewire@0.2 ffmpeg ungoogled-chromium vim-full \
emacs-no-x emacs-no-x-toolkit
On my laptop with an SSD, the wall-clock time is almost unchanged
when fetching lzip substitutes. You can see that the throughput
displayed while downloading is slightly lower than before, which
is consistent because lzip downloads are CPU-bound¹, but this is
compensated by the lack of processing time between substitutes.
With gzip substitutes, I see a 10% speedup on the wall-clock time
on my laptop.
Ludo’.
¹ https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html
Ludovic Courtès (6):
tests: Check the build trace for hash mismatches on substitutes.
daemon: Let 'guix substitute' perform hash checks.
tests: Check the mtime and permissions of substituted items.
daemon: Do not reset timestamps and permissions on substituted items.
tests: Make sure substituted items are deduplicated.
daemon: Delegate deduplication to 'guix substitute'.
guix/scripts/substitute.scm | 70 +++++++++++++++++++++++++-----
guix/serialization.scm | 8 +++-
nix/libstore/build.cc | 85 ++++++++++++++++++++-----------------
tests/store.scm | 82 +++++++++++++++++++++++++++++++++++
tests/substitute.scm | 58 ++++++++++++++++++++++---
5 files changed, 248 insertions(+), 55 deletions(-)
--
2.29.2
- [bug#45253] [PATCH 0/6] Pipeline substitute integrity check, deduplication, and canonicalization,
Ludovic Courtès <=
- [bug#45253] [PATCH 1/6] tests: Check the build trace for hash mismatches on substitutes., Ludovic Courtès, 2020/12/15
- [bug#45253] [PATCH 5/6] tests: Make sure substituted items are deduplicated., Ludovic Courtès, 2020/12/15
- [bug#45253] [PATCH 4/6] daemon: Do not reset timestamps and permissions on substituted items., Ludovic Courtès, 2020/12/15
- [bug#45253] [PATCH 6/6] daemon: Delegate deduplication to 'guix substitute'., Ludovic Courtès, 2020/12/15
- [bug#45253] [PATCH 3/6] tests: Check the mtime and permissions of substituted items., Ludovic Courtès, 2020/12/15
- [bug#45253] [PATCH 2/6] daemon: Let 'guix substitute' perform hash checks., Ludovic Courtès, 2020/12/15
- [bug#45253] [PATCH 0/6] Pipeline substitute integrity check, deduplication, and canonicalization, Ludovic Courtès, 2020/12/15
- bug#45253: [PATCH 0/6] Pipeline substitute integrity check, deduplication, and canonicalization, Ludovic Courtès, 2020/12/19