[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances
From: |
zimoun |
Subject: |
[bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances |
Date: |
Thu, 15 Jul 2021 09:33:26 +0200 |
Hi,
This is an attempt to improve the performance of "guix search". It is still
half baked but it allows to discuss further the idea about expanding the
current '/lib/guix/package.cache' and avoids to forget an IRL discussion. ;-)
Let start by what needs to be improved: the part when cache is not
authoritative. It is slower than the current approach because the package is
read twice, i.e., the module is indeed loaded twice, once by
'fold-available-packages' via 'fold-module-public-variables*' and then again
by 'find-packages-by-description' via 'read-package-from'. The issue is to
have a common interface for both cases (cache and no-cache). More thoughts
are required. ;-)
Then, using the cache is slower than expected. Therefore, something is maybe
twisted -- quick implementation before holidays ;-) -- with the use of
'fold-avaibale-packages' as proposed by Ludo [1]. Note that instead another
'fold-packages' (say 'fold-packages*') using the new cache should be used. As
it is done with v4 and the performances were as expected:
<http://issues.guix.gnu.org/39258#89>
1: <http://issues.guix.gnu.org/39258#93>
>From my understanding, the issue that 'package-relevance' accepts a 'package'
(and then all the chain until displaying) and 'fold-avaibale-packages' does
not return a package. Well, I do not know; especially where to put something
similiar to 'read-package-from'.
To test, after applying the patches, the command is:
./pre-inst-env guix pull --allow-downgrades --disable-authentication \
--url=$(pwd) --branch=search-v6 -p /tmp/new
Let compare only for cold cache and time this cache building (Guix 7db8fd6):
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix build --check $(guix gc --derivers $(readlink -f
~/.config/guix/current/lib/guix/package.cache))
real 0m28,848s
user 0m1,481s
sys 0m0,252s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix build --check $(guix gc --derivers $(readlink -f
/tmp/new/lib/guix/package.cache))
real 0m40,279s
user 0m1,582s
sys 0m0,232s
It seems longer but compared to the time of "guix pull" completion, it seems
acceptable. However, maybe it could become an issue when running a lot of
"guix time-machine"... Well, hard trade-off. ;-)
Let compare for some queries:
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search game | recsel -C -P name | wc -l
371
real 0m7,561s
user 0m3,525s
sys 0m0,391s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search game | recsel -C -P name | wc -l
371
real 0m9,814s
user 0m3,240s
sys 0m0,363s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search strategy game | recsel -C -P name | wc -l
16
real 0m8,565s
user 0m2,803s
sys 0m0,430s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search strategy game | recsel -C -P name | wc -l
16
real 0m9,679s
user 0m2,370s
sys 0m0,334s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search strategy game caesar | recsel -C -P name | wc -l
0
real 0m8,307s
user 0m2,388s
sys 0m0,366s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search strategy game caesar | recsel -C -P name | wc -l
0
real 0m3,626s
user 0m0,948s
sys 0m0,101s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search game strategy the | recsel -C -P name | wc -l
4
real 0m8,776s
user 0m2,903s
sys 0m0,454s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search game strategy the | recsel -C -P name | wc -l
4
real 0m9,495s
user 0m2,546s
sys 0m0,313s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search the game strategy | recsel -C -P name | wc -l
4
real 0m8,502s
user 0m2,534s
sys 0m0,388s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search the game strategy | recsel -C -P name | wc -l
4
real 0m9,508s
user 0m2,254s
sys 0m0,363s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search crypto library | recsel -C -P name | grep libb2
libb2
real 0m8,744s
user 0m2,875s
sys 0m0,374s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search crypto library | recsel -C -P name | grep libb2
libb2
real 0m9,229s
user 0m2,448s
sys 0m0,397s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search cuirass integration | recsel -C -P name | wc -l
1
real 0m8,132s
user 0m2,343s
sys 0m0,407s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search cuirass integration | recsel -C -P name | wc -l
1
real 0m8,940s
user 0m2,036s
sys 0m0,369s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search cuirass | recsel -C -P name | wc -l
2
real 0m8,240s
user 0m2,461s
sys 0m0,367s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search cuirass | recsel -C -P name | wc -l
2
real 0m8,863s
user 0m2,019s
sys 0m0,377s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search cuirass integration foo | recsel -C -P name | wc -l
0
real 0m8,258s
user 0m2,418s
sys 0m0,521s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search cuirass integration foo | recsel -C -P name |
wc -l
0
real 0m3,358s
user 0m0,867s
sys 0m0,139s
--8<---------------cut here---------------end--------------->8---
This last example suggests that 'read-package-from' is the slowdown.
(On a side note, maybe I am doing wrong, but there is no improvement by the
recent introduction of 'cut' for multi-terms as the query "the game strategy"
and "game strategy the". Another story. :-))
When cache is not authoritative, it is worse, as expected:
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search -L /tmp/my-pkgs cuirass integration foo |
recsel -C -P name | wc -l
0
real 0m12,503s
user 0m7,807s
sys 0m0,529s
and note that currently the performances of "guix search" is the same for both
cases (authoritative and not authoritative); i.e., see previous timing.
Last, two minor remarks about previous comments.
1. Ludo commented:
> Therefore the cache '/lib/guix/package.cache' contains more
> information.
This breaks the binary interface, so we’ll have to analyze the impact of
such a change and devise a strategy.
<http://issues.guix.gnu.org/39258#93>
and after some checking, this should be fine, IIUC. The '--news' is ok
because of '#:allow-other-keys'. And other parts are modified accordingly.
Guix revision N creates a cache that Guix revision N+1 will read but it should
not be an issue; see 'inferior-available-packages'.
2. And Ludo wrote:
I realize the other cache also has that problem, but it would be nice
to add a
version tag to the cache. Basically emit something like:
(package-metadata-cache (version 0) VECTOR …)
instead of just:
(VECTOR …)
<http://issues.guix.gnu.org/39258#93>
which is, after discussions, not necessary. Versioning does not make sense
here because the cache is read by the Guix which generates it. Therefore,
specify a version is extraneous here.
Comments are welcome for this work-in-progress. ;-)
Cheers,
simon
zimoun (2): DRAFT packages: Add fields to packages cache. DRAFT scripts:
package: Use cache in 'find-packages-by-description'.
gnu/packages.scm | 52 +++++++++++++++++++++++++++-------------
guix/scripts/package.scm | 46 +++++++++++++++++++++++++----------
2 files changed, 70 insertions(+), 28 deletions(-)
base-commit: 4196087f3d6fc254db5b4c47658e5679c835516f
--
2.32.0
- [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances,
zimoun <=