--- Begin Message ---
|
Subject: |
cuirass: Contention while registering new builds. |
|
Date: |
Tue, 22 Sep 2020 18:52:39 +0200 |
|
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) |
Hello,
The attached screenshot shows that 9 evaluations are currently "In
progress" for "guix-master" specification.
Evaluations 16725 to 16738 are completed, 7 hours ago and 56 minutes ago
respectively. They are still shown as "In progress" because the
"register" phase in "build-packages" is not over. It is also possible to
see that new builds are registered for those evaluations, but very
slowly.
As each evaluation has to register around 50k builds, there might be
some sort of "writing" contention on the database, explaining the long
build registration time.
Thanks,
Mathieu
--
https://othacehe.org
cuirass.png
Description: PNG image
--- End Message ---
--- Begin Message ---
|
Subject: |
Re: bug#43564: cuirass: Contention while registering new builds. |
|
Date: |
Mon, 28 Sep 2020 09:51:38 +0200 |
|
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) |
Hello,
> As each evaluation has to register around 50k builds, there might be
> some sort of "writing" contention on the database, explaining the long
> build registration time.
Turns out, once an evaluation is over, new builds are registered. This
registration tries to insert a new build for each derivation returned by
the evaluation phase. If the new build does not add a new output, the
insertion query is then rollbacked. This means that there are at least
as many insertion queries as new derivations.
SQlite allows at most one writer at a time, and even though we are using WAL
mode, performing a lot of insertions will reduce the reading performances.
When multiple evaluations are performed in parallel, the large number of
concurrent insertion queries also causes contention.
Having 10 pending evaluations means that we are probably trying to
insert around 500.000 records concurrently. This is something that
SQLite does not seem to be designed for.
To avoid those issues, we need to check first in the "Outputs" table
which derivations are already registered. This means that most of the
insertion queries will be replaced by reading queries, that are much
less expensive and more suitable for Cuirass concurrent implementation.
In the best case of one pending evaluation, the registration duration is
reduced from 1800 seconds to 320 seconds. I think that the gain is way
larger when there are multiple pending evaluations.
Pushed a fix as 461e07e14e1c8013343c0a2cb26c0e022e10d5e4.
Thanks,
Mathieu
--- End Message ---
cuirass.png