bug#35521: Mariadb test suite failures on x86_64-linux

[Marius Bakke]

Marius Bakke <address@hidden> writes:

> Mark H Weaver <address@hidden> writes:
>
>> Hi,
>>
>> Marius Bakke <address@hidden> writes:
>>
>>> Chris Marusich <address@hidden> writes:
>>>
>>>> Hi,
>>>>
>>>> I've been encountering this failure off and on for a few weeks now, and
>>>> I'd like to help fix it.  In short, it seems like non-deterministic test
>>>> failures, to me.  I think we should gather data and report the issue
>>>> upstream, and maybe disable the offending tests in the meantime.
>>>
>>> I agree.  I notice many of these failing tests are for the TokuDB
>>> backend, which I doubt anyone is using in Guix anyway.
>>>
>>> Here is a patch that disables all tests mentioned in this report.  I
>>> would like to push it to core-updates.  Are there others?
>>
>> I'm concerned by how frequently and casually we simply disable failing
>> tests.  What is the utility of running test suites at all, if this is
>> how we respond?
>
> I had no idea this issue was so widespread until I noticed Berlins
> builders hit it more often than not.  I have not been able to reproduce
> these failures on my machines.  So it was kind of a panic reaction,
> being the person responsible for running these tests and all.
>
> Looking further into the changes between 10.1.37 and 10.1.38, I notice
> the 'tokudb.*' tests were enabled:
>
> https://github.com/MariaDB/server/commit/4c490d6df63695dc97b2c808e59954e6877d3a51
>
> Watching the build on Berlin in real time, I also see that the test
> output grind nearly to a halt while running those.
> 'tokudb.hotindex-insert-2' took 2700439 milliseconds, or 45 minutes, if
> I'm reading the test output correctly.
>
> The default test case timeout is 40 minutes (as specified in the Guix
> package), but I'm using 80 for this build (60 was insufficient).
>
> I suspect the problem is that the 'tokudb.*' tests put a lot of strain
> on the file system, which causes these other tests to fail.  It's
> interesting that disabling parallel build was insufficient though.

Update: Berlin built mariadb twice on core-updates with this patch:

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 6bfeaad9a2..64bc0938b6 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -753,7 +753,7 @@ Language.")
                  (with-directory-excursion "mysql-test"
                    (invoke "./mtr" "--verbose"
                            "--retry=3"
-                           "--testcase-timeout=40"
+                           "--testcase-timeout=80"
                            "--suite-timeout=600"
                            "--parallel" (number->string (parallel-job-count))
                            "--skip-test-list=unstable-tests"))

Mark, Chris: Can you try this change with MariaDB 10.1.40 and see if it
works for you?

signature.asc
Description: PGP signature