Re: Can the automated tests be included in the source tarball?

[Ole Tange]

On Thu, Oct 1, 2020 at 5:09 PM Shlomi Fish <shlomif@shlomifish.org> wrote:
> On Tue, 29 Sep 2020 01:18:31 +0200
> Ole Tange <ole@tange.dk> wrote:
>
> > On Thu, Sep 24, 2020 at 8:37 AM Shlomi Fish <shlomif@shlomifish.org> wrote:
:
> > Unfortunately some of them are heavily dependent on running on the
> > same equipment as mine due to timing issues (and every time I change
> > hardware or upgrade software, I have to tweak the tests or results a
> > little).
>
> I see. I suppose you are aware of running on a pristine CI system, e.g:

I had a talk with a guy who used Jenkins. Jenkins is all good and fine
for tests that have predictable outcomes no matter the environment.

It is worse when you have 2 tests: One has to run while the machine is
swapping and one has to run when the machine is _not_ swapping. In
other words they cannot be run in parallel.

Or: A test has to run on a machine with high CPU load, while another
test has to run on an idle machine.

Or: A test has to be run on Centos 3, because other distributions do
not show the error.

I guess around 20% of my tests are in that category. The rest could
probably be run by a normal CI.

The current home-made test system also has the advantage that if a
test fails, it will be pushed to the top, so when I run the test suite
again, it will be the first test to run.

> > If you want to help, a good start would be to have
> > testsuite/REQUIREMENTS ported so it would work on more systems. It
> > (tries to) install and set up all required software to run the
> > testsuite.
>
> Wow, it looks intimidating. And I thought some of my FOSS projects had too 
> many
> prereqs!

The problem is that errors are often seen in the interaction with
other tools - sometimes only as race conditions.


/Ole