[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: master 262d0c6: Mark some tests as expensive
From: |
Michael Albinus |
Subject: |
Re: master 262d0c6: Mark some tests as expensive |
Date: |
Sat, 12 Sep 2020 16:43:17 +0200 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) |
Daniel Martín <mardani29@yahoo.es> writes:
Hi Daniel,
> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> Tramp tests need more time because they involve a remote system.
>> Moreover, the time taken by each Tramp test depends on the speed of
>> the connection, which you cannot know in advance.
>
> If they depend on the connection speed, or if a host is online or
> offline, they also introduce non-determinism that perhaps is not
> apparent now, but can cause problems when the Tramp codebase (and its
> number of tests) scale. For example, if a Tramp test has a 0.1% chance
> of failure because of an unrelated network problem, then if the Tramp
> test suite reaches a point where 10000 tests are run per day, people
> would be investigating 10 test flakes per day. That's a good reason for
> people to lose confidence in the Tramp test suite and ignore failures.
Tramp tests don't need a remote connection by default. They simulate a
connection by a "mock" method, which is in fact a local shell.
Real remote connections can be tested also, but this doesn't happen with
"make check". Read the Commentary section of tramp-tests.el for details.
> I'd say "we should try to run *some* tests as close to real-life
> conditions as possible". By abstracting the environment in some tests,
> one could potentially test an infinite number of environments and error
> conditions, not only what the test happens to run on. We would still
> have a few end-to-end tests that check that the program as a whole works
> fine, of course. The trade-off is that writing that kind of hermetic
> tests takes more time, specially for packages like Tramp.
For the records, I keep an ansible script which runs tramp-tests.el in
~75 different configurations, all of them using real remote hosts. I run
it at least prior releasing a new Tramp version; it takes up to 2 days
(the longest single run of tramp-tests.el takes about 10 hours). This
won't go into Emacs' "make check", of course.
Best regards, Michael.
- Re: master 262d0c6: Mark some tests as expensive, (continued)
- Re: master 262d0c6: Mark some tests as expensive, Michael Albinus, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Lars Ingebrigtsen, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Dmitry Gutov, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Daniel Martín, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Michael Albinus, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Dmitry Gutov, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Lars Ingebrigtsen, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Daniel Martín, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Eli Zaretskii, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Daniel Martín, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive,
Michael Albinus <=
- Re: master 262d0c6: Mark some tests as expensive, Daniel Martín, 2020/09/12
Re: master 262d0c6: Mark some tests as expensive, Michael Albinus, 2020/09/12