Re: master 262d0c6: Mark some tests as expensive

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mardani29@yahoo.es,  michael.albinus@gmx.de,  stefan@marxist.se,
>   emacs-devel@gnu.org
> Date: Sat, 12 Sep 2020 14:11:40 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How will that work if the change was in Tramp, or was related to
> > Tramp?  In my case, for example, it would most probably mean none of
> > the Tramp tests will ever run at all, because they all take more than
> > 1 sec down here.
> >
> > Also, there are some tests with inherent delays (auto-revert tests are
> > one example, but there are others) -- would that mean we don't run
> > these, either?
> 
> Yes, if they take more than a second to run.

So how do we make sure some change didn't introduce a bug in those
features?

> We've already made this decision with all the tests previously marked as
> expensive, so there's nothing new here.

There's a large gap between what is currently marked as "expensive"
tests and having entire packages not tested at all.  The latter sounds
too radical to me.  E.g., auto-revert is an important feature, used by
many people.  Not having it in regression testing sounds like a step
backward to me.

> > What I do for a "fast test" use case is to run only the tests directly
> > related to the change, usually the FOO-tests.el tests that correspond
> > to the file FOO I've changed.  Sometimes, there are related tests in
> > other files as well.  A simple "make FOO-tests" command is all that's
> > needed.
> 
> I do that, too, when working on something specific.  But being able to
> say "make check" gives me greater confidence that I've not messed up
> something in a related area (and let's face it, the entirety of Emacs is
> "a related area" :-)).

I'm talking about a balance here.  Losing the tests of complete
features just because we want tests to finish quickly sounds
sub-optimal to me.  Can we make a smarter balance?  After all, what
matters is not how long a single test runs, but how long the entire
suite runs.  Having a few tests that take more than a second doesn't
necessarily enlarge the total time significantly, especially since
quite a few of the tests are much shorter.  So maybe a better
criterion would be the time it takes to run the entire suite?