Re: mark expensive tests

[Eli Zaretskii]

> From: John Wiegley <address@hidden>
> Date: Sun, 03 Jan 2016 12:09:12 -0800
> Cc: address@hidden
> 
> Yes, we will have three test intensity levels:
> 
>   1. Sanity tests just make sure that the environment works, and the whole of
>      them should run in <30 seconds on typical hardware. Some shops call these
>      'smoke tests', as they are just there to make sure that Emacs can
>      function in the most basic ways.
> 
>   2. Regular tests do not incur intensive CPU or memory costs, and so can be
>      run on any hardware. These should finish on a scale of minutes, likely
>      <10 minutes at the most.
> 
>   3. Extensive tests are given free reign, and may not even be able to
>      complete on systems with CPU or memory constraints. These should feel
>      free to take up to an hour, maybe even beyond.
> 
>   4. Selective tests are never run automatically, but exist to stress test
>      some particular area of the system. These could take days, it doesn't
>      really matter what their requirements are.

I'm fine with this gradation, but here's some reality check:

  . The current test suite takes 3.5 min for a full run, including
    compilation of all the *.el files, 2.25 min if the *.el files are
    already compiled

  . For comparison, the test suite of the latest version of Texinfo I
    just built takes 4.5 min on the same machine, even though a lot of
    tests are skipped (due to some required infrastructure not being
    installed)

So it sounds like we have no candidates for #3 and #4, and the
division between #1 and #2 is questionable, since 2 min is not such a
long time to wait, IMO.

> I'd especially like the file-notify tests to move to #3, since these are what
> consistently bog down my testing environment.

file-notify-tests takes about 25 sec here, so I'm unsure why it should
be in #3.  Maybe that's because of remote tests (which are skipped on
my system), in which case perhaps the remote tests should be separated
into a separate test file and run on demand only.