Re: Treating tests as special case

[Chris Marusich]

Pjotr Prins <address@hidden> writes:

> I think we should have a switch for turning off tests. Let the builder
> decide what is good or bad. Too much nannying serves no one.

I think it would be OK to give users the choice of not running tests
when building from source, if they really don't want to.  This is
similar to how users can choose to skip the "make check" step (and live
with the risk) when building something manually.  However, I think we
should always run the tests by default.

Maybe you could submit a patch to add a "--no-tests" option?

address@hidden (Ludovic Courtès) writes:

> That is why I was suggesting putting effort in improving substitute
> delivery rather than trying to come up with special mechanisms.

Yes, I think that improving substitute availability is the best path
forward.  I'm willing to bet that Pjotr would not be so frustrated if
substitutes were consistently available.

Regarding Pjotr's suggestion to add a "test result substitute" feature:
It isn't clear to me how a "test result substitute" is any better than a
substitute in the usual sense.  It sounds like Pjotr is arguing that if
the substitute server can tell me that a package's tests have passed,
then I don't need to run the tests a second time.  But why would I have
to build the package from source in that case, anyway?  Assuming the
substitute server has told me that the package's tests have passed, it
is almost certainly the case that the package has been built and its
substitute is currently available, so I don't have to build the package
myself - I can just download the substitute!  Conversely, if a
substitute server says the tests have not passed, then certainly no
substitute will be available, so I'll have to build it (and run the
tests) myself.  Perhaps I am missing something, but it does not seem to
me that the existence of a "test result substitute" would add value.

I think what Pjotr really wants is (1) better substitute availability,
or (2) the option to skip tests when he has to build from source because
substitutes are not available.  I think (1) is the best goal, and (2) is
a reasonable request in line with Guix's goal of giving control to the
user.

Ricardo Wurmus <address@hidden> skribis:

> An idea that came up on #guix several months ago was to separate the
> building of packages from testing.  Testing would be a continuation of
> the build, like grafts could be envisioned as a continuation of the
> build.

What problems would that solve?

Pjotr Prins <address@hidden> writes:

> The building takes long enough. Testing takes incredibly long with
> many packages (especially language related) and are usually single
> core (unlike the build).

Eelco told me that in Nix, they set --max-jobs to the number of CPU
cores, and --cores to 1, since lots of software has concurrency bugs
that are easier to work around by building on a single core.  Notably,
Guix does the opposite: we set --max-jobs to 1 and --cores to the number
of CPU cores.  I wonder if you would see faster builds by adjusting
these options for your use case?

> It is also bad for our carbon foot print. Assuming everyone uses Guix
> on the planet, is that where we want to end up?

When everyone uses Guix on the planet, substitutes will be ubiquitous.
You'll be able to skip the tests because, in practice, substitutes will
always be available (which means an authorized substitute server ran the
tests successfully).  Or, if you are very concerned about correctness,
you might STILL choose to build from source - AND run the tests -
because you are concerned that your particular circumstances (kernel
version, file system type, hardware, etc.) was not tested by the build
farm.

> I know there are two 'inputs' I am not accounting for: (1) hardware
> variants and (2) the Linux kernel. But, honestly, I do not think we
> are in the business of testing those. We can assume these work.

Even if those components worked for the maintainers who ran the tests on
their own machines and made a release, they might not work correctly in
your own situation.  Mark's story is a great example of this!  For this
reason, some people will still choose to build things from source
themselves, even if substitutes are available from some other place.

-- 
Chris

signature.asc
Description: PGP signature