Re: Idea: Add .COMMANDCHANGE and .CACHE

[David A. Wheeler]

> On Mon, 2019-06-10 at 13:14 -0400, David A. Wheeler wrote:
> > Idea: Add .COMMANDCHANGE and .CACHE

On Tue, 11 Jun 2019 11:06:51 -0400, Paul Smith <address@hidden> wrote:
> Unlike some of the other ideas I think this one needs a lot of thought
> and consideration.  To me it seems the most complex and the most likely
> to have unintended side-effects or for the implementation to fail to
> meet the needs of users in a way which makes it hard to use generally,
> unless it's thought about deeply.
> 
> It's good to start thinking about what a feature like this would
> entail, but I would leave it until last.  At least.

I'm fine with working on these later after other things.

I think .COMMANDCHANGE (possibly with a new name) is not a big issue.
It's a *conservative* change from the point of view of correctness.

I *do* understand that there are reasonable concerns about .CACHE.
Like any cache, if it's given incorrect inputs it will produce incorrect 
outputs.
However, I think there are ways to mitigate that risk,
it's purely opt-in, it gives a potential massive speed increase, and
it's an addition that can be used by a large number of make users.
People really like massive speed increases :-).

Details below.

--- David A. Wheeler


============= DETAILS ===============================

I think .COMMANDCHANGE is not complex.
Since it caches the expanded command, at worst it will re-run a script
when before it would not have done so.
So it's relatively low risk; at worst, it'll run a command more often than 
before.
It won't rerun if only an environment variable changes, but that is *already* 
true.
I'm hoping that there's a better name than .COMMANDCHANGE.

I *agree* that .CACHE is more complex to use.
Its implementation is NOT complex - indeed, it can be
implemented in relatively few lines of code.
The challenge is that while .COMMANDCHANGE causes execution
to happen more than now, .CACHE can cause an execution to be
*completely* skipped and its cache results would be used instead.

The risk of .CACHE is that it can produce "wrong results", and
that is a valid concern.  In any cache system (including this one)
the cache key must be calculated from an absolutely complete
set of all relevant inputs, and the output (target) list *must*
be a complete statement of the resulting targets.
If there are multiple targets, using "&:" (not ":") may be critically
important, not just something that can be caught on the next
run of "make".  If environment variables' settings matter, they
need to be included in the inputs.  For example, the environment variables
could be expanded somewhere in the command (e..g, @printf '' "$MYVAR").
That means that .CACHE makes the system even more dependent
on accurate statements about targets & prerequisites.
For example: I didn't include read/write/execute permissions of prerequisites
in the inputs to the cache keys on purpose.  However, technically their
permission values can have an effect, and the current .CACHE doesn't handle 
that.

I'd love to hear ways of reducing the risks of .CACHE.  Some ideas:
1. We could define a "CACHEENV" variable, a list of all environment variables
   whose expansions would be included in the cache key.
   Adding a few environment variables to this list would mean that changing any
   of them would cause current cached values to be skipped.
2. When multiple targets are listed in a traditional rule (":" not "&:"),
   after running a rule in non-parallel mode, check to see if any of the
   other targets were modified.  If they were, warn or error out & explain the 
problem
   ("use &: instead of :, or create an intermediary value on line XYZ").
   This might be a good idea anyway, even if .CACHE isn't added...
   I just got bit by this mistake yesterday!  Automatically detecting common
   mistakes, and warning about them, is often a great idea.

I think it's still worth eventually having .CACHE.  Make already requires that
dependencies be accurately stated, and people
are already using cache systems like ccache.

When a new release of GNU Make comes out, some people may just
wonder "why would I want that?".  It's very valuable to support advanced
features, but if you haven't encountered the use case it may be hard for
people to appreciate a feature's advantages.  It will be *easy* to explain
why people might want a cache built into GNU make: if it works, their
edit-compile-run cycle time could be drastically reduced *regardless*
of what they use make for.
That would make caching, all by itself, a reason to upgrade to
a later version of GNU make.