Idea: Add .COMMANDCHANGE and .CACHE

[David A. Wheeler]

Idea: Add .COMMANDCHANGE and .CACHE

Problem:

Paul Smith noted on Sun, 09 Jun 2019 22:11:32 -0400:
> Most of the requests I see these days that would require a "last state
> database" wouldn't be helped by md5 checks: mainly they're asking for
> things like rebuilding targets automatically when compiler options have
> changed etc. Things like that can be done today with a lot of fancy
> trickery but people would rather it "just worked".

As noted above, "make"
detects when *external* files change, but if a command *inside* a Makefile
changes (including its expanded parameters) then make doesn't notice the change.
This could be resolved if make could automatically notice a change in
expanded command. Doing this will require the ability to store state somehow.

In addition, sometimes recipes take a while, and it'd be nice to be able to
cache old results. If we have to store state anyway,
we may as well support caching previous results.


Proposed solution:

I propose a new special target .COMMANDCHANGE.
Its dependencies (or all targets if no target is listed) add an implied 
dependency
on the expanded commands of the rule.  Thus, if the commands (once expanded)
change, the rule will be considered obsolete & be rerun.

This requires a way to notice a change.  Instead of a conventional database
(which is hard to share across instances & can be complicated to implement),
I propose using a directory specified in MAKESTATE
(by default ".makestate") for storing shared state.

Every time a non-.PHONY expanded target with .COMMANDCHANGE and a non-empty rule
is being considered for execution, the
change state file $(MAKESTATE)/$(FIRST TARGET).sha256 is first examined.
If the file doesn't exist, or its contents do not match the SHA-256 hash
of the expanded command for that target, then the target is considered
out-of-date and the rule is re-run.
If the set of commands executes successfully, then its SHA-256 hash
is written to $(MAKESTATE)/$(FIRST TARGET).sha256
(after creating its directory if necessary).
(Note: this could be stored in a directory name, instead of file contents,
so file opening wouldn't be necessary.  However, that would make it
easy to have a weird inconsistent state where there are multiple entries,
and I think it's worth avoiding that problem.)

If first_target is an absolute path the state path
will still be in $(MAKESTATE) by treating the target name as beginning as 
".TREEROOT".
E.g., if the target is /usr/share/mystuff/stuff,
then the change state file is .makestate//usr/share/mystuff/stuff.sha256.
(This might require some nonsense on Windows, but it's doable.)

In addition, if an executed rule is the target of .CACHE, has 1+ commands,
and MAKESTATECACHENUM is defined as a positive integer, then that rule's
execution results are cached:

* After after executing 
  all of the generated targets are copied (preserving all metadata possible)
  into $(MAKESTATE)/$(FIRST_TARGET).cache/$(SHA256 of expanded 
command)/$(TARGET_NAME).
  It then looks at "$(MAKESTATE)/$(FIRST_TARGET).cache" to see if there are
  more than MAKESTATECACHENUM entries, and if so, recursively deletes
  the oldest entries to get it down to that number.
  My guess is MAKESTATECACHENUM should default to 3.
* Before executing a rule that's outdated, check for
   $(MAKESTATE)/$(FIRST_TARGET).cache/$(SHA256 of expanded 
command)/$(TARGET_NAME).
   If it exists, copy its contents to the appropriate targets and *delete* 
targets
   of the rule if there's no corresponding entry.
   Note that .TREEROOT still applies.

I suspect .COMMANDCHANGE is a terrible name; better ones welcome.

--- David A. Wheeler