Re: Suggested feature design for https://savannah.gnu.org/bugs/?42125

[Tristan Wibberley]

On the basis of the following experiment which shows some current behaviours that are so unhelpful that they are ripe for replacement I'd like to propose a cross-product substitution variable:

Makefile
======================
FOO = sensible
$(info $(FOO) is sensible)
$(info $(FOO ) is only going to be used be crazy people)
$(info $(FOO x( (%<stem>.o)=(%<stem>.c) ) ) prints the syntactic parentheses )
$(info $(FOO x( (%<stem>.o)=(%<stem>.c) look they really are syntactic its crazy )
======================

output
======================
sensible is sensible
 is only going to be used be crazy people
=(%<stem>.c) ) ) prints the syntactic parentheses
Makefile:5: *** unterminated call to function 'info': missing ')'. Stop.
======================

So my proposed cross-product substitution variable syntax is as follows:

$(FOO x( (%<stem>.o)=(%<stem>.c) ) )

consisting of the following parts:

cross-product substitution variable =
    '$(' variablename whitespace 'x(' substitution-list ')' ')'

substitution-list =

    substitution [substitution-list]

substitution =

    '(' decomposition-pattern ')' = '(' replacement-pattern ')'

decomposition-pattern =

    named-pattern

replacement-pattern =

    named-pattern

named-pattern =

   string with %<name> littered throughout, literal < or > can be inserted by \ quoting

  - any %<name> in the decomposition pattern can be inserted into any place in the replacement pattern by using its name.

  - name may include only alphanumerics, '-' and '_'

  - <name> can be omitted but only for one '%' in the decomposition pattern

  - % or %<name> is replaced everywhere it appears in the replacement pattern

  - each word in the variable's value is matched against each substitution in the substitution list in turn until a match is found -   - then that word is replaced using the corresponding replacement pattern

  - newlines can be eliminated between, before, and after substitutiona in the substitution list using '\'

  - I'm not sure how to handle '(' or ')' within a pattern but I can start by disallowing them at all to make the task easier

In my proposed rule syntax :=: this could be used thusly (syntax modified from previous suggestion so as to support unambiguous detection of cross product replacement vs the basic no-replacement behaviour:

:=: %.o %.logdir/deps : %.c | %.logdir/
:=x word1 word2: ( replacement-list ) : ( replacement-list ) | ( replacement-list )

:=x decorator | word1 word2 : ( replacement-list ) : ( replacement-list ) | ( replacement-list )

each '( replacement-list )' can be just a word instead as in the no-replacement example. any '(' or ')' must be quoted in that case. Although this can be abused to become messy, I think it will typically help to make a makefile cleaner and easier to understand. And maybe I won't need to face systems that very slowly and incorrectly generate impossibly large and confusing makefiles in the future.

This syntax will be a lot of work but I think it's valuable. cross-product is something I want to do a lot to define multiple alternatives with succinct declarations instead of using either of:

    makefile generation
       which becomes a bit of a pain and which impacts
        tab-completion (which I value a great deal)

    programmatic rule definition via functions
       which is difficult to understand for many people
        that need to enhance a component's build definitions

        and is also very difficult to diagnose issues

On 24 April 2016 at 10:20, Tristan Wibberley <address@hidden> wrote:

On 24 Apr 2016 01:00, "Paul Smith" <address@hidden> wrote:

On Thu, 2016-04-21 at 08:29 +0100, Tristan Wibberley wrote:
> x86-64 x86 armel: %=build/%/main.o: main.c ; true
> factors :: path=subst : prerequisites ; recipe

Almost all the syntax you suggest that uses "=" is not possible, because
it already has a well-defined meaning: it defines target-specific
variables.

The strangest thing. I did a test quite carefully with just one line to see that make reported an error for the presence of an equal sign and it did but now it doesn't! I clearly did something else odd that made it invalid but I don't know what.

What I suggested at the time was to create a new type of separator;
today we have ":" and "::" and I envisioned two new separators, "&:" and
"|:" which would explicitly choose whether all the targets are created
from a single invocation of the recipe ("&:") or each target is created
by a separate invocation of the recipe ("|:").

I think that method leaves a limit in make that already causes people to avoid make so I'm worried that adding the feature that way will be work that doesn't help the engineering userbase much. I love your syntax though.

Although I messed up in my tests of what syntax I could use, I intended very strongly to produce something that is currently invalid so existing makefiles do not get broken and can be combined with new ones using only copy and paste or "include". I think these things become important to allow users to scale up their activities with multiple engineers and low communication overhead.

The general form I would like to propose for a variety of engineering reasons is:

- base form for a rule that generates many targets from just one invocation where it is an error for any of those targets to remain not updated after the recipe is executed.

- base form works much like implicit rules today, a template for a rule which matches a target that's needed against a pattern and using similar patterns to define the co-targets and prerequisites.

- the pattern is not required to cause this behaviour but it always works like that for all forms of target - for predictable behaviour on minimum knowledge.

- a declarative comprehension syntax (like list comprehensions) is available to repeat a rule definition succinctly based on a set of factors such as configuration/platform names.
      http://www.secnetix.de/olli/Python/list_comprehensions.hawk
      https://en.wikipedia.org/wiki/Set-builder_notation.

- the comprehension's behaviour can be tuned by decorating an existing rule in order to do things like "define the same rule pattern for each platform" then later on "make it so that a specific artifact is <platform>_dbg" followed some hours later by "reset to the normal behaviour". All with minimum change to the rule's definition while making it easy for engineers to choose to ensure that they and other's will see and rapidly understand the intention of such a change to that rule via a diff tool.

- The decoration feature should support more advanced scripting so the choice of adding _dbg is controlled by a separate spec file and can also force one or more of the targets to be invalid if that file changes.

- the behaviour of .SECONDEXPANSION is default for such rules or controlled by the decorator.

Now, its a lot of work to achieve all of that but if the initial syntax is chosen well then the work can happen in many phases.

Here's another syntax suggestion:

'\n:=[decorator-name|][domain]:' means rule generation with a comprehension (ie, assign several rules via a template over several key factors) - this syntax is currently invalid. "domain" is a list of words which are applied to the targets and prerequisites to create multiple rules of the same form. Initially only :=: (no decorator nor domain) is supported.

1) support existing implicit rule syntax - only parser,make database, recipe executor changed and comment added to make database output. I suspect that each rule can already be flagged separately for second expansion so that can be enabled for these rules from the start. BUT, it is an error if any target is not updated when the recipe runs.

:=: %.o %.d: %.c | $$(@D)/

2) support a new "cross-product mode" for substitution variables which substitutes all % symbols in the replacement instead of only the first. This mode is always used for the substitution in this new rule syntax but I will use the current substitution syntax in lieu of a new one.
 

3) support non-empty domain with cross product rules to generate many rules of the same form

:=x86_64 x86 armel: %=%/%%.o %/%%.d: %=%/%%.c | $$(@D)/

4) support decorator. the decorator's function hooks into the pattern substitutions to modify their behaviour. This way an individual file can be switched in all targets and all prerequisites, for example to choose a debug or trace logging build of a specific translation unit without modifying anything else. The interface to the decorator function can be enhanced later if necessary (for example special variables available to the function to access and modify extended flags for the rule and all sorts would be possible.

.DECORATOR_somename:=\
    subst=fn_name\
    future_hook=fn_name2

:=decorator-name|x86_64 x86 armel: %=%/%%.o %/%%.d: %=%/%%.c | $$(@D)/

All this should be possible without anything really difficult and it should limit future parser issues by consuming very little of the limited space for extension - although I don't know what to do about the syntax for cross-product substitutions.

Also, make database dump output needs to be considered each step.

What do you think?

Regards

Tristan