Re: How to rebuild on recipe change?

[Kirill Smelkov]

On Tue, Feb 16, 2010 at 03:19:31AM -0800, Philip Guenther wrote:
> On Tue, Feb 16, 2010 at 1:37 AM, Kirill Smelkov <address@hidden> wrote:
> ...
> > Yes, this sort of works. But how about e.g. target specific variables?
> > Look:
> >
> >> I.e., something like:
> >> --------
> >> # delete the normal rule, so that ours is preferred
> >> %.o: %.c
> >>
> >> # provide ours that expects %.o_c to contain the command for building
> >> the .o file
> >> %.o: %.o_c %.c
> >>         $(shell cat $(filter %.o_c,$^)) $(OUTPUT_OPTION) $(filter %.c,$^)
> >>
> >> # generate or update the file containing the command.  The FORCE is because
> >> # we need this to be rebuilt each time, but we can't declare it .PHONY (as 
> >> then
> >> # the %.o would always be rebuilt too, even if the %.o_c didn't change)
> >> %.o_c: FORCE
> >>         @+new='$(COMPILE.c)'; \
> >>         printf '%s\n' "$$new" | cmp -s - $@ || printf '%s\n' "$$new" >$@
> >
> > When you dump command, you expand $(COMPILE.c) in %.o_c context, and
> > that looses e.g. target specific CFLAGS for e.g.
> >
> > 1.o : CFLAGS += -O6
> 
> Nope, that works just fine with my example.  Seriously, give it a shot.

Yes, I've realized that too, but only after sending my email. 1.o_c will
inhering from 1.o whatever target-specific settings were done to it.
Dummy me - sorry for that.


> I just noticed a bug in my example: the .SECONDARY rule should be
> changed to .PRECIOUS, as .SECONDARY doesn't accept patterns.  (This
> has tripped me up before, yes)

Thanks

> > Also, if $<, $^ and $+ are present in $(COMPILE.c) they are not the
> > same they would be when building %.o .
> 
> Well, in my experience, auto-dependency generation already handles
> changes to prerequisite lists without any extra effort.  How are those
> variables changing without one of the files in the *previous* list
> actually being changed (i.e., mtime bump)?

Ok, here is an example:


libsmth_OBJS := 1.o 2.o 3.o

libsmth.so : $(libsmth_OBJS)
        $(CC) -shared -o $@ $+


Consider that initially everything is already built -- 1.o, 2.o, 3.o and
libsmth.so .

Then let's say we want to (or due to git bisect or whatever) remove 3.o
from libsmth -- so Makefile is updated ...

- libsmth_OBJS := 1.o 2.o 3.o
+ libsmth_OBJS := 1.o 2.o

... but everything else is unchanged.

Now make will think that all libsmth.so prerequisites are up-to-date
(1.o and 2.o) and would not rebuild libsmth.so . This is not correct and
important in some cases, since 3.o could contain e.g. ELF constructors.

And another case -- suppose, next we've somehow rebuild libsmth.so
(forcing rebuild by hand for example), and then we (or through git
bisect or whatever) add 3.o back (and this is the only change to the
sources):

- libsmth_OBJS := 1.o 2.o
+ libsmth_OBJS := 1.o 2.o 3.o


Now, 3.o is still there in the objtree, but its mtime is _older_ than
that of libsmth.so since libsmth.so was rebuild recently. So make will
consider libsmth.so as not needing a rebuild -- again, wrong behaviour.


> >  Yes, it is possible to leave
> > ``$<'' and friends in command output as is (i.e. unexpanded), and to
> > later substitute them by hand, but that's not correct -- e.g. $+ could
> > change between runs, and we would not notice.
> 
> Can you expand on how that would occur?

Maybe I'm misunderstanding you, but if you ask about how $+ could change
between runs - please see above.

> [Checking this over before sending it, I just realized what a horrible
> pun that sentence was...]

If you meant something else, please ask again.


> > I know default $(COMPILE.c) does not have auto variables, and so the
> > whole target cmd has additional
> >
> >    ... $(OUTPUT_OPTION) $(filter %.c,$^)
> >
> > but is that 100% correct? e.g. $(OUTPUT_OPTION) could change between
> > runs and we won't notice. More, e.g. for link commands, $^ could change
> > (e.g. user adds new .o to, or removes unneeded .o from list of library
> > objects), and this wouldn't be noticed either.
> 
> OUTPUT_OPTION is just "-o $@", so don't worry about other random
> variables in it.  I'm not sure why it was ever put into GNU make; I'm
> trying to think of a way to use it that wouldn't be *better* done by
> altering other variables and am coming up blank.  IMO, the right
> solution for this is to just use "-o $@" in that rule.  In fact,
> consider that another bug in what I originally suggested.
> 
> Note that in POSIX compliant mode, GNU make doesn't use OUTPUT_OPTION
> in the default rules.  Just Say No.

Ok, I agree about OUTPUT_OPTION - that's not important. What is
important is how to handle automatic variables.


> > The main problem here is that $(COMPILE.c) or whatever, gets expanded
> > not in the target context, and that's the main pity -- correct expansion
> > is only possible in target context in _recipe_ only.
> 
> Yes, automatic variables are only defined as necessary in the target
> rule's context.  You have two options:
> 1) detect changes to them via other means, or
> 2) abandon the idea of using make's built-in file dependency
>    logic to detect when commands have changed.
> 
> I've never had any problems doing choice #1, but perhaps you can
> illustrate how you're running into it.

I don't want to abandon the idea of using make as build driver based on
it's dependency logic.

See above about libsmth.so and $+ . Is it what you were asking for?

> 
> 
> ...
> > #         .------- says expand following in target context when rule is 
> > already
> > #         |        selected and add expansion result to prerequisites. We 
> > could
> > #         |        have ``&|'' as well.
> > #         v
> > %.o: %.c  & $(call @.force-if-cmd-changed,cmd_cc_o_c)
> >        $(cmd_cc_o_c)
> >        echo 'cmd_$@ := $(cmd_cc_o_c)' > address@hidden
> ...
> > How is that? Very similar to second expansion, but a bit different.
> 
> It sounds like (I haven't looked closely) there's a bug in
> .SECONDEXPANSION.

Some time ago I've thought that too, but it seems it's not a bug.

First there _is_ some bug (?) about second expansion which I've recently
hit:

https://savannah.gnu.org/bugs/?28456

I'll quote myself:

---- 8< ----
  It seems current make behaviour is inconsisten with its manual. I quote
  `Secondary Expansion of Implicit Rules':

  """
  As `make' searches for an implicit rule, it substitutes the stem and
  then performs secondary expansion for every rule with a matching target
  pattern.  The value of the automatic variables is derived in the same
  fashion as for static pattern rules.  As an example:

       .SECONDEXPANSION:

       foo: bar

       foo foz: fo%: bo%

       %oo: $$< $$^ $$+ $$*

     When the implicit rule is tried for target `foo', `$$<' expands to
  `bar', `$$^' expands to `bar boo', `$$+' also expands to `bar boo', and
  `$$*' expands to `f'.
  """

  So let's verify that:

  $ cat se-manual.mk
  .SECONDEXPANSION:

  foo: bar

  foo foz: fo%: bo%

  show-vars =
          $(info $$@ : $@)
          $(info $$< : $<)
          $(info $$^ : $^)
          $(info $$+ : $+)
          $(info $$* : $*)


  %oo: $$(show-vars)
          touch $@


  $ touch bar && make -f se-manual.mk foo
  $@ : foo
  $< : foo
  $^ : bar boo
  $+ : bar boo
  $* : f
  $@ : boo
  $< : boo
  $^ :
  $+ :
  $* : b
  touch foo


  Take a closer look to first 5 output lines starting with `$@ : foo'. The
  manual says "When the implicit rule is tried for
  target `foo', `$$<' expands to `bar', `$$^' expands to `bar
  boo', `$$+' also expands to `bar boo', and `$$*' expands to `f'"

  But it seems we have $$< = foo (!= bar).


  Could please anyone clarify, what is the right behaviour and how (if whether)
  we need to fix it.
---- 8< ----


But still, that's only about second expansion of $< . As to $+, $^ and
friends (and $< too), make manual explicitely states that there is a
_difference_. I quote ``3.10 Secondary Expansion'':

---- 8< ----
  ... However, there are some subtle
  differences and "corner cases" which come into play for the different
  types of rule definitions that `make' understands.  The subtleties of
  using the different automatic variables are described below.

  Secondary Expansion of Explicit Rules
  -------------------------------------

  During the secondary expansion of explicit rules, `$$@' and `$$%'
  evaluate, respectively, to the file name of the target and, when the
  target is an archive member, the target member name.  The `$$<'
  variable evaluates to the first prerequisite in the first rule for this
  target.  `$$^' and `$$+' evaluate to the list of all prerequisites of
  rules _that have already appeared_ for the same target (`$$+' with
  repetitions and `$$^' without).  The following example will help
  illustrate these behaviors:

       .SECONDEXPANSION:

       foo: foo.1 bar.1 $$< $$^ $$+    # line #1

       foo: foo.2 bar.2 $$< $$^ $$+    # line #2

       foo: foo.3 bar.3 $$< $$^ $$+    # line #3

     In the first prerequisite list, all three variables (`$$<', `$$^',
  and `$$+') expand to the empty string.  In the second, they will have
  values `foo.1', `foo.1 bar.1', and `foo.1 bar.1' respectively.  In the
  third they will have values `foo.1', `foo.1 bar.1 foo.2 bar.2', and
  `foo.1 bar.1 foo.2 bar.2' respectively.

  ...

  Secondary Expansion of Implicit Rules
  -------------------------------------

  As `make' searches for an implicit rule, it substitutes the stem and
  then performs secondary expansion for every rule with a matching target
  pattern.  The value of the automatic variables is derived in the same
  fashion as for static pattern rules.

  ...
---- 8< ----


So first, $$+ in prerequisites is not the same as $+ in recipe -- it
references prerequisites "_that have already appeared_ for the same
target".

And second, for implicit rules, make _tries_ every rule with matching
target pattern, so if we try to put something with side effects into
second-expansion phase, we'll got those side effects for _every tried
rule_, not only _the_ rule which will be used to build target.

That's the difference, and it is documented.


> If that bug was fixed, would you need this
> additional feature?  If no (the fix would be sufficient), then I
> strongly recommend that you just try to fix the bug (or work with
> Paul, Boris, et al to fix it) instead of trying to get a subtle
> feature added.

I'm all for fixing bugs, but it seems .SECONDEXPANSION can't help here.
At least it can't be 100% relied upon because of automatic variables.

> If nothing else, consider how much more work
> *documenting* this (with a good example of a problem it solves that
> can't easily be otherwise solved!) would be!

And also it will be years for updated make to propagate ... yes lots of
work and time.

That's why I'm here to first try to clearly understand what current make
can and can't and whether it needs fixes/enhancments if any.


Thanks again for answering,
Kirill