Re: [bug #30381] Don't avoid implicit rule recursion quite so soon. (fwd <- address@hidden)

[Brent W. Baccala]

On Jun 2, 2014 5:14 AM, "Reinier Post" <address@hidden> wrote:
> On Fri May 30 02:58:17 2014, address@hidden (Brent Baccala) wrote:
> > Follow-up Comment #15, bug #30381 (project make):
> >
> > I support this feature.
> >
> > I just tried to write a makefile that implicitly generates output files from
> > XML control files, that contain references to other files in the XML.  What I
> > wanted was this:
> >
> >
> > define futurebases
> >    $(shell xmllint --xpath "print(//futurebase/@filename)" $(1))
> > endef
> >
> > .SECONDEXPANSION:
> >
> > %.htb: %.xml $$(call futurebases, %.xml)
> >         $(HOFFMAN) -g $<
> >
> >
> > It doesn't work.
>
> I don't think this is related.
>
> My -M patch supports using the same rule multiple times on a path
> when traversing the dependency graph.  It takes the graph for granted, it just traverses more paths in it.
>
> What you need to do is determine the dependency graph dynamically.
> The problem is not how to traverse it, but how to construct it.
> This happens before the graph is traversed.  My patch won't help you.
>
> Apparently, you have a command that, given an *.xml file, lists the
> dependencies of a corresponding *.htb file.  You are trying to list
> these dependencies in the rule for making the *.htb file, so you
> can use them in the recipe.
>
> I think your basic approach is correct: with .SECONDEXPANSION,
> you *can* expand a list of dependencies on a per-target basis.
> However, some problems remain in the code above:
>
> 1)
>   Your recipe uses $<, that is, the first dependency only.
>   Is your list always 1 long?  If not, use $+ instead.
>   Besides, $< is the *.xml file; I don't think you want to use that
>   to create the *.htb file.

It's correct as written.  The program (hoffman) takes an xml file as input and produces an htb file as output.  However, the xml file can contain references to other htb files from previous runs.  These are also read as input.  Only one xml file is specified on the command line.

> 2)
>   The %.xml argument to your call will just be %.xml.
>   I've never used .SECONDEXPANSION, but judging by the manual
>
>     http://www.gnu.org/software/make/manual/html_node/Secondary-Expansion.html
>
>   I think you need to use $< here instead.

It would be clearer that way.  However, in implicit.c:pattern_search() appears this comment:

                  if (dep->need_2nd_expansion)
                    {
                      /* If the dependency name has %, substitute the stem.

                         Watch out, we are going to do something tricky
                         here. If we just replace % with the stem value,
                         later, when we do the second expansion, we will
                         re-expand this stem value once again. This is not
                         good especially if you have certain characters in
                         your stem (like $).

                         Instead, we will replace % with $* and allow the
                         second expansion to take care of it for us. This way
                         (since $* is a simple variable) there won't be
                         additional re-expansion of the stem.  */

So I can get away with it.  :-)

> I'm not sure if you have the right escapes to make everything
> expand at the right time.  Specifically, you need a way to ensure
> the xmllint command is only invoked during the secondary expansion.
> This is above my head but it should be possible in principle.
> However you still end up with rules that contain *.xml dependencies
> that really aren't dependencies of the targets, but of the command
> used to list the dependencies.

I put some debugging in there, and I think it all runs at the right time.  It just doesn't use that rule a second time on the recursively generated htb/xml dependencies.

> I think I'd take a different approach: generate the rules as a first
> phase.  There are two basic approaches to doing this:
>
> A)
>   Using $(eval): write a macro that generates a whole rule,
>   then $(eval) a $(call) of that maccro for each of your *,xml files.
>
> B)
>   Write a rule that generates a 1-rule makefile from a *.xml file
>   using a regular recipe.  That 1 rule creates the *.htb file from
>   its dependencies.  Then in your regular makefile either include
>   all of those rules or invoke them with $(MAKE) -f.

C) Call make recursively:

%.htb: %.xml
        @$(MAKE) $(call futurebases, $<)
        $(HOFFMAN) -g $<

That's the idea, at least.  Since the xml files might contain no futurebase elements at all, something more complicated is required:

        @$(if $(strip $(call futurebases, $<)), $(MAKE) $(call futurebases, $<))

This assumes that the makefile is called the usual name, that there are no target-dependant variables, and probably some other hidden assumptions, too.  It's what I've currently settled on, but recursive rules instead of recursive processes would be nice.

I'd suggest putting this feature in as a special target, rather than as a command line option, since makefiles shouldn't rely on options for their correct interpretation.

> Reinier Post
> TU Eindhoven

    agape
    brent