Re: Quoting special characters (was: Re: Possible solution for special characters in makefile paths)

[Frank Heckenbach]

Paul Smith wrote:

> On Tue, 2014-02-25 at 00:05 +0100, Frank Heckenbach wrote:
> > Let me chime in with my interpretation of Eli's suggestion. In this
> > case, FOO would be stored as is (foo\ bar) with make understanding
> > that (every) variable value is a string quoted according to shell
> > syntax,
> 
> Yes, I understand that part.
> 
> > so this would be treated as a single word (e.g. when used in
> > a prerequisite list, in $(addprefix) etc.).
> 
> Yes, I understand that as well.
> 
> It's not the parsing part I am concerned about, or even how the value is
> interpreted in contexts where we know that the result should be a "list
> of words":, like a target or prerequisite or a function like addprefix,
> etc.: that is very clear.
> 
> > In the command-line it would also substituted as is, so no change from
> > the existing behaviour here.
> 
> Ouch; this statement is not precise and this is THE CRUX of the problem.
> What exactly do you mean by "substituted as is"?  As is what?

As it is stored internally, i.e. with the backslash. But indeed, we
must consider the various ways in which variables get their values:

- In the makefile (as in this example): Take it literally, including
  any quotes.

- As a result of $(wildcard) or similar functions: Make must insert
  quoting as necessary (so the shell will interpret it as the actual
  filenames). As discussed, there are several ways to do this,
  either backslash every character, or single-quote the whole
  string, escaping embedded single-quotes. I'd prefer the latter for
  readability, but that's an implementation detail.

- Targets from the command-line: Same as $(wildcard).

- Values from the command-line or environment variables (I think the
  two are equivalent): Now that's a little problem. One might like
  to quote them like the previous items, so something like
    make FOO="my file name"
  would work as intended. But that would break compatibility when a
  variable is actually meant to contain a list of filenames, or see
  my LOG_START example for another case that must not be turned into
  a single word. So I'd say since make has no way of knowing what a
  variable is meant to be, it must assume separate words and store
  it as is (also for backward-compatibility). If something else is
  intended, the user will need to do additional quoting:
    make FOO="'my file name'" BAR="file1 file2 file3"

> I'll refer you to my previous email with Eli where, when I asked what is
> the result of this makefile:
> 
> > >   FOO = foo\ bar
> > >   $(FOO): ; @echo '"$@"' '"$(FOO)"' '"$(addsuffix .txt,$(FOO))"'
> 
> His position was that it should yield this (he hoped):
> 
> > >   "foo bar" "foo bar" "foo bar.txt"
> 
> However:
> 
> > >   FOO = foo\ bar
> > >   $(FOO): ; @echo '$(FOO)' > '$@'
> > 
> > I'd say it should run "echo 'foo\ bar' > 'foo\ bar'", i.e. write to a
> > file with an embedded backslash.
> 
> OK, so this is not, after all, what I understood Eli to suggest.  I
> assume from this that in the previous example, you would expect to get
> output like this:
> 
>   "foo\ bar" "foo\ bar" "foo\ bar.txt"
> 
> ?

Exactly, because of the additional manual quoting in the recipe.

> What you're saying is that when a variable is expanded the backslashes
> are always preserved in the expansion.  If the result of the expansion
> is used in a context where make will break it up into words (for example
> in a target or prerequisite list, or as an argument to a function like
> addsuffix or addprefix) then make will pay attention to the backslashes
> when parsing words.  If it's in some other context, the result will just
> contain backslashes and it's up to the user to deal with them.

Yes. Actually, I think/fear make will not only have to deal with
backslashes in word-splitting contexts, but with the full glory of
quoting as supported by POSIX sh, including single and double
quotes, so $^ will actually mean the same to the shell than it did
to make in the dependency list. That may be tricky, but certainly
doable.

> > That's probably not what was intended, but I'd blame this on the
> > users.
> 
> Well, I think it's a little harsh to blame the users for doing what 40
> years of make history and portability have taught them is best-practice.

Is this so? I've seen a lot of makefiles that don't quote address@hidden (Then
again, I've seen a lot of shell scripts that don't quote their
variables when they really should.) So probably the best we can say
it that there's a lot of mess out there. There's no hope to make all
of it work correctly, so we can only choose one variant we want to
make work. (Which is still progress, since now none of them works
with spaces in filenames.) And given the choice, I'd take the
unquoted version because it's easier for users to write and read,
and because it fits better with this proposal.

Another thing to consider is $ in a value. Suppose we have two
strangely called files "$$" and "$(foo)" and this makefile:

foo = bar
files = $(wildcard *)

all:
        echo '$(files)'

To my surprise, it actually works correctly currently, doing
echo '$$ $(foo)'. I'd have expected echo '$ bar'. So apparently it
does some special handling of $ in filenames already. (I assume it
duplicates $ in $(wildcard) results.) However:

foo = bar
files = $(wildcard *)

all: $(files)
        echo '$(files)'

make: *** No rule to make target '$(foo)', needed by 'all'.  Stop.

which is wrong since '$(foo)' does exist. I don't really understand
this. It doesn't give this error about '$$', so apparently it does
handle embedded $ in dependency lists. strace shows that it doesn
stat './$(foo)' (successfully), so I'm not sure why it gives this
error at all.

Perhaps this is a plain bug, and if fixed, embedded $ would be no
problem at all.

But $ in command-line targets also seem to behave strangely:

foo = bar
%:
        touch '$@'

% make '$$'
touch '$$'

As I'd expect. (Apparently it does duplicate the $'s here.)

% make '$'
make: *** empty variable name.  Stop.
% make '$(foo)'
touch '$'

Not what I'd expect, but it seems fixable. Note, I'm only talking
about command-line targets here which we know must be (phony or
real) names. For command-line/environment variables we probably must
keep the existing behaviour:

foo = bar
all:
        touch '$(X)'

% make X='$$'
touch '$'
% make X='$(foo)'
touch 'bar'