Re: Quoting special characters (was: Re: Possible solution for special characters in makefile paths)

[Frank Heckenbach]

Tim Murphy wrote:

> Note that this use of cat is not "space safe"
> > FOO = a'()'b
> > all: $(FOO); cat $(FOO)
> 
> It should be:
> > all: $(FOO); cat "$(FOO)"
> 
> ... in which case the shell would act the same as make does.

But not in the way intended, because it would act on a file called
«a'()'b», including quotes, while this was meant to refer to a file
called «a()b». You see this more clearly if you consider the planned
use of backslash escapes:

FOO = a\ b

which we seem to agree should stand for a file called «a b» not
«a\ b», but with your «cat "$(FOO)"» the shell would see
«cat "a\ b"» and access a file called «a\ b».

That's why I keep saying, no manual quoting under this proposal. It
seems strange to you and me at first glance because we've learned to
quote variables in the shell, but if implemented like this in make,
it's actually correct to not quote manually. (And most users have
never learned to quote variables in the shell in the first place, so
they would do the right thing accidentally. :)

In case you wonder why things are different (manual quoting required
in the shell, not recommended in a recipe), at least I did wonder:
The reason is that the shell handles quoting *while* substituting
variables, i.e. it substitutes variables *differently* inside and
outside of "". Whereas in a make recipe, make substitutes $(FOO)
textually without caring about surrounding "", and then the shell
interprets the result of this substitution again, so the
make-variable handling and the "" handling are detached.

> Programmers should be responsible for what they put in the recipe
> without make having to second-guess them.

There's no second-guessing in my proposal. Indeed, I say make should
preserve the value exactly as written, including quotes.

Paul Smith wrote:

> On Tue, 2014-02-25 at 21:09 +0100, Frank Heckenbach wrote:
> > > Why do you think this will be needed?
> > 
> > When the makefile explicitly single-/double quotes stuff:
> > 
> > FOO = a'()'b
> > all: $(FOO); cat $(FOO)
> > 
> > If make doesn't handle ', it will depend on a file called «a'()'b»
> > including quotes, and give this on the command line, which to the
> > shell means «a()b», i.e. a different file name.
> 
> This seems like a bit of circular reasoning to me: we want to support
> quoting because if we don't then quoting won't be supported :-).

That's not exactly what I was saying. My point is that both make (in
target/dependency lists) and the shell must interpret some string
into words. If they do it differently, there is potential for
confusion and breakage.

Sure, you can call it "quoting won't be supported", as in, if you
use "FOO = a'()'b" like this, you will get undefined behaviour. (Not
strictly undefined, but not useful.)

> I don't immediately see the advantage in having the full range of shell
> quoting capabilities in make variable values.  Makefiles are not shell
> scripts; they just contain shell scripts.
>
> We need a way to quote special characters to make, and a single quoting
> mechanism should enough IMO.

As I've argued before, it's all well as long as you only consider
what happens internally in a makefile, but a makefile that doesn't
call any commands is not very useful, so I think we must go the
extra step and give the user a reliable way to use values in
recipes. Yes, makefiles only contain shell scripts, but they must
cooperate closely with them. In particular, most recipes are meant
to operate on the same files used as targets and dependencies.

Really, what would you like make to do in the situation above?

- Depend on «a'()'b» and let the shell operate on «a()b», although
  it knows (or could know) exactly that it's wrong?

- Fail when such a value is used in a target/dependency list
  ("FATAL: quoted string would be interpreted differently by the shell
  and make")?

- Hope the user doesn't write code like this and always massages
  their values before passing them to the shell (which we know won't
  happen)? (*)

- Do the right thing (from a user's point of view) and handle the
  file name like the shell does?
  Sure, it's more effort in make. AFAICS, you basically need three
  functions (either way):
  - find the next separator of a given set in a quoted string
  - decode a quoted string into unquoted words
  - quote a word (e.g. result of $(wildcard) or command-like target)
  Of course, these are more complex with full shell-style quoting
  than with backslashes only, but not terribly so. I've written such
  kinds of functions in other projects and I can provide them if you
  like.

(*) Note that if make supports backslash-escapes (only) and passes
them through to the shell as proposed, it gets even harder for the
user to correctly quote strings which may contain quote characters:

FOO = a'()'b\'c\ d

In a dependency list, this would then be interpreted as
«a'()'b'c d».

If make substitutes FOO as is and the user writes
«cat $(call Q, $(FOO))» (with my Q function as before; or the same
with an automatic SHELL_QUOTE applied), it would become
«cat 'a'\''()'\''b\'\''c\ d'» which is wrong (one \ too many).

Therefore, I'd say if make supports backslash-escapes only, it
should *not* automatically pass them to the shell. Instead the whole
quoting should then be left to the user or be done with something
like SHELL_QUOTE. So this case is actually more like the original
encoding proposal (with \ taking the place of $[] at the parsing
level only) than like Eli's and my shell-style-quoting proposal.
(Note that I didn't reject the original proposal, but I think the
latter is more comfortable to the user and thus preferable if it
doesn't cause backward-compatibility or POSIX compliance problems.)

Executive summary: Quoting is hard. It gets even harder if we have
different quoting regimes in makefiles and the shell because they
must cooperate, like it or not. Putting the burden of (re)quoting on
the user is where it least belongs and least likely to get right.