Quoting special characters (was: Re: Possible solution for special characters in makefile paths)

[Paul Smith]

Thanks Eli, as the person probably most familiar with both non-POSIX
uses of make and also locales and encodings, I'm anxious for your
opinions on this.

On Sat, 2014-02-22 at 10:26 +0200, Eli Zaretskii wrote:
> But what about the alternatives, like quoting of some kind?  One
> possibility is to use ".." to quote file names with whitespace;
> another is to escape whitespace with a backslash, like we already do
> for a colon.
> 
> Given that recoding whitespace is not a straightforward solution, and
> possibly brings complications, I'd suggest to talk about those
> alternatives as well.  Perhaps their complications are easier handled,
> and they are definitely more natural in use.

That's a good point: I had thought about these but I didn't discuss my
thinking.  Let's consider this solution.

So the first thing to decide is whether the quoting exists only internal
to make and is removed before the string is passed to any external
interface (shell, stat(), stdout, environment, etc.), or if the quoting
will be preserved and passed along to the commands make invokes (not
stat() obviously).

I think it's clear that the solution MUST only be internal to make.
Regardless of the quoting behaviors make employs, we cannot be sure that
whatever utility we use to run recipes will treat quotes the same way.
Users can choose whatever SHELL value they want, for example, and I
certainly would not want to provide a solution that worked only when a
POSIX-y shell was used.

So, while quoting may look natural in the makefile, it doesn't help with
the issues Frank raised regarding writing recipes correctly when paths
contain special characters: we'd be stripping that quoting before
running the recipe in any event.

Next lets consider the parsing of these values.  Here I think trying to
re-use any existing quoting runs into very significant problems.  There
may be a few handfuls of makefiles that use the mapping special
characters, but there must be many thousands of makefiles that include
quotes and/or backslashes today.  Changing the way make interprets these
very common characters would be a HUGE backward-compatibility break, I
think.  Remember we're not just talking about paths appearing in target
and prerequisite lists where we at least have some context and can
perhaps expect that the special characters are uncommon: we must also
consider variable values which have no context: some may be used as
target/prerequisites, but others will just be used in a recipe for
completely unrelated purposes.  And, due to make functions, we want to
chop these variable values up into words _all the time_.  That
word-chopping internal to make must take into account quoting/escaping.

Today if you have:

    FOO := "foo "bar "biz
    BAR := $(addsuffix ",$(FOO))

the result will be: "foo" "bar" "biz".  How do we deal with this if
quoting is special to make?  Using backslashes has similar problems,
especially on Windows systems where you might expect backslashes to be
used in variables as a matter of course.  Today make does not reserve
backslashes at all UNLESS they're quoting a character that's special to
make... and whitespace isn't.

Personally, I can't begin to articulate a coherent set of rules that
would allow this to work... but I'm certainly open to the idea that I'm
missing something.

If make had a coherent backslash escaping mechanism (for example,
something like the shell where backslash always escapes the next
character and if you want a backslash you must always write "\\") then
we would be all set.  But instead make's implementation tries to be too
smart and only interprets backslashes as escapes when they appear in the
context of a special character... this means we can never create a new
special character without enormous backward-compat problems!  Consider
how many variable values out there are using backslashes to quote
whitespace in preparation for the shell, that would have to be changed!
Consider:

    SWITCH = switch "$(FOO)" in (foo\ bar) echo hi ;; esac

!!

So I believe that we still need to invent a new quoting mechanism in GNU
make, such as $[...] or similar, and we cannot appropriate any of the
existing common quoting syntax and bend it to a new meaning, without
massive breakage.

The only remaining issue is the form of the internal representation and
how that might impact existing makefiles.  I have another thought about
that, that I'll post in a separate message shortly.