parameter expansion with `:` does not work

[lisa-asket]

From: Greg Wooledge <greg@wooledge.org>
To: bug-bash <bug-bash@gnu.org>
Subject: Re: parameter expansion with `:` does not work
Date: 08/07/2021 05:58:57 Europe/Paris

On Thu, Jul 08, 2021 at 04:38:25AM +0200, lisa-asket@perso.be wrote:
> I'd rather understand what's going on, rather than simply never use it.

OK. Let's start from the beginning.

In sh and bash, anything that begins with a $ is potentially a
substitution, a.k.a. an expansion. The parser will try to unravel
the punctuation soup to figure out where the beginning and ending of
the substitution are. Then, that piece of the command will be replaced
with zero or more words. All of it is dependent on context, and rules
that have grown organically and chaotically over a span of decades.

Let's say you have a command like this:

foo $bar

The thing on the right hand side, which begins with $ and ends with r,
is a substitution. Specifically, it's a parameter expansion, which means
a "parameter" (which is either a variable or a special parameter, in this
case a variable) will have its value pulled from memory and used for the
substitution.

If we have a variable named bar, its value gets substituted into the
command. Since $bar is not quoted, that value undergoes two more rounds
of substitutions (word splitting, and filename expansion). At the end
of all that action, we will have a list of zero or more words, and these
words will become the arguments of the foo command.

OK so far? Good.



*** Ok



Now let's say we have this command:

: $bar

Once again, the value of the variable named bar is substituted, and then
undergoes word splitting and filename expansion, and the results of that
become the arguments to the : command.

The : command does nothing, so the end result of all this work is ...
nothing. Except that we may hit the file system a few times in order to
perform filename expansions, if there are any glob characters in the
variable's value. But that's it.

So, why would anyone write a command like this? It's because some
substitutions have side effects.

Let's look at this command next:

: $((x++))

Now, this is a silly command, and you wouldn't write this in real life,
because it's just more complex than it needs to be. But I'm demonstrating
something, so stick with it for a moment, please.

This time, we don't have a parameter expansion. We have an arithmetic
substitution instead.

The stuff inside the $(( )) gets passed to a special arithmetic parser,
which has its own special rules. These rules look a lot like the rules
of the C language, by some strange coincidence.

This particular arithmetic expression x++ uses the post-increment operator
++ to add 1 to the value of an existing (or even nonexistent) variable.
This is a *side effect*, meaning that it does something more than just
producing a value for our substitution. It has some lasting effect.

So, what happens here? In order:

1) The value of x is pulled from memory and stored in a temporary spot.
If x doesn't exist, we use the value 0. If x contains a string that
can be treated as an integer, we use that value. Otherwise, we attempt
to perform recursive arithmetic evaluation. We won't cover all of that
right now.

2) We take the value from step 1, add 1 to it, and store this back into x.

3) The value from step 1 (before we added 1) is used as the value of the
substitution.

4) The value of the substitution would undergo word splitting and filename
expansion because of the lack of quotes, except that the result of
an arithmetic expansion is always an integer, and therefore can't do
those things.

5) The value of the substitution is used as the argument of the : command,
which does nothing.

So, the whole point of this demonstration was what happens in step 2. The
value of x is changed, even though we used a command that normally does
nothing. The change happens *during* the expansion. It's independent of
the command that we used.

The only reason we have the : command here at all, is so that we don't
try to execute the value of x as a command. If we left out the : we
would get something like

bash: 0: command not found

We don't want that. So that's why the : is there.

Now that you understand everything that's going on, let's look at this
crazy shit from 1977 that you've fallen in love with:



*** I understood





: ${foo:=bar}

What happens here? We have a parameter expansion with a special modifier.
According to the documentation which has been quoted at you multiple
times already, this is a two-step expansion. The value of the variable
named foo is pulled from memory. If this value is the empty string, or
if the variable foo does not currently exist, then an *assignment* takes
place, and the string bar is stored in the variable foo, and then that
string (bar) also becomes the value of the substitution.

So, we perform the following steps:

1) The value of the variable foo is pulled from memory.

2) If the value from step 1 is the empty string (or if there's no variable
named foo yet), the string bar is *assigned* to the variable foo, and
the string bar also becomes the value that we pulled from memory.

3) The string we pulled from memory undergoes word splitting and filename
expansion, because of the lack of quotes.

4) The list of words from step 3 become the arguments of the : command,
which does nothing.

So, once again we have a command that normally does nothing, *except* that
we've used a substitution that has a side effect. The side effect is the
entire purpose of this thing. We wanted to assign this "default" value
(bar) to our variable (foo) if our variable wasn't already set to some
other value.

And, once again, we had to put a command (we used : which does nothing)
in there, so that the value of the variable isn't used as a command. If
we left out the : command, we would get something like

bash: bar: command not found

And we don't want that. So that's why the : is there.



*** Till here I also understand



Now, all of this is complete nonsense and is totally wrong for the problem
you are trying to solve.

Why?

Because the problem you are trying to solve DOES NOT INVOLVE THERE EVER
BEING A PREEXISTING VALUE IN THE VARIABLE.

You want to initialize a variable (in your case, it's a list of filename
extensions) to a default value, which will be used if the user does not
provide a list of their own.

But the user will NOT provide this list in a variable. So there's no
variable that could possibly contain the user's overriding list. So
there's no reason to use the conditional "assign a default value if
the variable is unset or empty" syntax.



*** This is where the misunderstanding starts.  If the user supplies

a value, it is put in the variable.  Then the default assignment will

not happen.



It simply makes no sense.

Also, the construct is dimensionally wrong. Your default value is
a list of two elements. If the user provides a list of extensions,
it will be of indeterminate length; you'll have to assign it to an array.
But the "assign default value" construct doesn't understand array variables.
It's from the Bourne shell which didn't HAVE array variables. It only
had string variables. It only deals with string values.

You don't have string values. You have lists.

So it's just WRONG. Twice.



*** Not really.  I use strings, the array comes later.  I use "assign default 
value" 

to a string variable.



You're following an anti-pattern that I see all the time. You've come
across some new thing and you want to use this new thing, because it
excites you. So you're trying desperately to find a way to use this
shiny new thing.

The problem is, this thing is not suited to what you're actually doing.

And yet, you're bending over backwards trying to find SOME WAY to use
it, twisting everything around, digging deeper and deeper into code that
simply should not be written. Not for this project.

You want to pass an optional list of extensions to a script, using a
comma-delimited list inside a single string argument? OK, fine. It's
not a design choice that I would use, but it's one of the valid

possibilities. 



*** At least we agree that it is a possibility.  I understand this possibility 
but have

problem understanding your design choice.



And clearly you WILL NOT STOP until you have a working
answer for it, so fine. HERE IT IS. I would rather write it for you
than watch you continue doing the crap you've been doing.



*** Mostly because I understand how it would work with strings as option.

Might be wrong, but my impression is that you would pass an actual bash list

construct to your function.  But that would be too advanced for me.  Still I 
can try.



#!/bin/bash

# Usage: print-stuff [-e extlist] startline endline [startdir]
# extlist is a list of extensions, separated by commas. Don't include
# the dot.

# Default list of extensions. Use this if the user doesn't supply one.
exts=(foo bar)

# Default starting directory. Use this if the user doesn't provide a
# startdir (third non-option) argument.
startdir=.

# Process the options, if any exist.
while true; do
case $1 in
-e) IFS=, read -ra exts <<< "$2,"
shift 2;;
--) shift; break;;
*) break;;
esac
done

# Count the non-option arguments.
case $# in
2) : ;;
3) startdir=$3;;
*) echo "usage: ..." >&2; exit 1;;
esac

# At this point, we have:
# startline in $1
# endline in $2
# startdir (string var, either the user's or the default)
# exts (array var, either the user's or the default)

# The rest of the script goes here. Already been shown. Generate the
# array of find arguments dynamically, and call find | awk.

# You will note that this script does not use ${foo:-bar} or ${foo:=bar}
# because there is no REASON to use either of them.



*** Now I understand you.  Because you set the defaults first.  I set them 
after,

cheking if the local variable exists.



# The only time it makes sense to use one of them is if the script accepts
# optional inputs in environment (or shell) variables. We are not using
# environment variables to provide options in this script. Or in most
# scripts that are used in interactive shell sessions.

# What kind of script takes environment or shell variables for options?
# Typically sysv-rc boot scripts, which read optional configurations
# by dotting in files from /etc/default/. It's a highly specialized

# problem space. And you are not in it.



*** Ok, so you use the construct for global environment variables, not

for local variables as well.



Was not too hard to understand actually.