Re: posix command search and execution

[Chet Ramey]

On 11/6/23 10:48 AM, Mike Jonkmans wrote:

> POSIX Command Search
>
> According to these docs (what I make of it), resolving is done in steps,
> the first applicable step is used:
>
> 1) No slash in the name.
>
> 1a) Run the builtin if the name matches that of a Special Builtin.
>
> 1b) List several names that have unspecified results.

This is an ad-hoc list of builtins that shells implement, not necessarily
common across all shells.

> 1c) Use a function, for functions not matching standard utilities.
>
> 1d) Lists 20 fixed utility names (like alias, cd etc.) that are
>      to be invoked at his point. No PATH search yet.
>         These are the `regular builtins'.
>         (These need not exist as builtin).

These are the historical common builtins.

> 1e) Search via PATH.
> 1eI) Search is successful.
> 1eIa) Check for `regular builtins' and functions
>        and invoke that regular builtin/function.
>        Q: Shouldn't this specify an ordering for builtins/functions?

The text seems to imply that you can't have both, doesn't it? My feeling,
without testing anything, is that most shells would allow functions to
override builtins here.

This has been an area of significant disagreement. The two camps basically
break down into giving users more flexibility by prioritizing functions to
giving script writers more power to determine exactly what is executed.
This covers both functions that override `regular' builtins and those that
override non-builtin utilities. (There is more nuance than in this
description.)

POSIX is pretty definite about which camp the standard is in: "The sequence
selected for the Shell and Utilities volume of POSIX.1-2017 acknowledges
that special built-ins cannot be overridden, but gives the programmer full
control over which versions of other utilities are executed."

>           Q: Why check for regular builtins? That was already done in 1d.

Implementations can provide other builtins. The check in 1d is only for
those specific ones.

> 1eIb) Run the utility.
>        (This is where ordinary builtins should run).
>           (It seems logical that a builtin takes precedence over PATH).

You'd be surprised. Note that this seems to require that you can only run
a builtin if it exists (or something with that name exists) in $PATH.

Look at https://www.austingroupbugs.net/view.php?id=854 for a discussion
of this issue.

> 1eII) Nothing in PATH, exit with 127

So if you have a builtin that doesn't exist in $PATH and isn't listed as
one of the regular builtins, what do you do? Even the unspecified list
doesn't give much help.

> 2) Slash in name? Try to execute; prescribed exits: 127 or 126.
>
> I hope I have understood POSIX correctly on these points.
                [...]

> In posix mode, it seems that bash:
> - 1a) Honors special builtins (but see 1c).
> - 1b) With source as a special builtin - which is ok (as unspecified).
> - 1c) Doesn't allow you to define a function with the name
>        of a special builtin.

This is a quality of implementation feature. Why confuse users by allowing
them to define a function that will never be executed?

>        A function defined before `set -o posix' will mask a
>           special builtin. (This seems to be ok).

It will not, at least not while posix mode is enabled. If you mean that a
function will be executed before a special builtin when not in posix mode,
you are correct, because there are no special builtins when not in posix
mode.

>        Utilities are masked by a function of that name.
> - 1d) All names in the list, except newgrp, are bash-builtins
>        and are used.
> - 1eIa)
>        Functions don't need a successful path search - per 1c.
> - 1eIa & b)
>        Builtins are also ran regardless of path search.
> - *) ok.
>
> Regarding source and 1c in bash, a mail from our beloved maintainer:

Flatterer.

> https://lists.gnu.org/archive/html/bug-bash/2014-03/msg00084.html


> Thus in posix mode, bash does not follow this part of the standard.

Exactly which part of the standard are you saying bash is not following?
The requirements concerning PATH search and builtins are different in the
next version of POSIX, the result of interp 854. The standard already
says this about functions with the same name as a special builtin:

"The function is named fname; the application shall ensure that it is a name
(see XBD Name) and that it is not the name of a special built-in utility."

> But should it?
> I would rather have POSIX modified to *also* accept the, more logical,
> bash way (i.c. first matching functions, then builtins, then PATH).
> Would that be a feasible modification to suggest to the Austingroup?

I think the resolution to interpretation 854 addresses this. Shells
who want this odering just declare all the builtins they implement as
`intrinsic' so they're not subject to a PATH search. That way there's no
difference between the regular builtins and the ones an implementation
chooses to provide. It still leaves posix special builtins, but I think
those are with us forever.


> - Regarding 1eIb.
>    The shells posh, dash, ksh and zsh
>    also run builtins, even when not found in PATH.
>    Checked with the `test' builtin (mv /usr/bin/test{,.sav})
>    on the versions found on Ubuntu 22.04.

Yes, this is part of the discussion of interp 854. The business of running
builtins other than the ones listed only after a PATH search was always
ahistorical.

> - The 'newgrp' utility (mentioned in 1d) is not a builtin in bash.
>    This is ok. The regular builtins from 1d need not be provided. See:
>    https://lists.gnu.org/archive/html/bug-bash/2005-02/msg00129.html
>    Builtins are defined in:
>    
> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_83
>    Q: Isn't that incorrect in stating where regular builtins are defined?

Kind of; regular builtins aren't really defined anywhere. They're builtins
that aren't special builtins, and they're mostly defined in terms of how
they differ from special builtins.

> - In bash's code builtins/builtins.c (made from builtins/mkbuiltins.c),
>    the regular builtins are flagged with POSIX_BUILTIN.
>    But `hash' is not. Omission or intention?

Omission, thanks. hash/type/ulimit were not added to this list until 2016
(one of the TC releases to the 2013 standard).

https://www.austingroupbugs.net/view.php?id=705 is where the discussion
happened.

>    Also, I don't see any use of the POSIX_BUILTIN flag. Remove?

I'm going to leave it in case POSIX ever gets stricter about the
requirements for the builtins listed in 1d. For a while, it seemed like
that's how it was going to go.


> - Utilities:
>    https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html
>    Q: Where is `standard utilities' defined - as used in 1d.

These are the standard utilities.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/