Re: posix command search and execution

[Chet Ramey]

On 11/6/23 9:39 PM, Robert Elz wrote:

>    | > 1eI) Search is successful.
>    | > 1eIa) Check for `regular builtins' and functions
>    | >        and invoke that regular builtin/function.
>    | >        Q: Shouldn't this specify an ordering for builtins/functions?
>    |
>    | The text seems to imply that you can't have both, doesn't it?
>
> While I suppose you could have both, it would be very unusual.

Granted.

> Again, the functions that can get invoked here are only the
> standard utilities implemented as functions, all others would
> have been invoked earlier, and we would never be here.

It does imply that the implementation can't use a shell function to
override one of the utilities in 1d, since those are standard.

There goes your implementation-provided `cd' replacement function. :-)

Luckily, interp 854 resulted in changes here.

>    | My feeling, without testing anything, is that most shells would allow
>    | functions to override builtins here.
>
> Since I have never seen any shell implement any standard utility
> as a function, it would be very hard to test. 

I think ksh93 has some, or used to.

>  Further if the
> did, also implementing the same thing as a builtin would be
> even harder to imagine - why do it twice when one of the two
> would never be used?   So not just hard to test - probably
> impossible.

Presumably the shell function has features beyond the builtin, and uses
the builtin for the basics. There's no good -- or useful -- way to do the
opposite.

>    | This has been an area of significant disagreement.
>
> It has indeed.
>
>    | > 1eIb) Run the utility.
>    | >        (This is where ordinary builtins should run).
>    | >         (It seems logical that a builtin takes precedence over PATH).
>    |
>    | You'd be surprised.
>
> Yes.   But almost all shells implement it that way, so the
> seemingly logical assumption is mostly backed by experience.

See below about the business of not invoking builtins that aren't in 1d
unless they're found via a PATH search.

>    | Note that this seems to require that you can only run
>    | a builtin if it exists (or something with that name exists) in $PATH.
>
> A builtin for a standard utility, yes.  

If the standard wants to say that, it can. It doesn't. The language in 1e
doesn't restrict itself to "standard utilities." It's any simple command
that the shell may have implemented as a regular builtin. That's one of
the problems here.


> Unless the implementation has
> defined it as intrinsic (which the forthcoming standard allows, but
> discourages). 

It shouldn't discourage that practice. It's a way for a shell to provide
users with certainty about lookup order.

> Applications (which includes users) who invoke non
> standard utilities are stepping outside the standard, so get
> unspecified results (so implementations can add new non-standard
> builtins without also adding a matching command in PATH without
> issues.

Not necessarily. Invoking a command that isn't defined in the standard
results in command behavior and effects that are outside the standard (of
course), but the way that command is invoked and the order in which
builtins/instrinsics/functions/executables are found is in the standard.


>    | So if you have a builtin that doesn't exist in $PATH and isn't listed as
>    | one of the regular builtins, what do you do? Even the unspecified list
>    | doesn't give much help.
>
> If it is a standard utility it is required to exist in PATH.

The standard's language doesn't restrict itself to standard utilities.

> If PATH has been changed so that is no longer true, then that
> is a non-conforming environment, and anything is OK. 

That's not necessary.

> Similarly
> if the builtin is not a standard utility (like declare or
> enable for example).

The language in 1e doesn't restrict itself to standard utilities. If
that was the intent, the standard should have made it explicit. You can
always say "all bets are off if the command name isn't the name of one
of the utilities defined in this standard" but that isn't practical, and
the standard itself doesn't say that.

>    | This is a quality of implementation feature.
>    | Why confuse users by allowing them to define a function that
>    | will never be executed?
>
> Indeed - but you could also write that as "Why confuse users by
> allowing them to define a function that can never be invoked?"
> and by so doing, encourage more portable scripts.

What's the difference? Either way, you can't define a function with the
same name as a special builtin.

> In practice this distinction (unlike some of the other properties
> os special builtins) rarely matters, as users typically have no
> reason to define functions that override the special builtins.

Ha, you'd be surprised. It's rare, but it happens. `exit' is the one I've
seen most often (yes, even in the presence of the EXIT trap).

>    | I think the resolution to interpretation 854 addresses this. Shells
>    | who want this ordering just declare all the builtins they implement as
>    | `intrinsic' so they're not subject to a PATH search.
>
> Yes.
>
>    | Kind of; regular builtins aren't really defined anywhere.
>
> They are, kind of (look in XBD) - they are just any utility
> the implementation happens to have implemented inside the shell.

Right. Except for the command lookup properties (and the stupid $PATH
searching requirements), they're mostly defined by how they differ from
special builtins.

> Aside from execution speed, you're not supposed to be able to
> tell the difference.   That's the inspiration for the PATH
> searching nonsense. 

The motivation seems to be giving the script writer control of what gets
executed, by allowing them to set $PATH and disable all non-standard
builtins (`enable' was always the favorite example, since it used to be
a command in a printing package). The quote from the rationale I posted
supports that.

https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_09_02

with the additional nonsense that the standard seems to say that regular
builtins can't affect the shell environment (in the non-normative text).

> Or seems to be - but that isn't really
> correct, as if a user was to install their own version of
> a standard utility in PATH before the normal one (which is
> what all this is supposed to allow) 

The rationale doesn't cover that, but seems very concerned with the shell
programmer not knowing what's built into the shell. I suppose if a script
restricted itself to calling utilities defined by the standard, it would
be ok, but as you can see from the historical `enable' discussion, that
wasn't the point.

I re-read the discussion from 2014-2015 and there was considerable
emphasis on ensuring that users can run commands from the file system
that have the same names as shell builtins.

>    | >    Q: Where is `standard utilities' defined - as used in 1d.
>    | These are the standard utilities.
>
> I am not sure what "these" was intended to refer to, but standard
> utilities are the set of all the utilities defined in the standard.

That's what I meant -- standard utilities are the ones defined in the
standard. The link Mike posted immediately above that question is to
the index of the POSIX utilities.

Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/