Re: posix command search and execution

[Mike Jonkmans]

On Tue, Nov 07, 2023 at 09:39:33AM +0700, Robert Elz wrote:
>     Date:        Mon, 6 Nov 2023 14:28:24 -0500
>     From:        Chet Ramey <chet.ramey@case.edu>
>     Message-ID:  <0ab6075e-22bf-43cd-992c-b2476f62629f@case.edu>
> 
>   | On 11/6/23 10:48 AM, Mike Jonkmans wrote:
>   | > According to these docs (what I make of it), resolving is done
>   | > in steps, the first applicable step is used:
> This is one of the most debated, and stupidest, parts of posix.

Unneeded complexity, I would say.

>   | > 1b) List several names that have unspecified results.
>   | This is an ad-hoc list of builtins that shells implement,
>   | not necessarily common across all shells.
> If it were just builtins it would not be important, the issue
> is more that some shells implement some of that list as reserved
> words, or aliases, and if that's done what applications can do
> alters dramatically.   So avoiding using those words as command
> names, except when using the known features of a specific shell,
> is the best way to remain portable.

Since we are dealing with 'Simple Commands', I hadn't yet even
considered reserved word and aliases. Makes sense though.

>   | > 1c) Use a function, for functions not matching standard utilities.
> No, that's not what it says, it is except of standard utilities
> implemented as functions.   More on that below.

I see the nuance. Thanks for pointing that out.
So a function is called, unless it is provided by the implementation and
matches a standard utility.
In particular, a user function with the name of a standard utility,
will be called at this point.

>   | > 1d) Lists 20 fixed utility names (like alias, cd etc.) that are
>   | >      to be invoked at his point. No PATH search yet.
>   | >         These are the `regular builtins'.
> In the next standard the ones listed are the intrinsic builtins,
> and includes only those that must be builtin to work.   But
> implementations can add more to the list.

Chet mentioned that. But I find the Austin-discussion hard to read.
It makes sense to partition the builtins in three categories with
a separate name for each.

>   | > 1eI) Search is successful.
>   | > 1eIa) Check for `regular builtins' and functions
>   | >       and invoke that regular builtin/function.
>   | >       Q: Shouldn't this specify an ordering for builtins/functions?
>   | The text seems to imply that you can't have both, doesn't it?
> While I suppose you could have both, it would be very unusual.

Unusuality sketch:
- the shell provides a builtin for a standard utility
- the distributor provides a function for the same utility in /etc/profile
  (maybe to mitigate some security issue)

Are scripts in /etc/profile considered part of the implementation?

...

>   | My feeling, without testing anything, is that most shells would allow
>   | functions to override builtins here.
> Since I have never seen any shell implement any standard utility
> as a function, it would be very hard to test.   Further if the
> did, also implementing the same thing as a builtin would be
> even harder to imagine - why do it twice when one of the two
> would never be used?   So not just hard to test - probably
> impossible.
> 
> It is also unclear to me why anyone would ever implement a standard
> builtin as a function - implementing builtins is simpler for the
> implementation than functions (in my experience anyway) and in
> any case, if the rules in the standard are followed, there is
> no way (except possibly by using "command", and even that is not
> clear to me) to tell if the implementation used a function or
> a builtin (maybe the output from type might make it clear, but
> not necessarily).

It is all kind of theoretical. What wonders me is that the POSIX
specifications and definitions sometimes are imprecise or lacking.

>   | This has been an area of significant disagreement.
> It has indeed.

Agreed to disagree.
 
>   | > 1eIb) Run the utility.
>   | >       (This is where ordinary builtins should run).
>   | >           (It seems logical that a builtin takes precedence over PATH).
>   | You'd be surprised.
> Yes.   But almost all shells implement it that way, so the
> seemingly logical assumption is mostly backed by experience.

Again this is not too precise for a standard.

>   | Note that this seems to require that you can only run
>   | a builtin if it exists (or something with that name exists) in $PATH.
> A builtin for a standard utility, yes.  Unless the implementation has
> defined it as intrinsic (which the forthcoming standard allows, but
> discourages).  Applications (which includes users) who invoke non
> standard utilities are stepping outside the standard, so get
> unspecified results (so implementations can add new non-standard
> builtins without also adding a matching command in PATH without
> issues.

It doesn't sound like the easiest way out.

>   | So if you have a builtin that doesn't exist in $PATH and isn't listed as
>   | one of the regular builtins, what do you do? Even the unspecified list
>   | doesn't give much help.
> If it is a standard utility it is required to exist in PATH.

Good to know. I was wondering about that.
But where is that being specified?
I already can't find the definition of standard utilities.

> If PATH has been changed so that is no longer true, then that
> is a non-conforming environment, and anything is OK.  Similarly
> if the builtin is not a standard utility (like declare or
> enable for example).

Ok. I was not aware that all standard utilities have to be in PATH.
But it makes sense.

...
> In practice this distinction (unlike some of the other properties
> os special builtins) rarely matters, as users typically have no
> reason to define functions that override the special builtins.

One might want to override `.' (source in bash parlance) for logging.

>   | I think the resolution to interpretation 854 addresses this. Shells
>   | who want this ordering just declare all the builtins they implement as
>   | `intrinsic' so they're not subject to a PATH search.
> Yes.

Can this intrinsic list be amended with any user loaded builtins?

>   | Kind of; regular builtins aren't really defined anywhere.
> They are, kind of (look in XBD) - they are just any utility
> the implementation happens to have implemented inside the shell.
> Aside from execution speed, you're not supposed to be able to
> tell the difference.   That's the inspiration for the PATH
> searching nonsense.  Or seems to be - but that isn't really
> correct, as if a user was to install their own version of
> a standard utility in PATH before the normal one (which is
> what all this is supposed to allow) then either that version
> implements the standard, in which case a user using only standard
> features still cannot tell the difference between that one and
> the builtin (speed ignored) or the standard no longer applies,
> and the results (including which gets invoked) are unspecified
> anyway.

Funny.

>   | >    Q: Where is `standard utilities' defined - as used in 1d.
>   | These are the standard utilities.
> I am not sure what "these" was intended to refer to, but standard
> utilities are the set of all the utilities defined in the standard.

That referred to the POSIX utilities link.
https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html

Thanks for the explanations.

-- 
Regards, Mike Jonkmans