Bash Ref Manual on tokenization omits consideration of expansions

[Allen Halsey]

Configuration Information [Automatically generated, do not change]:
Machine: i686
OS: cygwin
Compiler: gcc-4
Compilation CFLAGS:  -DPROGRAM='bash.exe' -DCONF_HOSTTYPE='i686' 
      -DCONF_OSTYPE='cygwin' -DCONF_MACHTYPE='i686-pc-cygwin' 
      -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' 
      -DSHELL -DHAVE_CONFIG_H -DRECYCLES_PIDS   -I.  
      -I/usr/src/bash-3.2.49-23/src/bash-3.2 
      -I/usr/src/bash-3.2.49-23/src/bash-3.2/include 
      -I/usr/src/bash-3.2.49-23/src/bash-3.2/lib   -O2 -pipe 
uname output: CYGWIN_NT-5.1 universi-c1eec2 1.7.0(0.218/5/3) 
      2009-12-04 17:08 i686 Cygwin
Machine Type: i686-pc-cygwin

Bash Version: 3.2
Patch Level: 49
Release Status: release

Description:

 This bug report against the Bash Reference Manual stems from a discussion 
 on bug-bash [1].

 The Bash Reference Manual's description of Shell Operation [2] omits 
 from its description of tokenization (Step 2) that the shell must 
 recognize expansion constructs.

 Step 2 says:

  [The shell] breaks the input into words and operators, obeying the 
  quoting rules described in Quoting. These tokens are separated by 
  metacharacters. Alias expansion is performed by this step (see 
  Aliases).

 Applying this tokenization description to a simple example:

  [me@host ~] $ echo The date is $(date +"%Y-%m-%d")

 would result in this list of tokens:

  -------     -------------
  echo        word
  <space>     metacharacter
  The         word
  <space>     metacharacter
  date        word
  <space>     metacharacter
  is          word
  <space>     metacharacter
  $           word
  (           metacharacter
  date        word
  <space>     metacharacter
  +"%Y-%m-%d" word
  )           metacharacter


 A reading of POSIX re Shell Token Recognition [3] indicates that shell 
 tokenization respects substitution candidates intact. The list of tokens 
 is the simple example then become:

  ------------------- -------------
  echo                word
  <space>             metacharacter
  The                 word
  <space>             metacharacter
  date                word
  <space>             metacharacter
  is                  word
  <space>             metacharacter
  $(date +"%Y-%m-%d") word

[1] http://thread.gmane.org/gmane.comp.shells.bash.bugs/14377
    or
    http://lists.gnu.org/archive/html/bug-bash/2010-02/msg00119.html
    
[2] http://www.gnu.org/software/bash/manual/bashref.html#Shell-Operation

[3]
http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_03


Repeat-By:


Fix:

 I recommend that the first sentence of Step 2 be appended with the 
 following:

  "and keeping intact sequences of characters comprising expansions (see
   Shell Expansions)."

Allen
--
Allen Halsey