Proposed changes to shell.el and misc.texi.

[Luc Teirlinck]

I propose below patches for shell.el and misc.texi.  The main problem
they address concerns `shell-prompt-pattern' and
`comint-use-prompt-regexp-instead-of-fields', although the patch to
misc.texi also contains various other changes and introduces a new
node to better explain the subtleties of
`comint-use-prompt-regexp-instead-of-fields'.

Just to make sure that I am not documenting a bug, here is the
situation, which was incorrectly described in both the Emacs manual
and the `shell-prompt-pattern' docstring.  That docstring says that
the variable has no effect if `comint-use-prompt-regexp-instead-of-fields'
is nil.  This is correct if you do not care about paragraphs, but
wrong otherwise.

The Emacs manual says that, in Shell mode, only prompts start new
paragraphs.  Strictly speaking, this is perfectly correct, but it is
also very misleading.  If `comint-use-prompt-regexp-instead-of-fields'
is non-nil, it is really a perfectly valid description.  However, if
`comint-use-prompt-regexp-instead-of-fields' is nil, the default, then
for all practical purposes, the paragraph commamds behave as if there
were three kind of paragraphs: input, prompts and non-prompt output.
(This is because the paragraph commands are constrained by fields).
But then again, this description is only valid if
`shell-prompt-pattern' correctly describes prompts.  Otherwise, the
paragraph commands are completely unpredictable (and hence essentially
useless) if `comint-use-prompt-regexp-instead-of-fields' has its
default value.

The patches below treat all of this as being just a documentation bug.
(They assume that constraining paragraph commands to fields makes
sense if fields are used.)

===File ~/shell.el-diff=====================================
*** shell.el    29 Dec 2003 13:27:11 -0600      1.125
--- shell.el    06 Apr 2005 19:00:06 -0500      
***************
*** 136,143 ****
  This variable is used to initialise `comint-prompt-regexp' in the
  shell buffer.
  
! This variable is only used if the variable
! `comint-use-prompt-regexp-instead-of-fields' is non-nil.
  
  The pattern should probably not match more than one line.  If it does,
  Shell mode may become confused trying to distinguish prompt from input
--- 136,144 ----
  This variable is used to initialise `comint-prompt-regexp' in the
  shell buffer.
  
! If `comint-use-prompt-regexp-instead-of-fields' is nil, then this
! variable is only used to determine paragraph boundaries.  See
! Info node `Shell Prompts' for how Shell mode treats paragraphs.
  
  The pattern should probably not match more than one line.  If it does,
  Shell mode may become confused trying to distinguish prompt from input
============================================================

===File ~/misc.texi-diff====================================
*** misc.texi   29 Mar 2005 12:31:09 -0600      1.64
--- misc.texi   06 Apr 2005 19:09:01 -0500      
***************
*** 342,347 ****
--- 342,348 ----
  * Single Shell::           How to run one shell command and return.
  * Interactive Shell::      Permanent shell taking input via Emacs.
  * Shell Mode::             Special Emacs commands used with permanent shell.
+ * Shell Prompts::          Two ways to recognize shell prompts.
  * History: Shell History.  Repeating previous commands in a shell buffer.
  * Directory Tracking::     Keeping track when the subshell changes directory.
  * Options: Shell Options.  Options for customizing Shell mode.
***************
*** 405,411 ****
  either or both of these default address@hidden
  
    Both @kbd{M-!} and @kbd{M-|} wait for the shell command to complete,
! unless you end the command with @samp{&} to make it asyncronous.  To
  stop waiting, type @kbd{C-g} to quit; that terminates the shell
  command with the signal @code{SIGINT}---the same signal that @kbd{C-c}
  normally generates in the shell.  Emacs waits until the command
--- 406,412 ----
  either or both of these default address@hidden
  
    Both @kbd{M-!} and @kbd{M-|} wait for the shell command to complete,
! unless you end the command with @samp{&} to make it asynchronous.  To
  stop waiting, type @kbd{C-g} to quit; that terminates the shell
  command with the signal @code{SIGINT}---the same signal that @kbd{C-c}
  normally generates in the shell.  Emacs waits until the command
***************
*** 506,513 ****
  At end of buffer send line as input; otherwise, copy current line to
  end of buffer and send it (@code{comint-send-input}).  When a line is
  copied, any prompt at the beginning of the line (text output by
! programs preceding your input) is omitted.  (See also the variable
! @code{comint-use-prompt-regexp-instead-of-fields}.)
  
  @item @key{TAB}
  @kindex TAB @r{(Shell mode)}
--- 507,514 ----
  At end of buffer send line as input; otherwise, copy current line to
  end of buffer and send it (@code{comint-send-input}).  When a line is
  copied, any prompt at the beginning of the line (text output by
! programs preceding your input) is omitted.  @xref{Shell Prompts}, for
! how Shell mode recognizes prompts.
  
  @item @key{TAB}
  @kindex TAB @r{(Shell mode)}
***************
*** 562,568 ****
  @kindex C-c C-u @r{(Shell mode)}
  @findex comint-kill-input
  Kill all text pending at end of buffer to be sent as input
! (@code{comint-kill-input}).
  
  @item C-c C-w
  @kindex C-c C-w @r{(Shell mode)}
--- 563,570 ----
  @kindex C-c C-u @r{(Shell mode)}
  @findex comint-kill-input
  Kill all text pending at end of buffer to be sent as input
! (@code{comint-kill-input}).  If point is not at end of buffer,
! this only kills the part of this text that precedes point.
  
  @item C-c C-w
  @kindex C-c C-w @r{(Shell mode)}
***************
*** 684,693 ****
  @end example
  @end table
  
-   Shell mode also customizes the paragraph commands so that only shell
- prompts start new paragraphs.  Thus, a paragraph consists of an input
- command plus the output that follows it in the buffer.
- 
  @cindex Comint mode
  @cindex mode, Comint
    Shell mode is a derivative of Comint mode, a general-purpose mode for
--- 686,691 ----
***************
*** 704,709 ****
--- 702,747 ----
  in a subprocess using unmodified Comint mode---without the
  specializations of Shell mode.
  
+ @node Shell Prompts
+ @subsection Shell Prompts
+ 
+ @vindex shell-prompt-pattern
+ @vindex comint-prompt-regexp
+ @vindex comint-use-prompt-regexp-instead-of-fields
+ @cindex prompt, shell
+   Normally, any text output by a program at the beginning of an input
+ line is considered a prompt.  However, if the variable
+ @code{comint-use-prompt-regexp-instead-of-fields} is address@hidden,
+ then Comint mode uses a regular expression to recognize prompts.  In
+ general, the variable @code{comint-prompt-regexp} specifies the
+ regular expression; Shell mode uses the variable
+ @code{shell-prompt-pattern} to set up @code{comint-prompt-regexp} in
+ the shell buffer.
+ 
+   The value of @code{comint-use-prompt-regexp-instead-of-fields} also
+ affects many motion and paragraph commands.  If the value is
+ address@hidden, motion commands behave as usual.  However, if the
+ value is @code{nil}, the default, then Shell mode divides the buffer
+ into two types of ``fields'' (ranges of consecutive characters having
+ the same @code{field} text property): input and output.  Prompts are
+ output.  Most motion commands do not cross field boundaries, unless
+ they move over multiple lines.  For instance, when point is in input
+ on the same line as a prompt, @kbd{C-a} puts point at the beginning of
+ the input if @code{comint-use-prompt-regexp-instead-of-fields} is
+ @code{nil} and at the beginning of the line otherwise.
+ 
+   Shell mode customizes the paragraph commands so that only shell
+ prompts start new paragraphs.  (For this purpose, it uses
+ @code{shell-prompt-pattern}, regardless of
+ @code{comint-use-prompt-regexp-instead-of-fields}.)  All paragraph
+ commands behave accordingly if
+ @code{comint-use-prompt-regexp-instead-of-fields} is address@hidden
+ However, if the value of that variable is @code{nil}, most paragraph
+ commands behave as if prompt, input and non-prompt output all were
+ separate paragraphs (assuming @code{shell-prompt-pattern} has a correct
+ value).  This is because they are constrained by field boundaries
+ inside the paragraph.
+ 
  @node Shell History
  @subsection Shell Command History
  
***************
*** 810,816 ****
  
  @vindex shell-input-ring-file-name
    Some shells store their command histories in files so that you can
! refer to previous commands from previous shell sessions.  Emacs reads
  the command history file for your chosen shell, to initialize its own
  command history.  The file name is @file{~/.bash_history} for bash,
  @file{~/.sh_history} for ksh, and @file{~/.history} for other shells.
--- 848,854 ----
  
  @vindex shell-input-ring-file-name
    Some shells store their command histories in files so that you can
! refer to commands from previous shell sessions.  Emacs reads
  the command history file for your chosen shell, to initialize its own
  command history.  The file name is @file{~/.bash_history} for bash,
  @file{~/.sh_history} for ksh, and @file{~/.history} for other shells.
***************
*** 876,894 ****
  @key{SPC} perform history expansion by binding @key{SPC} to the
  command @code{comint-magic-space}.
  
- @vindex shell-prompt-pattern
- @vindex comint-prompt-regexp
- @vindex comint-use-prompt-regexp-instead-of-fields
- @cindex prompt, shell
    Shell mode recognizes history references when they follow a prompt.
! Normally, any text output by a program at the beginning of an input
! line is considered a prompt.  However, if the variable
! @code{comint-use-prompt-regexp-instead-of-fields} is address@hidden,
! then Comint mode uses a regular expression to recognize prompts.  In
! general, the variable @code{comint-prompt-regexp} specifies the
! regular expression; Shell mode uses the variable
! @code{shell-prompt-pattern} to set up @code{comint-prompt-regexp} in
! the shell buffer.
  
  @node Directory Tracking
  @subsection Directory Tracking
--- 914,921 ----
  @key{SPC} perform history expansion by binding @key{SPC} to the
  command @code{comint-magic-space}.
  
    Shell mode recognizes history references when they follow a prompt.
! @xref{Shell Prompts}, for how Shell mode recognizes prompts.
  
  @node Directory Tracking
  @subsection Directory Tracking
***************
*** 1119,1126 ****
  happens automatically; there is no special password processing.)
  
    When you log in to a different machine, you need to specify the type
! of terminal you're using.  Terminal types @samp{ansi} or @samp{vt100}
! will work on most systems.
  
  @c   If you are talking to a Bourne-compatible
  @c shell, and your system understands the @env{TERMCAP} variable,
--- 1146,1156 ----
  happens automatically; there is no special password processing.)
  
    When you log in to a different machine, you need to specify the type
! of terminal you're using, by setting the @env{TERM} environment
! variable in the environment for the remote login command.  (If you use
! bash, you do that by writing the variable assignment before the remote
! login command, without separating comma.)  Terminal types @samp{ansi}
! or @samp{vt100} will work on most systems.
  
  @c   If you are talking to a Bourne-compatible
  @c shell, and your system understands the @env{TERMCAP} variable,
***************
*** 2077,2083 ****
  paste (yank), and @kbd{C-z} undo.  Standard Emacs commands like
  @kbd{C-x C-c} still work, because @kbd{C-x} and @kbd{C-c} only take
  effect when the mark is active.  However, if you don't want these
! bindings at all, set @code{cua-enable-cua-keys} to nil.
  
  In CUA mode, using @kbd{Shift} together with the movement keys
  activates the region over which they move.  The standard (unshifted)
--- 2107,2113 ----
  paste (yank), and @kbd{C-z} undo.  Standard Emacs commands like
  @kbd{C-x C-c} still work, because @kbd{C-x} and @kbd{C-c} only take
  effect when the mark is active.  However, if you don't want these
! bindings at all, set @code{cua-enable-cua-keys} to @code{nil}.
  
  In CUA mode, using @kbd{Shift} together with the movement keys
  activates the region over which they move.  The standard (unshifted)
============================================================