bug#8162: 23.3; Crash with invalid default-process-coding-system value

[Kenichi Handa]

In article <E1Pv6fH-0003zX-AI@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> 
writes:

> > Date: Thu, 03 Mar 2011 19:20:15 +0900
> > From: YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>
> > Cc: 
> > 
> > Steps to reproduce:
> > 
> >   1. $ emacs -Q
> >   2. M-x shell RET
> >   3. M-: (setq default-process-coding-system '(dummy . utf-8-unix)) RET
> >   4. C-u M-x shell RET RET
> >   5. RET (in the *shell*<2> buffer)
> > 
> > Result:
> > 
> >   Crash with SIGSEGV at process.c:5737

> Don't you see an error message regarding an invalid coding-system,
> before you type the last RET (which causes the crash)?

> set-process-coding-system signals an error when it sees the `dummy'
> part, but comint.el does nothing about that.  It should kill the shell
> buffer it just created, IMO.

But, at least, a process associated with the shell buffer
must be killed when an error occurs in Fstart_process.
Actually, that function has this code:

  /* If an error occurs and we can't start the process, we want to
     remove it from the process list.  This means that each error
     check in create_process doesn't need to call remove_process
     itself; it's all taken care of here.  */
  record_unwind_protect (start_process_unwind, proc);

and the "Invalid coding system" error is caused in
create_process called near the end of Fstart_process.

The reason why start_process_unwind doesn't kill the process
is that create_process sets XPROCESS (process)->pid to -1
too late; i.e. after set_process_coding_system (which
signals an error) is called.

The attached patch (against emacs-23 branch) will fix this
bug.  Perhaps "XPROCESS (process)->pid = -1" should be done
much earlier because an error may be signaled before, but,
I'm not sure.  deactivate_process may not handle such
situation well.

By the way, as Emacs 23.2 also had this bug, strictly
speaking, the fix is not for regression.  Should I install
it for emacs-23 or for emacs-24?

---
Kenichi Handa
handa@m17n.org


2011-03-03  Kenichi Handa  <handa@m17n.org>

        * process.c (create_process): Call setup_process_coding_systems
        after the pid of the process is set to -1.

=== modified file 'src/process.c'
--- src/process.c       2011-01-02 23:50:46 +0000
+++ src/process.c       2011-03-03 12:30:28 +0000
@@ -1999,7 +1999,6 @@
 
   XPROCESS (process)->pty_flag = pty_flag;
   XPROCESS (process)->status = Qrun;
-  setup_process_coding_systems (process);
 
   /* Delay interrupts until we have a chance to store
      the new fork's pid in its process structure */
@@ -2046,6 +2045,10 @@
      processes to get their return values scrambled.  */
   XPROCESS (process)->pid = -1;
 
+  /* This must be called after the above line because it may signal an
+     error. */
+  setup_process_coding_systems (process);
+
   BLOCK_INPUT;
 
   {