Re: gnulib openpty module on Solaris

[Emanuele Giaquinta]

On Fri, Apr 03, 2015 at 01:23:08AM +0200, Bruno Haible wrote:
> Hi,
> 
> Emanuele Giaquinta wrote in private mail (better CC bug-gnulib):
> > I am a developer of rxvt-unicode, an rxvt fork, and I am writing you
> > concerning the gnulib openpty module, to which you added support for
> > Solaris back in 2010. I recently became aware of an issue with pty
> > allocation on Solaris which affects rxvt-unicode and might also affect
> > the gnulib openpty module. The problem is that loading the "ptem"
> > STREAMS module has the side-effect of making the tty controlling if the
> > process does not have one, as documented in
> > 
> > http://docs.oracle.com/cd/E26502_01/html/E35856/termsub15-44781.html
> > 
> > "In its open routine, ptem sends an M_SETOPTS message upstream
> > requesting allocation of a controlling TTY."
> > 
> > This behaviour seems to date back to SVR4. If the ptem/ldterm modules
> > are loaded in the child process after calling setsid, which seems to be
> > the common pattern, this is not an issue since the result is that the
> > tty becomes controlling as desired. Instead, if the modules are loaded
> > in the parent process after opening the pty, which is the case in
> > rxvt-unicode and in the gnulib openpty module, then, if the parent has
> > no controlling terminal, the following happens: 1) the tty becomes the
> > controlling terminal of the parent process; 2) the child process
> > inherits the controlling terminal, loses it after calling setsid, and is
> > unable to make the tty controlling because the tty is already the
> > controlling terminal of another process.
> > 
> > One solution is loading the ptem/ldterm modules in the child process
> > (which would translate to loading them in the login_tty module).
> > However, this method has the drawback that the parent cannot perform
> > ioctl on the pty until the child has loaded the modules. Another
> > solution is to open a new pty without O_NOCTTY (so that it becomes
> > controlling) before loading ptem and close it afterwards.
> > 
> > What do you think?
> > 
> > Best,
> > 
> > Emanuele
> 
> I'm not sure I understand.
> 
> You are talking about a parent and a child process, but the openpty function
> is defined [1] without reference to parent and child processes and without
> mention of controlling ttys.

See

http://git.savannah.gnu.org/cgit/gnulib.git/tree/lib/forkpty.c

http://git.savannah.gnu.org/cgit/gnulib.git/tree/lib/login_tty.c

I am talking about this kind of usage. If a process with no controlling
terminal uses the forkpty module (or equivalent code with the openpty
module) on Solaris, then there is a bug, namely that login_tty or
equivalent fails to make the tty controlling in the child process (I
verified that gnulib has this bug too).

Note also that the login_tty module does not verify that the process has
a controlling terminal after line 63 (it should check that opening
/dev/tty succeeds), so it returns 0 regardless.

> Should we conclude that openpty() is not suitable in the context of the
> "common pattern" with parent and child processes and controlling ttys,
> on Solaris?

I don't think so. However, for the pattern to work the openpty() caller
must have a controlling terminal.

> Or should we conclude that it's the "common pattern" which is wrong?

Why would it be wrong?

> Or should we conclude that openpty should take two additional boolean
> arguments, that specify whether to O_NOCTTY for the master fd and for
> the slave fd, respectively?

This would not solve the problem, because loading ptem makes the tty
controlling regardless of whether the master/slave was opened with
O_NOCTTY.

Emanuele