Re: gnulib openpty module on Solaris

[Bruno Haible]

Hi,

Emanuele Giaquinta wrote in private mail (better CC bug-gnulib):
> I am a developer of rxvt-unicode, an rxvt fork, and I am writing you
> concerning the gnulib openpty module, to which you added support for
> Solaris back in 2010. I recently became aware of an issue with pty
> allocation on Solaris which affects rxvt-unicode and might also affect
> the gnulib openpty module. The problem is that loading the "ptem"
> STREAMS module has the side-effect of making the tty controlling if the
> process does not have one, as documented in
> 
> http://docs.oracle.com/cd/E26502_01/html/E35856/termsub15-44781.html
> 
> "In its open routine, ptem sends an M_SETOPTS message upstream
> requesting allocation of a controlling TTY."
> 
> This behaviour seems to date back to SVR4. If the ptem/ldterm modules
> are loaded in the child process after calling setsid, which seems to be
> the common pattern, this is not an issue since the result is that the
> tty becomes controlling as desired. Instead, if the modules are loaded
> in the parent process after opening the pty, which is the case in
> rxvt-unicode and in the gnulib openpty module, then, if the parent has
> no controlling terminal, the following happens: 1) the tty becomes the
> controlling terminal of the parent process; 2) the child process
> inherits the controlling terminal, loses it after calling setsid, and is
> unable to make the tty controlling because the tty is already the
> controlling terminal of another process.
> 
> One solution is loading the ptem/ldterm modules in the child process
> (which would translate to loading them in the login_tty module).
> However, this method has the drawback that the parent cannot perform
> ioctl on the pty until the child has loaded the modules. Another
> solution is to open a new pty without O_NOCTTY (so that it becomes
> controlling) before loading ptem and close it afterwards.
> 
> What do you think?
> 
> Best,
> 
> Emanuele

I'm not sure I understand.

You are talking about a parent and a child process, but the openpty function
is defined [1] without reference to parent and child processes and without
mention of controlling ttys.

Should we conclude that openpty() is not suitable in the context of the
"common pattern" with parent and child processes and controlling ttys,
on Solaris?

Or should we conclude that it's the "common pattern" which is wrong?

Or should we conclude that openpty should take two additional boolean
arguments, that specify whether to O_NOCTTY for the master fd and for
the slave fd, respectively?

Bruno

[1] 
http://www.gnu.org/software/libc/manual/html_node/Pseudo_002dTerminal-Pairs.html