Terminal emulation problems in GUI (+ solutions)

After looking through "less" source code, I quickly found out that it was reading commands from /dev/tty, hence the problem was obviously that the created PTY is not the controlling terminal of octave and its child processes. However, changing the controlling terminal of the current process is not as easy as it sounds (because the process must not have any controlling terminal, it must be session leader, but not process group leader...). In the end, the only working solution I found is to fork the main process immediately and calls setsid(), which disconnects the process from the controlling terminal. Then in qterminal, you can use ioctl(..., TIOCSCTTY, ...) to use the created PTY as the controlling terminal. The patches in attachment implements a draft solution for this. Note that this is a Linux-oriented solution, I don't know how portable it is to other OS like OS X or Solaris. Note that forking is also not required on Windows. So please don't consider the patches as ready-to-apply.

The downside of this is that one must fork the main process. If anybody has a better idea, that's great. This one is the only one I found. Maybe instead of exiting the parent process after forking, one could use waitpid to wait for the child to terminate, with SIGINT/SIGTERM catching to terminate the child process. That would allow to keep some control on the octave-gui process when starting it from a terminal.

This one was actually quite easy. The SIGWINCH signal is never sent to the PTY, because the code to do this has been commented out in TerminalModel.cpp (due to code simplification, I suppose). The solution is to re-add the necessary code to propagate the terminal size change to the PTY on widget resize event. This is also implemented in the attached patches.

From:	Michael Goffioul
Subject:	Terminal emulation problems in GUI (+ solutions)
Date:	Sun, 1 Jul 2012 23:45:55 +0100