Re: need help diagnosing readline issue under cygwin rxvt

[Michael Goffioul]

On Thu, Feb 14, 2008 at 2:33 PM, Matt Hortman <address@hidden> wrote:
> Hello,
>
> I'm a longtime matlab user and I've recently been giving Octave a shot
> and I have to say that I like what I see.  My adviser was also
> impressed and might be recommending it to his students...
>
> I have Octave 3.0.0 for Windows installed (via the octave-forge
> installer) on Windows XP SP2.
>
> Running Octave from the installed shortcut works great.  Inside the
> octave session the command line editing works as advertised (up/down
> arrows, tab, etc).
>
> However, I want to run Octave inside of cygwin rxvt (from a zsh
> prompt) but when I do this, the up/down arrow and tab keys do not
> behave correctly.  Up/down arrows simply move up and down on the
> screen and the tab key inserts an actual tab...
>
> In fact, running with no command line options, I get the startup
> messages but it doesn't print a prompt (although I can type commands).
>  I have to specify the "-i" option to get the prompt to display.
> Specifying "--line-editing" as well does not change anything.
>
> I have the latest version of cygwin installed (as of yesterday).
> I do NOT have the cygwin octave package installed.
> The up/down arrows and tab work correctly at the zsh prompt.
> The pager (less) is also not working inside rxvt but I don't really
> care about that.
>
> I'm not sure what the next step is for diagnosing this problem.
> Octave appears to think it is in batch mode when started inside rxvt.
> Is this something strange with the stdin/out provided by zsh/rxvt or
> readline misbehaving?  Is octave detecting that it is running inside
> the console and taking actions based on that?

The problem is that under Win32 there's no concept of pseudo-terminal
and any Win32 terminal emulation actually operates over pipes, and not
over what the Win32 C runtime library would consider as a terminal.
As a consequence, any "isatty" call will return false in such terminal
emulation; in the end, octave thinks it runs in batch mode. You can
force interactive mode by using flags "-i --line-editing", then you'll see
the octave prompt appearing.

The readline library provided with the octave package for Windows is
compiled with termcap support(*); so I would expect that octave would
work correctly in any terminal emulation (once the interactive mode
has been forced). And it actually works OK in a Java-based terminal
emulation I used. However, when I try it in mingw rxvt or cygwin xterm,
it does not work (I know mingw rxvt terminal emulation is poor, but
I expected it would work within cygwin xterm): the cursor keys have
the expected behavior (history search), but the cursor is also moved
up and down, which messes up the terminal display.

I don't really the time to look at that problem, so if anybody feels looking
into it, please go ahead.

Michael.

(*): as there's no SIGWINCH signal under Win32, terminal resize is not
handled nicely; to make it work OK, you should define LINES and
COLUMNS environment variables to the actual terminal size before starting
octave; and do not try to resize the terminal, octave won't be notified of the
size change.