[Qemu-discuss] How do I get unbuffered console output between qemu and pexpect?

[Joachim Durchholz]

Hi all,

I am trying to drive qemu from pexpect ("Python expect").

For some reason, characters do not arrive at expect's matcher until the 
line end has been produced.
The symptom is that I can match completed lines just fine, but the 
script does not see prompts like "login: ", "# ", or "$ " until after it 
sends a line of input (which will cause the login process to echo the 
input and terminate the line). Obviously, having to match the line that 
comes *before* any prompt is going to be a PITA, so I am wondering how 
to deal with this.

I have no idea under what circumstances qemu itself would buffer stuff; 
does anybody know?
If qemu isn't the culprit, I'd want to check whether it's the pty pair, 
or maybe the Python runtime or pexpect, if somebody happens to be around 
who knows.

I'm appending the observations I already made, and attaching the actual 
Python script I'm using for testing.
What am I doing wrong?

Regards,
Jo

====================

Observations (Python snippets are somewhat handwavy, refer to the 
attached simple.py to conduct experiments):

=====
Experiment 1 (baseline):

On the shell,

  qemu-system-x86_64 -snapshot FreeBSD-12.0-RELEASE-amd64.qcow2

will give me a standard interactive window. The boot process is normal 
if lengthy (around 60 seconds if simply waiting), and ends with this:

--
FreeBSD/amd64 (freebsd) (ttyv0)

login:
--

When I type "root", I get a welcome message and the "address@hidden:~ # " 
prompt.

=====

Experiment 2 (match the login prompt):

From pexpect, I do:

pexpect.spawn ('qemu-system-x86_64 -snapshot -monitor none -display 
curses -chardev stdio,id=char0 FreeBSD-12.0-RELEASE-amd64.qcow2', 
timeout=90, maxread=1)
expect('login: ')
child.sendline('root')

This times out.

BTW The maxread=1 option tells pexpect to read a character at a time; I 
was hoping this would avoid buffering at the pexpect side but it didn't 
help (but it may still be necessary so I'm leaving it in).

=====

Experiment 3 (match the line *before* the login prompt):

pexpect.spawn ('qemu-system-x86_64 -snapshot -monitor none -display 
curses -chardev stdio,id=char0 FreeBSD-12.0-RELEASE-amd64.qcow2', 
timeout=90, maxread=1)
expect(re.escape('FreeBSD/amd64 (freebsd)'))
time.sleep(1) # MUCH longer than actually needed, just to be safe
sendline('root')

THIS WORKS! The code continues to the point after the sendline() call 
(my real script has more stuff after this which did happen with this, 
and didn't happen in the previous experiment).

BTW expect() expects a regular expression, that's why the string to 
match is wrapped in an re.escape() call.

=====

Experiment 4 (use -nographic, because many SO posts use that):

pexpect.spawn ('qemu-system-x86_64 -snapshot -nographic 
FreeBSD-12.0-RELEASE-amd64.qcow2', timeout=90, maxread=1)

This does not seem to generate any output.
I guess that's the classic "works as designed" case: -nographic hooks 
the vm's serial port to stdio, not the keyboard/VGA combination, and 
FreeBSD isn't told to talk to the serial port.
(Is this analysis correct?)

simple.py
Description: Text Data