qmp-shell for GSoC/Outreachy? (Was: Re: Making QEMU easier for management tools and applications)

[John Snow]

I'm forking the subject as I believe Markus wanted to focus on the
machine interface aspect.

I feel that a new human interface is *related* to that goal: the
splitting of, and commitment to, separate human and machine interfaces
powered by a single root schema.

I am a big believer in making QEMU usable directly to human users as I
feel the pipeline of "tinker to deployment" is important for a
successful project, for many reasons:

- QEMU should be easy to pick up and learn.

- Supporting QEMU's use directly as an "end-user" program increases
proficiency in the user population at large, which (can) lead to better
answers and engagement on e.g. Reddit, StackOverflow, IRC

- Evolving deployments from QEMU-only to libvirt+ or above (RHV, oVirt,
kubevirt) should be a smooth and gradual process as additional
complexity is desired.

- Focusing on QEMU's usability allows our project to be consumed easier
by new cloud-focused projects. If they are already familiar with (and
happy with) our project, it is more likely to be used instead of seeking
out alternatives. This is about reducing friction.

So, for those reasons ... even though I feel that a machine-focused API
is our #1 priority as it caters to our existing users, we should also
focus on what it will take to grow mindshare for QEMU's value in the
ecosystem.

Slick interfaces and documentation go a long, long way to doing that.

So: I feel that any new machine-only paradigm or overhaul needs to be
accompanied with some new sugar to help the medicine go down, so-to-speak.

On 2/5/20 8:09 AM, Kevin Wolf wrote:
> Am 28.01.2020 um 11:59 hat Kevin Wolf geschrieben:
>>>> The other part that it needs to solve is how to be available by default
>>>> without specifying anything on the command line. Basically, if I press
>>>> Ctrl-Alt-2, I want to get to a monitor shell. If that shell is
>>>> implemented internally or by an external Python process, I don't mind.
>>>
>>> That is a harder part. (I rarely use Ctrl-Alt-2 actually; I mostly
>>> use HMP on stdin).
>>
>> I don't think it would be that hard, actually.
>>
>> If you have a -qmp-shell option that takes a chardev and defaults to vc,
>> you've solved the part with both stdio and Ctrl-Alt-2. Now all you need
>> to do is launch the Python child process, pass it a pair of pipes for
>> communication and forward everything between the pipes and the chardev.
>>
>> (That's the theory anyway.)
> 
> If someone is interested, I did a quick proof-of-concept hack:
> 
>     https://repo.or.cz/qemu/kevin.git/shortlog/refs/heads/qmp-shell
> 
> It doesn't clean up anything properly (including the qmp-shell processes
> it starts), but it spawns a usable qmp-shell on a user-specified
> character device. stdio seems to work, though without readline
> functionality (I suppose I still have line-buffering somewhere), vc
> doesn't really work at all yet.
> 
> Try it out like this:
> 
>     $ ./qemu-storage-daemon --chardev stdio,id=m --monitor m,mode=qmp-shell
>     monitor_qmp_event: 1
>     Welcome to the QMP low-level shell!
>     Connected to QEMU 4.2.50
> 
>     (QEMU) query-version
>     {"return": {"qemu": {"micro": 50, "major": 4, "minor": 2}, "package": 
> "v4.2.0-1188-gd95a3885a9"}}
>     (QEMU) quit
> 
> (Or use x86_64-softmmu/qemu-system-x86_64, but it's based on the
> refactorings in the storage daemon branch, so why not try both at once?)
> 
> Polishing this to make it mergable would still require substantial work,
> so at the moment I'm not planning to do this. But if someone wants to
> pick it up, feel free (just let us know).
> 
> Hm, in fact... A qmp-shell GSoC project?
> 

That would be great. I worry that we should have a clear vision for the
syntax before we give this project to an intern, though. With a clear
vision and an outline for deliverables, it's an incredibly appropriate
project.

Some things I think we want to define before we start:

1. What are we trying to achieve with a standalone shell?
2. What syntax should it use?

I think those are the hardest parts.


Below, some musings:

- An integrated QMP shell would be a great usability boost to users of
bare QEMU.

- It is undesirable in general to support two interfaces. Feature
disparity is a problem, as is needing to document and test two separate
interfaces. The quality disparity between the two is also an issue.

- Offering HMP via the GTK interface but not QMP is a discoverability
problem. Unfamiliar users might assume that HMP is our flagship
interface. It is not.

- We are unlikely to re-expand HMP to cover everything QMP does; writing
a QMP shell that makes QMP easy to interface with is a better solution
for removing redundancy and complexity.

- I suspect that the target audience for users of naked QEMU are:
  - QEMU developers
  - Upper-layer developers (RHV, oVirt, KubeVirt, libvirt, kata, et al)
researching, testing, and debugging integration.
  - Devops professionals testing, implementing and debugging
    configuration & infrastructure
  - Security/infosec researchers
  - Embedded platform developers
  - Academic researchers



So please correct me if I am off the mark;

Design Goals:
  - The removal of HMP
  - An easy-to-use interface that remains reasonably "close" to the
machine API such that it provides a smooth transition to scripting QEMU.
  - Integration with our GTK interface for discoverability and convenience

Syntax:
  - TBD? Do we agree that the current syntax in qmp-shell is "bad" and
should be replaced? If yes, what should it look like?