[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

python asyncio QMP library (AQMP)

From: John Snow
Subject: python asyncio QMP library (AQMP)
Date: Thu, 19 Nov 2020 17:00:56 -0500
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.3.1

I've been doodling a new Asyncio-based QMP library that might fix some of the problems[1][2] with our older, non-async QMP library and provide a better basis for a proper distributable python package for people to write their own toy scripts to control QEMU.

It's a very early prototype, but I think I have the basics down and working now. I figured I would float my extremely early copy of it in case interested parties wanted to take a peek. (Or just to formally announce that "Hey, I'm working on this!")


The design uses two independent coroutines in the bottom half, a reader and a writer. execute() works something like this:

     +---------------- |execute()| <----------+
     |                 +---------+            |
     |                                        |
     v                                        |
+----+----+    +-----------+           +------+-------+
|Mailboxes|    |Event Queue|           |Outbound Queue|
+----+----+    +------+----+           +------+-------+
     |                |                       ^
     v                v                       |
  +--+----------------+---+       +-----------+-----------+
  | Reader Task/Coroutine |       | Writer Task/Coroutine |
  +-----------+-----------+       +-----------+-----------+
              |                               ^
              v                               |
        +-----+------+                  +-----+------+
        |StreamReader|                  |StreamWriter|
        +------------+                  +------------+

Execute deposits a message in the outbound queue; the writer issues it to the server. Execute goes to sleep waiting for mail in its inbox. The reader awakens upon new data available in the stream, the new data is placed in the mailbox, which awakens the caller.

The Reader/Writer are each executed by _task_runner, which ensures that any error that occurs in the BH will create a disconnection Task that will quiesce everything in the bottom half.

Any pending execute actions will be cancelled by the disconnection task; and will raise "DisconnectedError" to the caller.

Upon learning of a disconnection event, the client code responsible for managing the overall state of the connection can call disconnect(), which will raise any exceptions that occurred in the bottom halves that caused the unscheduled disconnect. From there, the client can decide to reconnect or abort, depending.

classes and things of interest:

aqmp.py - Mailbox: Simple data structure pairing an asyncio.Event with a Message. This is done to allow the disconnect task to wake up any waiting clients even if no message arrived. Python's asyncio.Queue does not offer a cancel primitive, so this approximates it.

aqmp.py - AQMP: The core of the loop logic is written here; the public-facing methods are connect(), disconnect(), execute() and execute_obj(), with a "make_execute_msg" class method available for creating Messages that can be executed later.

message.py - Message: This class comprises a JSON Object as read off the wire; putting it in a class like this allows me to write stricter type guarantees and offer _serialize/_deserialize class methods.

models.py - This module offers strictly typed and validated deserializations of core QMP structures that are not (generally) defined by the QAPI schema; i.e., it offers types, de/serialization, and validation for things such as "Greeting", "SuccessResponse", "AsynchronousEvent", etc.

What's left to do?

- Work on Exception naming and hierarchy,
  more helpful/pretty display of errors.
- Expand the event handling system to be more useful (Maybe?)
- Add QGA support
- Re-add socket creation support (Presently, it relies on the socket being created already.)
- Documentation, tests, etc.

Might be nice:

- Look into creating a "Server" variant to be used for testing the library, or even other clients. - Look into synchronous wrappers that offer thread-safe primitives that can be used from traditional, synchronous contexts. (preventing "async creep".)

Long term goal:

- Get it merged in-tree; replace the legacy QMP module over time.

- Create an auto-generated SDK layer based on our QAPI schema that adds a library of type-safe commands, responses, and events that uses this QAPI-agnostic QMP library as its core.

- Add my "JobRunner" patches from earlier this year as a full-fledged jobs API that is designed to make running long-running tasks very trivial from a python console / qmp-shell.


[1] qmp-shell is not capable of displaying events as they arrive asynchronously very well. [2] Our use of non-blocking sockets and socket timeouts in qmp.py relies quite a bit on undefined behavior that thankfully has not broken yet.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]