Hi,
we were reading through this quickly today, and these are some of the questions
that
we think can came up when reading this. Answers to some of these questions we
think
we have figured out, but I think it's important to put this information into the
documentation.
I will quote the file in its entirety, and insert some questions inline.
Device Specification for Inter-VM shared memory device
------------------------------------------------------
The Inter-VM shared memory device is designed to share a region of memory to
userspace in multiple virtual guests.
What does "to userspace" mean in this context? The userspace of the host, or
the userspace in the guest?
What about "The Inter-VM shared memory device is designed to share a memory region
(created on the host via the POSIX shared memory API) between multiple QEMU processes
running different guests. In order for all guests to be able to pick up the shared memory
area, it is modeled by QEMU as a PCI device exposing said memory to the guest as a PCI
BAR."
Whether in those guests the memory region is used in kernel space or userspace,
or there is even any meaning for those terms is guest-dependent I would think
(I think of an OSv here, where the application and kernel execute at the same
privilege level and in the same address space).
The memory region does not belong to any
guest, but is a POSIX memory object on the host.
Ok that's clear.
One thing I would ask is, but I don't know if it makes sense to mention here,
is who creates this memory object on the host?
I understand in some cases it's the contributed server (what you provide in contrib/), in some
cases it's the "user" of this device who has to write some server code for that, but
is it true that also the qemu process itself can create this memory object on its own, without
any external process needed? Is this the use case for host<->guest only?
Optionally, the device may
support sending interrupts to other guests sharing the same memory region.
This opens a lot of questions here which are partly answered later (If I
understand correctly, not only interrupts are involved, but a complete
communication protocol involving registers in BAR0), but what about staying a
bit general here, like
"Optionally, the device may also provide a communication mechanism between
guests sharing the same memory region. More details about that in the section
'OPTIONAL ivshmem guest to guest communication protocol'.
Thinking out loud, I wonder if this communication mechanism should be part of
this device in QEMU, or it should be provided at another layer..
The Inter-VM PCI device
-----------------------
*BARs*
The device supports three BARs. BAR0 is a 1 Kbyte MMIO region to support
registers. BAR1 is used for MSI-X when it is enabled in the device. BAR2 is
used to map the shared memory object from the host. The size of BAR2 is
specified when the guest is started and must be a power of 2 in size.
Are BAR0 and BAR1 optional? That's what I would think by reading the whole, but
I'm still not sure.
Am I forced to map BAR0 and BAR1 anyway? I don't think so, but..
If so, can we separate the explanation into the base shared memory feature, and
a separate section which explains the OPTIONAL communication mechanism, and the
OPTIONAL MSI-X BAR?
For example, say that I am a potential ivshmem user (which I am), and I am
interested in the shared memory but I want to use my own communication
mechanism and protocol between guests, can we make it so that I don't have to
wonder whether some of the info I read applies or not?
The solution to that I think is to put all the OPTIONAL parts into separate
sections.
*Registers*
Ok, so this should I think go into one such OPTIONAL sections.
The device currently supports 4 registers of 32-bits each. Registers
are used for synchronization between guests sharing the same memory object when
interrupts are supported (this requires using the shared memory server).
So use of BAR0 goes together with interrupts, and goes together with the shared
memory server (is it the one contributed in contrib/?)
The server assigns each VM an ID number and sends this ID number to the QEMU
process when the guest starts.
enum ivshmem_registers {
IntrMask = 0,
IntrStatus = 4,
IVPosition = 8,
Doorbell = 12
};
The first two registers are the interrupt mask and status registers. Mask and
status are only used with pin-based interrupts. They are unused with MSI
interrupts.
Status Register: The status register is set to 1 when an interrupt occurs.
Mask Register: The mask register is bitwise ANDed with the interrupt status
and the result will raise an interrupt if it is non-zero. However, since 1 is
the only value the status will be set to, it is only the first bit of the mask
that has any effect. Therefore interrupts can be masked by setting the first
bit to 0 and unmasked by setting the first bit to 1.
IVPosition Register: The IVPosition register is read-only and reports the
guest's ID number. The guest IDs are non-negative integers. When using the
server, since the server is a separate process, the VM ID will only be set when
the device is ready (shared memory is received from the server and accessible
via
the device). If the device is not ready, the IVPosition will return -1.
Applications should ensure that they have a valid VM ID before accessing the
shared memory.
So the guest ID number is 32bits, but actually the doorbell is 16-bit, can we be
more explicit about this? So does it follow that the maximum number of guests
is 65536?
Doorbell Register: To interrupt another guest, a guest must write to the
Doorbell register. The doorbell register is 32-bits, logically divided into
two 16-bit fields. The high 16-bits are the guest ID to interrupt and the low
16-bits are the interrupt vector to trigger. The semantics of the value
written to the doorbell depends on whether the device is using MSI or a regular
pin-based interrupt. In short, MSI uses vectors while regular interrupts set
the
status register.
Regular Interrupts
If regular interrupts are used (due to either a guest not supporting MSI or the
user specifying not to use them on startup) then the value written to the lower
16-bits of the Doorbell register results is arbitrary and will trigger an
interrupt in the destination guest.
Message Signalled Interrupts
A ivshmem device may support multiple MSI vectors. If so, the lower 16-bits
written to the Doorbell register must be between 0 and the maximum number of
vectors the guest supports. The lower 16 bits written to the doorbell is the
MSI vector that will be raised in the destination guest. The number of MSI
vectors is configurable but it is set when the VM is started.
The important thing to remember with MSI is that it is only a signal, no status
is set (since MSI interrupts are not shared). All information other than the
interrupt itself should be communicated via the shared memory region. Devices
supporting multiple MSI vectors can use different vectors to indicate different
events have occurred. The semantics of interrupt vectors are left to the
user's discretion.
Maybe an example of a full exchange would be useful to explain the use of these
registers, making the protocol used for communication clear; or does this only
provide mechanisms that can be used by someone else to implement a protocol?
IVSHMEM host services
---------------------
This part is optional (see *Usage in the Guest* section below)
Ok this section is optional, but its role is not that clear to me.
So are there exactly 3 ways this can be used:
1) shared memory only, PCI BAR2
2) full device including registers in BAR0 but no MSI
3) full device including registers in BAR0 and MSI support in BAR1
?
To handle notifications between users of a ivshmem device, a ivshmem server has
been added. This server is responsible for creating the shared memory and
creating a set of eventfds for each users of the shared memory.
Ok this is the first time eventfds are mentioned, after we spoke about
interrupts in the other section before..
It behaves as a
proxy between the different ivshmem clients (QEMU): giving the shared memory fd
to each client,
telling each client which /dev/name to shm_open?
allocating eventfds to new clients and broadcasting to all
clients when a client disappears.
What about VM Ids, are they also decided and shared by the server?
Apart from the current ivshmem implementation in QEMU, a ivshmem client can be
written for debug, for development purposes, or to implement notifications
between host and guests.
Usage in the Guest
------------------
The guest should map BAR0 to access the registers (an array of 32-bit ints
allows simple writing) and map BAR2 to access the shared memory region itself.
Ok, but can I avoid mapping BAR0 if I don't use the registers?
The size of the shared memory region is specified when the guest (or shared
memory server) is started. A guest may map the whole shared memory region or
only part of it.
So what does it mean here, I can choose to start the optional server
contributed in contrib/
with a shared memory size parameter determining the size of the actual shared
memory region,
and then the guest has the option to map only part of that?
Or can also the guest (or better, the QEMU process running the guest) create
the shared memory region by itself?
Which parameters control these behaviours?
Btw I would expect there to be a separate section with all the QEMU command
line configuration parameters and their effect on behavior of this device. Also
for the contributed code in contrib/, especially for the server, we need
documentation about the command line parameters, env variables, whatever can be
configured and which effect they have on this.
ivshmem provides an optional notification mechanism through eventfds handled by
QEMU that will trigger interrupts in guests. This mechanism is enabled when
using a ivshmem-server which must be started prior to VMs and which serves as a
proxy for exchanging eventfds.
Here also, a simple description of such a sequence of exchanges would be
welcome, I would not mind some ASCII art as well.
It is your choice how to use the ivshmem device.
Good :)
- the simple way, you don't need anything else than what is already in QEMU.
If the server becomes part of the QEMU package, then this sentence is a bit
unclear right? This was probably written at the time the server was not
contributed to QEMU, right?
You can map the shared memory in guest, then use it in userland as you see
fit
In userland.. ? Can I create the shared memory by just running a qemu process
with some parameters? Does this mean I now share memory between guest and host?
If I run multiple guest providing the same device name, can I make them use the
same shared memory without the need of any server?
(memnic for example works this way http://dpdk.org/browse/memnic),
I'll check that out..
- the more advanced way, basically, if you want an event mechanism between the
VMs using your ivshmem device. In this case, then you will most likely want
to
write a kernel driver that will handle interrupts.
Ok.
Let me ask you this, what about virtio?
Can I take this shared memory implementation, and then run virtio on top of
that, which already has primitives for communication?
I understand this would restricts me to 1 vs 1 communication, while with the
optional server in contrib/ I would have any to any communication available.
But what about the 1 to 1 guest-to-guest communication, is in this case in
theory possible to put virtio on top of ivshmem and use that to make the two
guests communicate?
This is just a list of questions that we came up with, but anybody please weigh
in with your additional questions, comments, feedback. Especially I would like
to know if the idea to have a virtio guest to guest communication is possible
and realistic, maybe with minimal extension of virtio, or if I am being insane.
Thank you,
Claudio