Re: Call for GSoC and Outreachy project ideas for summer 2023

[Stefan Hajnoczi]

On Sun, 5 Feb 2023 at 03:15, Eugenio Perez Martin <eperezma@redhat.com> wrote:
>
> On Fri, Jan 27, 2023 at 4:18 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >
> > Dear QEMU, KVM, and rust-vmm communities,
> > QEMU will apply for Google Summer of Code 2023
> > (https://summerofcode.withgoogle.com/) and has been accepted into
> > Outreachy May 2023 (https://www.outreachy.org/). You can now
> > submit internship project ideas for QEMU, KVM, and rust-vmm!
> >
> > Please reply to this email by February 6th with your project ideas.
> >
> > If you have experience contributing to QEMU, KVM, or rust-vmm you can
> > be a mentor. Mentors support interns as they work on their project. It's a
> > great way to give back and you get to work with people who are just
> > starting out in open source.
> >
> > Good project ideas are suitable for remote work by a competent
> > programmer who is not yet familiar with the codebase. In
> > addition, they are:
> > - Well-defined - the scope is clear
> > - Self-contained - there are few dependencies
> > - Uncontroversial - they are acceptable to the community
> > - Incremental - they produce deliverables along the way
> >
> > Feel free to post ideas even if you are unable to mentor the project.
> > It doesn't hurt to share the idea!
> >
> > I will review project ideas and keep you up-to-date on QEMU's
> > acceptance into GSoC.
> >
> > Internship program details:
> > - Paid, remote work open source internships
> > - GSoC projects are 175 or 350 hours, Outreachy projects are 30
> > hrs/week for 12 weeks
> > - Mentored by volunteers from QEMU, KVM, and rust-vmm
> > - Mentors typically spend at least 5 hours per week during the coding period
> >
> > For more background on QEMU internships, check out this video:
> > https://www.youtube.com/watch?v=xNVCX7YMUL8
> >
> > Please let me know if you have any questions!
> >
> > Stefan
> >
>
> Appending the different ideas here.

Hi Eugenio,
Thanks for sharing your project ideas. I have added some questions
below before we add them to the ideas list wiki page.

> VIRTIO_F_IN_ORDER feature support for virtio devices
> ===
> This was already a project the last year, and it produced a few series
> upstream but was never merged. The previous series are totally useful
> to start with, so it's not starting from scratch with them [1]:

Has Zhi Guo stopped working on the patches?

What is the state of the existing patches? What work remains to be done?

>
> Summary
> ---
> Implement VIRTIO_F_IN_ORDER in QEMU and Linux (vhost and virtio drivers)
>
> The VIRTIO specification defines a feature bit (VIRTIO_F_IN_ORDER)
> that devices and drivers can negotiate when the device uses
> descriptors in the same order in which they were made available by the
> driver.
>
> This feature can simplify device and driver implementations and
> increase performance. For example, when VIRTIO_F_IN_ORDER is
> negotiated, it may be easier to create a batch of buffers and reduce
> DMA transactions when the device uses a batch of buffers.
>
> Currently the devices and drivers available in Linux and QEMU do not
> support this feature. An implementation is available in DPDK for the
> virtio-net driver.
>
> Goals
> ---
> Implement VIRTIO_F_IN_ORDER for a single device/driver in QEMU and
> Linux (virtio-net or virtio-serial are good starting points).
> Generalize your approach to the common virtio core code for split and
> packed virtqueue layouts.
> If time allows, support for the packed virtqueue layout can be added
> to Linux vhost, QEMU's libvhost-user, and/or QEMU's virtio qtest code.
>
> Shadow Virtqueue missing virtio features
> ===
>
> Summary
> ---
> Some VirtIO devices like virtio-net have a control virtqueue (CVQ)
> that allows them to dynamically change a number of parameters like MAC
> or number of active queues. Changes to passthrough devices using vDPA
> using CVQ are inherently hard to track if CVQ is handled as
> passthrough data queues, because qemu is not aware of that
> communication for performance reasons. In this situation, qemu is not
> able to migrate these devices, as it is not able to tell the actual
> state of the device.
>
> Shadow Virtqueue (SVQ) allows qemu to offer an emulated queue to the
> device, effectively forwarding the descriptors of that communication,
> tracking the device internal state, and being able to migrate it to a
> new destination qemu.
>
> To restore that state in the destination, SVQ is able to send these
> messages as regular CVQ commands. The code to understand and parse
> virtio-net CVQ commands is already in qemu as part of its emulated
> device, but the code to send the some of the new state is not, and
> some features are missing. There is already code to restore basic
> commands like mac or multiqueue, and it is easy to use it as a
> template.
>
> Goals
> ---
> To implement missing virtio-net commands sending:
> * VIRTIO_NET_CTRL_RX family, to control receive mode.
> * VIRTIO_NET_CTRL_GUEST_OFFLOADS
> * VIRTIO_NET_CTRL_VLAN family
> * VIRTIO_NET_CTRL_MQ_HASH config
> * VIRTIO_NET_CTRL_MQ_RSS config

Is there enough work here for a 350 hour or 175 hour GSoC project?

The project description mentions "there is already code to restore
basic commands like mac and multiqueue", please include a link.

> Shadow Virtqueue performance optimization
> ===
> Summary
> ---
> To perform a virtual machine live migration with an external device to
> qemu, qemu needs a way to know which memory the device modifies so it
> is able to resend it. Otherwise the guest would resume with invalid /
> outdated memory in the destination.
>
> This is especially hard with passthrough hardware devices, as
> transports like PCI imposes a few security and performance challenges.
> As a method to overcome this for virtio devices, qemu can offer an
> emulated virtqueue to the device, called Shadow Virtqueue (SVQ),
> instead of allowing the device to communicate directly with the guest.
> SVQ will then forward the writes to the guest, being the effective
> writer in the guest memory and knowing when a portion of it needs to
> be resent.
>
> As this is effectively breaking the passthrough and it adds extra
> steps in the communication, this comes with a performance penalty in
> some forms: Context switches, more memory reads and writes increasing
> cache pressure, etc.
>
> At this moment the SVQ code is not optimized. It cannot forward
> buffers in parallel using multiqueue and multithread, and it does not
> use posted interrupts to notify the device skipping the host kernel
> context switch (doorbells).
>
> The SVQ code requires minimal modifications for the multithreading,
> and these are examples of multithreaded devices already like
> virtio-blk which can be used as a template-alike. Regarding the posted
> interrupts, DPDK is able to use them so that code can also be used as
> a template.
>
> Goals
> ---
> * Measure the latest SVQ performance compared to non-SVQ.

Which benchmark workload and which benchmarking tool do you recommend?
Someone unfamiliar with QEMU and SVQ needs more details in order to
know what to do.

> * Add multithreading to SVQ, extracting the code from the Big QEMU Lock (BQL).

What do you have in mind? Allowing individual virtqueues to be
assigned to IOThreads? Or processing all virtqueues in a single
IOThread (like virtio-blk and virtio-scsi do today)?

> * Add posted thread capabilities to QEMU, following the model of DPDK to it.

What is this about? I thought KVM uses posted interrupts when
available, so what needs to be done here? Please also include a link
to the relevant DPDK code.

>
> Thanks!
>
> [1] 
> https://wiki.qemu.org/Google_Summer_of_Code_2022#VIRTIO_F_IN_ORDER_support_for_virtio_devices
>