Re: Issue with VDUSE (QSD vduse-blk export) and vhost-vdpa

[Stefano Garzarella]

On Wed, Oct 26, 2022 at 12:17 PM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> On Wed, Oct 26, 2022 at 05:39:23PM +0800, Yongji Xie wrote:
> >Hi Stefano,
> >
> >On Wed, Oct 26, 2022 at 5:12 PM Stefano Garzarella <sgarzare@redhat.com> 
> >wrote:
> >>
> >> Hi Xie,
> >> I was testing libblkio [1] with QSD vduse-blk export and had some
> >> issues.
> >>
> >> In a nutshell, QSD prints me the following messages when using
> >> vhost-vdpa to access the device:
> >>
> >>   Failed to get vq[0] iova mapping
> >>   Failed to update vring for vq[0]
> >>
> >> This happens only with vhost-vdpa, using virtio-vdpa instead the device
> >> works fine.
> >> I'm using Linux v6.0 and QEMU master (commit
> >> 214a8da23651f2472b296b3293e619fd58d9e212).
> >>
> >> I haven't had much time to investigate, I hope to do it next week, but
> >> maybe it's much faster for you.
> >>
> >> I saw that ioctl(VDUSE_IOTLB_GET_FD) in libvduse.c returns -1 (EPERM),
> >> so IIUC in the kernel vduse_dev_broken() was called, and the device is
> >> in a broken state.
> >>
> >>
> >> We will use libblkio in QEMU [2] to access vDPA devices via vhost-vdpa.
> >> But I'm doing these tests without QEMU for now, using an example inside
> >> the libblkio repo:
> >>
> >> # Build libblkio and examples
> >>     # Fedora/CentOS/RHEL
> >>     dnf install -y git meson rust cargo python3-docutils rustfmt
> >>     # Debian/Ubuntu
> >>     apt-get install -y git meson rustc cargo python3-docutils
> >>
> >>     git clone https://gitlab.com/libblkio/libblkio.git
> >>
> >>     cd libblkio
> >>     git checkout v1.1.0
> >>
> >>     meson setup build
> >>     meson compile -C build
> >>
> >>
> >> # On terminal 1
> >>     modprobe vduse
> >>     modprobe vhost-vdpa
> >>
> >>     qemu-img create -f qcow2 -o preallocation=full /path/to/test.qcow2 1g
> >>
> >>     qemu-storage-daemon \
> >>       --blockdev 
> >> file,filename=/path/to/test.qcow2,cache.direct=on,aio=native,node-name=file
> >>  \
> >>       --blockdev qcow2,file=file,node-name=qcow2 \
> >>       --object iothread,id=iothread0 \
> >>       --export 
> >> vduse-blk,id=vduse0,name=vduse0,num-queues=1,node-name=qcow2,writable=on,iothread=iothread0
> >>
> >>
> >> # On terminal 2
> >>     vdpa dev add name vduse0 mgmtdev vduse
> >>
> >>     cd libblkio/build
> >>
> >>     # blkio-bench executes
> >>     ./examples/blkio-bench virtio-blk-vhost-vdpa \
> >>       path=/dev/vhost-vdpa-0 --runtime=5 --readwrite=randread
> >>
> >>     # after this step, QSD (running on terminal 1) prints the following 
> >> messages:
> >>       Failed to get vq[0] iova mapping
> >>       Failed to update vring for vq[0]
> >>
> >> I don't know if I'm doing something wrong or in libblkio we have some
> >> issue, but using vdpa-sim-blk works correctly, so maybe there is
> >> something in vduse that is missing.
> >>
> >> Any help or suggestion is welcome :-)
> >>
> >
> >I'd like to know whether bio-bench uses the shared memory
> >(tmpfs/hugetlbfs) as the vdpa memory region. This is what VDUSE needs.
>
> Okay, so IIUC every memory regions should have an associated fd.
>
> The buffers in libblkio are already allocated in this way, but it is not
> true for the virtqueue memory, I'll change it and test.

I just changed the virtqueue memory allocation as you suggested, and it worked!

MR here: https://gitlab.com/libblkio/libblkio/-/merge_requests/141

Thanks for the help,
Stefano