Re: [Qemu-devel] [RFC PATCH RDMA support v2: 5/6] connection-setup code between client/server

[Michael R. Hines]

On 02/18/2013 03:24 AM, Orit Wasserman wrote:
> Hi Michael, The guest device state is quite small (~100K probably 
> less) especially when compared to the guest memory and we already are 
> pinning the guest memory for RDMA any way I was actually wondering 
> about the memory pinning, do we pin all guest memory pages as 
> migration starts or on demand? 

The patch supports both methods. There's a function called 
"rdma_server_prepare()" which optionally pins all the memory in advanced.

We prefer on-demand pinning, of course, because we want to preserve the 
ability to do ballooning and the occasional madvise() calls.

The patch defaults to pinning everything right now for performance 
evaluation.......... later I'll make sure to switch that off when we've 
coalesced on a solution for the actual RDMA transfer itself.

> For the guest memory pages, sending the pages directly without 
> QemuFile (that does buffering) is better, I would suggest implementing 
> an QemuRDMAFile for this. It will have a new API for the memory pages 
> (zero copy) so instead of using qemu_put_buffer we will call 
> qemu_rdma_buffer or it can reimplement qemu_put_buffer (you need to 
> add offset). As for device state which is sent in the last phase and 
> is small you can modify the current implementation. (well Paolo sent 
> patches that are changing this but I think buffering is still an 
> option) The current migration implementation copies the device state 
> into a buffer and than send the data from the buffer 
> (QemuBufferedFile). You only need to pin this buffer, and RDMA it 
> after all the device state was written into it. Regards, Orit 
I like it .......... can you help me understand - how much different is 
this design than the "QEMUFileOps" design that Paulo suggested?

Or it basically the same? ....reimplementing 
qemu_put_buffer/get_buffer() for RDMA purposes......

- Michael