Re: [Qemu-devel] [RFC PATCH RDMA support v2: 5/6] connection-setup code between client/server

[Michael S. Tsirkin]

On Thu, Feb 14, 2013 at 02:29:09PM -0500, Michael R. Hines wrote:
> Orit (and anthony if you're not busy),
> 
> I forgot to respond to this very important comment:
> 
> On 02/13/2013 03:46 AM, Orit Wasserman wrote:
> 
>     Are you still using the tcp for transferring device state? If so you can
>     call the tcp functions from the migration rdma code as a first step but I
>     would prefer it to use RDMA too.
> 
> 
> This is the crux of the problem of using RDMA for migration: Currently all of
> the QEMU migration control logic and device state goes through the the 
> QEMUFile
> implementation. RDMA, however is by nature a zero-copy solution and is
> incompatible with QEMUFile.

RDMA might be overkill but you could reuse the same connection,
using send instructions.

> Using RDMA for transferring device state is not recommended: Setuping an RDMA
> requires registering the memory locations on both sides with the RDMA 
> hardware.
> This is not ideal because this would require pinning the memory holding the
> device state and then issuing the RDMA transfer for *each* type of device -
> which would require changing the control path of every type of migrated device
> in QEMU.
> 
> Currently the Patch we submitted bypasses QEMUFile. It does just issues the
> RDMA transfer for the memory that was dirtied and then continues along with 
> the
> rest of the migration call path normally.
> 
> In an ideal world, we would prefer a hyrbid approach, something like:
> 
> Begin Migration Iteration Round:
> 1. stop VCPU
> 2. start iterative pass over memory
> 3. send control signals (if any) / device state to QEMUFile
> 4. When a dirty memory page is found, do:
>      a) Instruct the QEMUFile to block
>      b) Issue the RDMA transfer
>      c) Instruct the QEMUFile to unblock
> 5. resume VCPU
> 
> This would require a "smarter" QEMUFile implementation that understands when 
> to
> block and for how long.
> 
> Comments?
> 
> - Michael