[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [PATCH v11 14/15] rdma: introduce MIG_STATE_NONE and ch

From: Michael R. Hines
Subject: Re: [Qemu-devel] [PATCH v11 14/15] rdma: introduce MIG_STATE_NONE and change MIG_STATE_SETUP state transition
Date: Tue, 25 Jun 2013 20:31:02 -0400
User-agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130329 Thunderbird/17.0.5

On 06/25/2013 05:06 PM, Paolo Bonzini wrote:
Il 25/06/2013 22:56, Michael R. Hines ha scritto:
I was wrong - this does require a protocol extension.

This is because the RDMA transfers are asynchronous, and thus
we cannot know in advance that it is safe to unregister the memory
associated with each individual transfer before the transfer actually

While the destination currently uses the protocol to participate in
*registering* the page, the destination does not participate in the
RDMA transfers themselves, only the source does, and thus would
require a new exchange of messages to block and instruct the
destination to unpin the memory.
Yes, that's what I recalled too (really what mst told me :)).  Does it
need to be blocking though?  As long as the pinning is blocking, and
messages are processed in order, the source can proceed immediately
after sending an unpin message.  This assumes of course that the chunk
is not being transmitted, and I am not sure how easy the source can
determine that.


No, they're not processed in order. In fact, not only does the device
write out of order, but also the PCI bus writes out of order.
This was such a problem in fact, that I fixed several bugs as a result
a few weeks ago (v7 of the patch with an in-depth description).

The destination simply cannot assume whatsoever what the ordering
of the writes are - that's really the whole point of using RDMA in the
first place so that the software can get out of the way of the transfer process
to lower the latency of each transfer.

The only option is to send a blocking message to the other side to
request the unpinning (in addition to unpinning on the source first upon
completion of the original transfer).

As you can expect, this would be very expensive and we must ensure
that we have *very* good a-priori information that this memory will
not need to be re-registered anytime in the near future.

- Michael

reply via email to

[Prev in Thread] Current Thread [Next in Thread]