[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [PATCH v11 14/15] rdma: introduce MIG_STATE_NONE and ch

From: Paolo Bonzini
Subject: Re: [Qemu-devel] [PATCH v11 14/15] rdma: introduce MIG_STATE_NONE and change MIG_STATE_SETUP state transition
Date: Wed, 26 Jun 2013 08:37:09 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130514 Thunderbird/17.0.6

Il 26/06/2013 02:31, Michael R. Hines ha scritto:
> On 06/25/2013 05:06 PM, Paolo Bonzini wrote:
>> Il 25/06/2013 22:56, Michael R. Hines ha scritto:
>>> I was wrong - this does require a protocol extension.
>>> This is because the RDMA transfers are asynchronous, and thus
>>> we cannot know in advance that it is safe to unregister the memory
>>> associated with each individual transfer before the transfer actually
>>> completes.
>>> While the destination currently uses the protocol to participate in
>>> *registering* the page, the destination does not participate in the
>>> RDMA transfers themselves, only the source does, and thus would
>>> require a new exchange of messages to block and instruct the
>>> destination to unpin the memory.
>> Yes, that's what I recalled too (really what mst told me :)).  Does it
>> need to be blocking though?  As long as the pinning is blocking, and
>> messages are processed in order, the source can proceed immediately
>> after sending an unpin message.  This assumes of course that the chunk
>> is not being transmitted, and I am not sure how easy the source can
>> determine that.
> No, they're not processed in order. In fact, not only does the device
> write out of order, but also the PCI bus writes out of order.
> This was such a problem in fact, that I fixed several bugs as a result
> a few weeks ago (v7 of the patch with an in-depth description).
> The destination simply cannot assume whatsoever what the ordering
> of the writes are - that's really the whole point of using RDMA in the
> first place so that the software can get out of the way of the transfer
> process to lower the latency of each transfer.

The memory is processed out of order, but what about the messages?
Those must be in order.

Note that I wrote above "This assumes of course that the chunk is not
being transmitted".  Can the source know when an asynchronous transfer
finished, and delay the unpinning until that time?


> The only option is to send a blocking message to the other side to
> request the unpinning (in addition to unpinning on the source first upon
> completion of the original transfer).
> As you can expect, this would be very expensive and we must ensure
> that we have *very* good a-priori information that this memory will
> not need to be re-registered anytime in the near future.
> - Michael

reply via email to

[Prev in Thread] Current Thread [Next in Thread]