|
From: | Michael R. Hines |
Subject: | Re: [Qemu-devel] [RFC PATCH RDMA support v5: 03/12] comprehensive protocol documentation |
Date: | Fri, 12 Apr 2013 09:47:08 -0400 |
User-agent: | Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 |
On 04/12/2013 06:48 AM, Michael S. Tsirkin wrote:
On Thu, Apr 11, 2013 at 04:33:03PM -0400, Michael R. Hines wrote:On 04/11/2013 03:15 PM, Michael S. Tsirkin wrote:On Thu, Apr 11, 2013 at 01:49:34PM -0400, Michael R. Hines wrote:On 04/11/2013 10:56 AM, Michael S. Tsirkin wrote:On Thu, Apr 11, 2013 at 04:50:21PM +0200, Paolo Bonzini wrote:Il 11/04/2013 16:37, Michael S. Tsirkin ha scritto:pg1 -> pin -> req -> res -> rdma -> done pg2 -> pin -> req -> res -> rdma -> done pg3 -> pin -> req -> res -> rdma -> done pg4 -> pin -> req -> res -> rdma -> done pg4 -> pin -> req -> res -> rdma -> done It's like a assembly line see? So while software does the registration roundtrip dance, hardware is processing rdma requests for previous chunks.Does this only affects the implementation, or also the wire protocol?It affects the wire protocol.I *do* believe chunked registration was a *very* useful request by the community, and I want to thank you for convincing me to implement it. But, with all due respect, pipelining is a "solution looking for a problem".The problem is bad performance, isn't it? If it wasn't we'd use chunk based all the time.Improving the protocol does not help the behavior of any well-known workloads, because it is based on the idea the the memory footprint of a VM would *rapidly* shrink and contract up and down during the steady-state iteration rounds while the migration is taking place.What gave you that idea? Not at all. It is based on the idea of doing control actions in parallel with data transfers, so that control latency does not degrade performance.Again, this parallelization is trying to solve a problem that doesn't exist. As I've described before, I re-executed the worst-case memory stress hog tests with RDMA *after* the bulk-phase round completes and determined that RDMA throughput remains unaffected because most of the memory was already registered in advance.This simply does not happen - workloads don't behave that way - they either grow really big or they grow really small and they settle that way for a reasonable amount of time before the load on the application changes at a future point in time. - MichaelWhat is the bottleneck for chunk-based? Can you tell me that? Find out, and you will maybe see pipelining will help. Basically to me, when you describe the protocol in detail the problems become apparent. I think you worry too much about what the guest does, what APIs are exposed from the migration core and the specifics of the workload. Build a sane protocol for data transfers and layer the workload on top.What is the point in enhancing a protocol to solve a problem will never be manifested? We're trying to overlap two *completely different use cases* that are completely unrelated: 1. Static overcommit 2. Dynamic, fine-grained overcommit (at small time scales... seconds or minutes) #1 Happens all the time. Cram a bunch of virtual machines with fixed workloads and fixed writable working sets into the same place, and you're good to go. #2 never happens. Ever. It just doesn't happen, and the enhancements you've described are trying to protect against #2, when we should really be focused on #1. It is not standard practice for a workload to expect high overcommit performance in the *middle* of a relocation and nobody in the industry that I have met over the years has expressed any such desire to do so.Depends on who you talk to I guess. Almost everyone overcommits to some level. They might not know it. It depends on the amount of overcommit. You pin all (at least non zero) memory eventually, breaking memory overcommit completely. If I overcommit by 4kilobytes do you expect performance to go completely down? It does not make sense.Workloads just don't behave that way. Dynamic registration does an excellent job at overcommitment for #1 because most of the registrations are done at the very beginning and can be further optimized to cause little or no performance loss by simply issuing the registrations before the migration ever begins.How does it? You pin all VM's memory eventually. You said your tests have the OOM killer triggering.
That's because of cgroups memory limitations. Not the protocol. Infiband was never designed to work with cgroups - that's a kernel problem, not a QEMU problem or a protocol problem. Why do we have to worry about that exactly?
Performance for #2 even with dynamic registration is excellent and I am not experiencing any problems associated with it.Well previously you said the reverse. You keep vaguely speaking about performance. We care about these metrics: 1. total migration time: measured by: time ssh dest qemu -incoming &;echo migrate > monitor time 2. min allowed downtime that lets migration converge 3. average host CPU utilization during migration, on source and destination 4. max real memory used by qemu Can you fill this table for TCP, and two protocol versions? If dynamic works as well as static, this is a good reason to drop the static one. As the next step, fix the dynamic to unregister memory (this is required for _GIFT anyway). When you do this it is possible that pipelining is required.
First, yes, I'm happy to fill out the table - let me address Paolo's last requested changes (including the COMPRESS fix) Second, there are not two protocol versions. That's incorrect. There's only one protocol which can operate in different ways as any protocol can operate in different ways. It has different command types, not all of which need to be used by the protocol at the same time. Second, as I've explained, I strongly, strongly disagree with unregistering memory for all of the aforementioned reasons - workloads do not operate in such a manner that they can tolerate memory to be pulled out from underneath them at such fine-grained time scales in the *middle* of a relocation and I will not commit to writing a solution for a problem that doesn't exist. If you can prove (through some kind of anaylsis) that workloads would benefit from this kind of fine-grained memory overcommit by having cgroups swap out memory to disk underneath them without their permission, I would happily reconsider my position. - Michael
So, we're discussing a non-issue. - MichaelThere are two issues. 1. You have two protocols already and this does not make sense in version 1 of the patch. You said dynamic is slow so I pointed out ways to improve it. Now you says it's as fast as static? so drop static then. At no point does it make sense to have management commands to play with low level protocol details.Overcommit has two
[Prev in Thread] | Current Thread | [Next in Thread] |