Re: VMMs and Over-commitment

[olafBuddenhagen]

Hi,

On Thu, Aug 09, 2007 at 05:42:54PM +0200, Neal H. Walfield wrote:
> At Thu, 2 Aug 2007 21:05:27 +0200, <address@hidden> wrote:

> > Yet, I'm not sure the option of decentralized management should be
> > completely discarded. Think of virtual machines: In this case, it's
> > totally out of question to have a central resource manager that
> > controls all allocations within the guest systems. The guests want
> > to keep total control over how things are managed inside their VMs.
> 
> Totally out of the question is quite exaggerated.  VMWare ESX Server
> implements a number of techniques to facilitate the over-commitment of
> memory including paging, ballooning and content sharing [1].
> 
> ESX Server introduces another layer of paging.  The decisions are, of
> course, less informed (as the VMM has less data) and can result in
> double paging.  To this end, ESX uses a randomized strategy.
> 
> To avoid paging, ESX uses so-called ballooning.  A driver is loaded
> into a guest OS that interacts with the hypervisor.  The hypervisor
> signals the balloon to change its size according to memory pressure
> and its current allocation strategy.  When the balloon grows, the OS's
> own memory management strategy comes into effect.  The memory
> allocated to the balloon is then returned to the hypervisor.
> 
> Finally, ESX implements context based sharing.  Using idle CPU time,
> it scans memory looking for identical pages.

Well, the point is: While we can see these approaches to faciliate some
interaction between host and guest to improve resource utilization (and
these are quite interesting indeed; I wasn't aware this field is so
advanced), they don't really change anything about the hierarchical
resource management: The host does *not* have any control or even
knowledge about the resource usage of individual processes within the
guest. Management of resources inside the guest is completely up to the
guest system. The host only manages resources for the VM as a whole.

[changing order of sections]
> > Of course, the ability to relatively easily create custom
> > sub-environments in the Hurd is not quite comparable to full-blown
> > VMs; yet it might be an interesting direction to look in...
> 
> I'm not sure what you mean here.  I think you mean: providing support
> in the OS for isolation may be an interesting direction.  If so, I
> whole-heartedly agree.

Well, first of all, to avoid confusion, I should clarify that I probably
mean something different by "isolation" than you do.

To make this clear, let's think about EROS (and similar systems) on the
one Hand, and the Hurd on the other. EROS provides isolation between
processes by allowing strict control over communication channels.
However, the system mechanisms are all fixed -- in fact, Shapiro even
argues that customization is often dangerous.

The Hurd is quite the opposite: The original design doesn't care much
about communication channels. (In one mail on the ancient hurd-folks
list, Michael/Thomas indeed stated that although using capabilities
under the hood, the Hurd is *not* supposed to be any more secure than
UNIX.) The Hurd is all about customization, however: It allows running
processes or process groups, even whole sessions, in a custom
environment, that differs in some regards from the default environment
provided by the system. That gives users and programs some independance.

The latter is *also* a kind of isolation IMHO; but it's certainly
confusing to use the same term for very different things. In lack of a
better term, I'll refer to it as "independance" for now.

VMs provide both isolation of communication channels *and* independance.

> > The current popularity of VMs shows that in many cases, the benefits
> > of the strong isolation outweigh the disadvantages of missing
> > centralization and integration.
> 
> I disagree.  The commodity use of VMs to isolate programs demonstrates
> a deficiency in commodity operating systems.  That many researchers
> are focusing their efforts exclusively on VMs rather than considering
> how to improve OS interfaces is another, separate problem.  This has
> been identified by others [3].

Well, to put it bluntly: Systems with improved interfaces failed in the
market, while VMs are very successsful...

I don't mean to say it's hopeless and we should resign. What I mean to
say is that it can't be all attributed to wrong research focus. There
must be much more tangible reasons, and it's sure worthwile to
understand these.

I can think of two very decisive reasons. One is that with VMs, instead
of switching to something completely new, you only need a specific bit
of additional infrastructure (the VM itself), and can otherwise stick
with all the mature and familiar software they have been using before.

The other reason is that interfaces allowing for more isolation and/or
independance in a single system are very abstract; it's hard to
understand how to use them, or why one wants it. VMs on the other hand
are a concept that is very easy to grasp: They just provide a sealed
room within the system, with total isolation and independance, allowing
to run another system (or another instance of the same system) in there.

It's also wrong to attribute all interest in VMs to isolation alone. The
independance property seems just as important.

Returning to the previous question, traditional systems where everything
is intermingled on the on hand, and VMs providing total isolation and
independance on the other hand, are the two extremes. In practice
however you often want something in between. So on one side we get
integrated systems that try to provide some more isolation (EROS) or
independance (Hurd); and on the other side we get VMs that try to
provide some controlled communication channels and some interaction
between host and guest, for better reource usage and efficiency; i.e.
reducing isolation and independance in a controlled fashion.

What I'm getting at is that considering how much easier to grasp VMs are
as a concept, it might be beneficial to think of the Hurd less as a
single system optionally allowing for some independance, but more as a
framework that allows for interacting but mostly independant subsystems
(mini-VMs).

In this spirit, really hierarchical resoure management doesn't seem
totally without merit.

-antrik-