Re: Resource Revocation

[olafBuddenhagen]

Hi,

On Fri, Jul 13, 2007 at 05:16:08PM +0200, Neal H. Walfield wrote:

> Just as we have activities assign preferences to composite activities,
> we can provide a mechanism to allow activities to assign preferences
> to pages.  We can also assign policies to pages. For instance, instead
> of sending a page to backing store, the application may mark a page as
> discardable.  If such a page is chosen for eviction, it should simply
> be freed.  (In this case, a fault needs to be triggered on the next
> access.)  This strategy removes the major source of destructive
> interference: the scheduler no longer waits on the task; the task acts
> beforehand on its own initiative or it reacts to a signal (not a
> request).

I'm not sure I fully understand your proposal; but it may be quite
ironic in some sense...

When I first saw some mention of market-based resource management (in
the context of Hurd/L4), there were very few details, and I thought up
some stuff on my own. My idea was that every task assigns priorities to
all it's physical pages. (In the market model, the maximum rent it's
willing to pay.) Whenever there is memery pressure in the system, the
rent goes up. A central pager then checks all tasks for pages with a
priority below the current rent, and evicts these.

Leaving the market stuff aside, this just means that tasks assign
priorities to their own pages, and a central pager evicts pages based on
these priorities when there is memory pressure.

I was quite sceptical about this design myself, fearing that it might
require too much management for the tasks to assign (and constantly
reassign as demand changes) page priorities, and for the pager to find
the current lowest-priority pages.

Later I learned that your original design was totally different: Tasks
simply get a certain number of physical pages (determined by a
market-based approach or some other rules), and have to do self-paging
within these. The number of available pages changes from time to time,
and the task has to adapt to that. (Where reducing the amount of pages
requires immediate action.)

While self-paging is appealing in a hurdish sense (it allows endless
flexibility changing the implementation), I never really warmed up to
it, because of the problems you described above with explicit
revocation, and suboptimal resource usage due to lack of a global
picture.

Last year you proposed a design where a global pager would evict pages
when the number of pages available to a task is reduced due to memory
pressure; but otherwise tasks have full control over paging by deciding
what happens on a page fault. This looked like a quite promising
compromise: Avoiding the problems of explicit revocation, but still
leaving very much flexibility.

Now you are proposing a design which -- unless I misunderstood your
description -- seems pretty much the same as my original idea. Could you
confirm whether this is indeed the case? And if so, how do you want to
implement it, to avoid too much management overhead?

> This solution is clearly more limiting: the appropriate strategies
> must be identified beforehand.  However, not that many common
> strategies come to mind.  In fact, I suspect that either sending to
> backing store or simply freeing cover the most common scenarios.

Well, I can think only of one other approach: Some applications might
work on large data sets, which can't be discarded because they are not
easy to regenerate, but can be converted to some more compact
representation temporarily at a low cost. This compacting is obviosly
application-specific, so it can't be handled by a generic mechanism.

However, this variant is pretty academic. It seems rather unlikely that
there are actually cases where an application would profit from such
compacting on memory pressure, but on the other hand it would be too
costly to just always do it. And even if such cases indeed exist, I very
much doubt anyone would bother to relly implement this...

What I'm saying is that I agree that swapping or discarding are probably
the only useful actions. I'm not sure how much a global paging mechanism
is limiting possible policies, though...

-antrik-