Re: Resource Management on General-Purpose Systems

[Jonathan S. Shapiro]

On Thu, 2007-07-12 at 16:41 +0200, Neal H. Walfield wrote:
> The problem is not how to more intelligently choose the page to evict,
> but how to intelligently choose the page to free.  That is, I agree
> with Tanenbaum that if an active program's working set does not fit in
> core and must be swapped, there is a problem.  However, memory
> management is still interesting: when there is memory pressure, the
> scheduler should concentrate on enabling adaptation.
> 
> Reactions?

As a counter-example to his assertion, we need only consider commodity
movie-editing software that ships with every home movie camera on the
market. As a second counter-example, consider Digital Audio Workstation
software (probably less widely used, but not insignificant). As it
happens, I am an occasional user of some of these types of video
software, and a frequent user of DAW software.

In the video case, it is not at all uncommon for the working files to
reach 8 to 24 gigabytes, because smart users do raw captures to avoid
compromising quality at the source. This obviously doesn't fit in memory
and needs to be streamed -- which is why these applications have very
poor interactive immediacy.

In the audio case, let's do some math. For CD-quality audio:

  24-bit samples (3 bytes each)
  96 Khz per channel
  4 minute recording

the raw sample size is 70 megabytes **per channel**. It's surprisingly
easy to have 16 channels active when effects processing is taken into
account. That's about 1.1 gigabytes, which is more than most users have.
The DAW software engines use various techniques to mitigate this.

Recall that because of hardware design stupidity, a 32-bit commodity PC
has an effective max physical memory size of 3GB. Upper memory gets
stolen by PCI, primarily video.

My point is: these applications are going through all kinds of nasty
hair for two reasons:

  1. 32-bit virtual address limits. That is slowly getting cured.
  2. Bad paging interfaces -- these apps would be a lot simpler
     if they could just use mmap().

So: if you believe that we will remain crippled by 32-bit address limits
forever, then Andy is right. But if you believe that 64-bit virtual
addresses are an inevitable transition (as I do), then I think Andy's
assertion is questionable.

shap