[Qemu-devel] [RFC] Host vs Guest memory allocation

[Richard Henderson]

The Problem:

CONFIG_USER_ONLY kinda sorta tries to manage the distinction between
qemu memory and guest memory. This can be seen in the PAGE_RESERVED
frobbing and qemu_malloc etc. However, it doesn't handle random malloc
calls eg from libc itself or other libraries in use.

Possible solutions:

There are several possible solutions as I see it, each depending on the
pairing of host and guest address space characteristics:

(0) Do nothing.  That is, don't even pretend to record host memory and
    validate guest access that may or may not overlap.  To my mind this
    is in fact an improvement over the kinda-sorta solution we have now.

(1) Enable softmmu for userland. This is of course the highest overhead,
    but will work for all combinations.

(2) Pre-allocate the entire guest address space in the host.  With
    Linux mmap w/ MAP_NORESERVE or Windows VirtualAlloc w/ MEM_RESERVE
    and a possibly reduced guest address space this is doesn't seem so bad.

    Reducing the guest's virtual address space via a command-line switch
    seems like a fairly good idea to help this method apply when emulating
    a 32-bit guest on a 32-bit host.  With few exceptions, it seems like
    most guests could behave well under these conditions, as it isn't
    entirely dis-similar to RLIMIT_AS.

    (2a) Arrange the qemu binary itself so that this pre-allocation can
         happen in low-memory for qemu.

         For 64-bit hosts, this can include compiling the application such
         that it's placed in memory above 4GB.  Of course, this movement
         must be balanced against the performance hit that might be incurred
         beyond the setting of GUEST_BASE to a non-zero value, e.g:

         On x86-64, one could compile the application with -fpie to arrange
         for the compiler to access its code and data with rip-relative
         addressing instead of absolute addressing.  As far as I remember,
         this is a minor code size penalty, but doesn't affect the runtime
         performance at all.

         On sparc64, which has no pc-relative addressing mode, moving the
         application outside the low 4G would be a significant penalty to
         its code size and performance.  However, there's very little 
         overhead in a non-zero GUEST_BASE, since we store GUEST_BASE in
         a register during TCG-generated code and emit accesses to guest
         memory with a reg+reg addressing mode.

         On ia64 and alpha, the application is *always* placed in high
         memory somewhere, so nothing at all needs to change.

         On ppc64, code is compiled pic by default, and so there is no
         penalty for moving the base of the application out of low memory.

    (2b) On x86, use segmentation to allow the guest address space anywhere,
         and at the same time check that the guest memory accesses are in
         range for the guest.

         Clearly this can be done on Linux via modify_ldt(2), and on
         the BSDs via i386_set_ldt.  I can't find an MSDN documented
         method for this for Windows, although there seems to be plenty
         of security advisories that suggest that it's possible.  ;-P

(3) Provide an implementation of malloc for use in qemu which keeps the
    PAGE_RESERVED entries up-to-date all the time.

    Glibc, and probably most other libc implementations, are set up to
    allow malloc to be overridden by the main executable.  There are
    plenty of good malloc implementations out there which we could modify
    to record the bits we need.  For 64-bit hosts, it could likely also
    arrange for the allocated memory to not conflict with the guest address
    space most of the time.

(4) Implement all of 1-3 and select among them at startup.  For instance,
    begin by trying #2, and if the reservation of the guest address space
    fails then fall back to either #1 or #3 based on some command-line switch.

    This is of course my favorite option.  Hopefully a lot of the if-deffery
    concerning CONFIG_USER_ONLY and CONFIG_SOFTMMU can be eliminated in the
    process.

Comments?


r~