[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] Migration without memory page transfer
From: |
Eric Wheeler |
Subject: |
Re: [Qemu-devel] Migration without memory page transfer |
Date: |
Sat, 5 May 2018 20:30:24 +0000 (UTC) |
User-agent: |
Alpine 2.11 (LRH 23 2013-08-11) |
On Fri, 27 Apr 2018, Peter Xu wrote:
> On Thu, Apr 26, 2018 at 11:33:53PM +0000, Eric Wheeler wrote:
> > Hello all,
>
> Hi, Eric,
>
> >
> > This is my first time inside of the qemu code, so your help is greatly
> > appreciated!
> >
> > I have been experimenting with stop/start of VMs to/from a migration
> > stream that excludes RAM pages and let the RAM pages come from memory file
> > provided by the memory-backend-file called '/dev/shm/mem'.
> >
> > To disable writing of memory pages to the migration stream, I've disabled
> > calls to ram_find_and_save_block in ram_save_iterate() and
> > ram_save_complete() (see patch below). Thus, the migration stream has the
> > "ram" SaveStateEntry section start/ends, but no pages:
> >
> > qemu-system-x86_64 \
> > -object
> > memory-backend-file,prealloc=no,mem-path=/dev/shm/mem,id=ram-node0,host-nodes=0,policy=bind,size=64m,share=on
> > \
> > -numa node,nodeid=0,cpus=0,memdev=ram-node0\
> > -m 64 -vnc 0:0
> >
> > Once the VM is running, I press ctrl-B to get the IPXE prompt and then
> > run 'kernel http://192.168.0.1/foo' to start a network request and watch
> > it in tcpdump.
> >
> > Once the download starts, I save the migration file:
> > migrate "exec:cat > /dev/shm/t"
> >
> > # ls -lh /dev/shm/t
> > -rw-r--r-- 1 root root 321K Apr 26 16:06 /dev/shm/t
> >
> > Now I can kill qemu and boot it again with -incoming:
> >
> > qemu-system-x86_64 \
> > -object
> > memory-backend-file,prealloc=no,mem-path=/dev/shm/mem,id=ram-node0,host-nodes=0,policy=bind,size=64m,share=on
> > \
> > -numa node,nodeid=0,cpus=0,memdev=ram-node0\
> > -m 64 -vnc 0:0 \
> > -incoming 'exec:cat /dev/shm/t'
> >
> > It seems to work. That is, network traffic continues (http from IPXE)
> > which I can see from tcpdump. I can type into the console and it moves
> > the cursor around---but there is nothing on the screen except the blinking
> > text-mode cursor! I can even blindly start a new transfer in ipxe: kernel
> > http://192.168.0.222/foo2 and see it in tcpdump.
> >
> > So what am I missing here? Is the video memory not saved to /dev/shm/mem?
> >
> > Or perhaps it is saved, but VGA isn't initialized to use what is
> > already in /dev/shm/mem? I've tried the cirrus, std, and vmware drivers
> > to see if they behave differently, but the do not seem to.
>
> My wild guess is that we might still need to migrate some RAM besides
> the /dev/shm/mem file. We have at least these ramblocks to migrate:
>
> $ ./x86_64-softmmu/qemu-system-x86_64 -monitor stdio -m 2G
>
> QEMU 2.12.0 monitor - type 'help' for more information
> (qemu) info ramblock
> Block Name PSize Offset Used
> Total
> pc.ram 4 KiB 0x0000000000000000 0x0000000080000000
> 0x0000000080000000
> vga.vram 4 KiB 0x0000000080080000 0x0000000001000000
> 0x0000000001000000
> /address@hidden/acpi/tables 4 KiB 0x0000000081100000
> 0x0000000000020000 0x0000000000200000
> pc.bios 4 KiB 0x0000000080000000 0x0000000000040000
> 0x0000000000040000
> 0000:00:03.0/e1000.rom 4 KiB 0x00000000810c0000 0x0000000000040000
> 0x0000000000040000
> pc.rom 4 KiB 0x0000000080040000 0x0000000000020000
> 0x0000000000020000
> 0000:00:02.0/vga.rom 4 KiB 0x0000000081080000 0x0000000000010000
> 0x0000000000010000
> /address@hidden/table-loader 4 KiB 0x0000000081300000
> 0x0000000000001000 0x0000000000001000
> /address@hidden/acpi/rsdp 4 KiB 0x0000000081340000
> 0x0000000000001000 0x0000000000001000
Yes, that makes sense!
>
> And my understanding is that /dev/shm/mem only corresponds to the
> "pc.ram" entry. I suspect the rest of RAMBlocks will still need to be
> migrated. For example, the VGA ram.
The patch Dr. David Alan Gilbert mentioned is exactly what I was looking
for:
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg02250.html
> Meanwhile, could I ask about where will this be used? Is there
> anything to do with something like a "distributed memory cache" that
> provide memory service across multiple hosts?
I'm mostly interested in quickly restoring without a memory dump, possibly
implementing a "fork()" to quickly clone VMs. Also remote memory would be
neat.
Do you remember OpenMOSIX? I wonder if it would be possible to launch
qemu instances on different nodes such that each instance is a remote NUMA
node. Cache coherency would need to be worked out, and it might require
an OS port to handle new synchronization primitives---but if there was a
way to do it without modifying the OS then you could create really big
single-system-image NUMA servers.
--
Eric Wheeler
>
> Best Regards,
>
> >
> > Thanks for your help!
> >
> > --
> > Eric Wheeler
> >
> >
> > diff --git a/migration/ram.c b/migration/ram.c
> > index 021d583..9f4bfff 100644
> > --- a/migration/ram.c
> > +++ b/migration/ram.c
> > @@ -2267,9 +2267,9 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
> > t0 = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
> > i = 0;
> > while ((ret = qemu_file_rate_limit(f)) == 0) {
> > - int pages;
> > + int pages = 0;
> >
> > - pages = ram_find_and_save_block(rs, false);
> > + if (0) pages = ram_find_and_save_block(rs, false);
> > /* no more pages to sent */
> > if (pages == 0) {
> > done = 1;
> > @@ -2338,9 +2338,9 @@ static int ram_save_complete(QEMUFile *f, void
> > *opaque)
> >
> > /* flush all remaining blocks regardless of rate limiting */
> > while (true) {
> > - int pages;
> > + int pages = 0;
> >
> > - pages = ram_find_and_save_block(rs, !migration_in_colo_state());
> > + if (0) pages = ram_find_and_save_block(rs,
> > !migration_in_colo_state());
> > /* no more blocks to sent */
> > if (pages == 0) {
> > break;
> >
> >
>
> --
> Peter Xu
>
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- Re: [Qemu-devel] Migration without memory page transfer,
Eric Wheeler <=