[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH RFC v3 00/27] COarse-grain LOck-stepping(COLO) V
From: |
Dr. David Alan Gilbert |
Subject: |
Re: [Qemu-devel] [PATCH RFC v3 00/27] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service |
Date: |
Tue, 24 Feb 2015 20:13:47 +0000 |
User-agent: |
Mutt/1.5.23 (2014-03-12) |
* zhanghailiang (address@hidden) wrote:
> This is the 3th version of COLO, it is only COLO frame part, include: VM
> checkpoint,
> failover, proxy API, block replication API, not include block replication.
> The block part has been sent by wencongyang:
> '[RFC PATCH 00/14] Block replication for continuous checkpoints'
>
> You can get the integrated qemu colo patches from github:
> https://github.com/coloft/qemu/commits/colo-v1.0
>
> Compared with the previous version, we have realized all parts of COLO frame,
> and it is works now.
>
> The main change since last version is, we use colo proxy mode instead of
> colo agent, they are all used for network packets compare, but proxy is more
> efficient, it is based on netfilter.
> Another modification is we implement new block replication scheme,
> you can get more info from wencongyang's block patch series
>
> If you don't know about COLO, please refer to below link for detailed
> information.
>
> The idea is presented in Xen summit 2012, and 2013,
> and academia paper in SOCC 2013. It's also presented in KVM forum in 2013:
> http://www.linux-kvm.org/wiki/images/1/1d/Kvm-forum-2013-COLO.pdf
>
> Previous posted RFC proposal:
> http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg05567.html
> http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg04459.html
>
> The below is the detail about test COLO, you can also get this info
> from http://wiki.qemu.org/Features/COLO.
> * Hardware requirements
> There is at least one directly connected nic to forward the network requests
> from client to secondary VM. The directly connected nic must not be used by
> any other purpose.
>
> * Network link topology
> =================================normal ======================================
> +--------+
> |client |
> master +----+---+ slave
> -------------------------+ | + -------------------------+
> PVM | + | |
> +-------+ +----[eth0]-----[switch]-----[eth0]---------+ |
> |guest | +---+-+ | | +---+-+ |
> | [tap0]--+ br0 | | | | br0 | |
> | | +-----+ [eth1]-----[forward]----[eth1]--+ +-----+ SVM |
> +-------+ | | | +-------+|
> | | | +-----+ | guest ||
> [eth2]---[checkpoint]---[eth2] +--+br1 |-[tap0] ||
> | | +-----+ | ||
> | | +-------+|
> -------------------------+ +--------------------------+
> e.g.
> master:
> br0: 192.168.0.33
> eth1: 192.168.1.33
> eth2: 192.168.2.33
>
> slave:
> br0: 192.168.0.88
> br1: no ip address
> eth1: 192.168.1.88
> eth2: 192.168.2.88
> (Actually, you can also use eth0 as checkpoint channel)
> Note: in normal, SVM will always be linked to br1 like above until
> failover.
>
> * Test environment prepare:
> 1. Set Up the Bridge and network environment
> You must setup you network environment like above picture,
> In master, setup a bridge br0, using command brctl, like:
> # ifconfig eth0 down
> # ifconfig eth0 0.0.0.0
> # brctl addbr br0
> # brctl addif br0 eth0
> # ifconfig br0 192.168.0.33 netmask 255.255.255.0
> # ifconfig eth0 up
> In slave, setup two bridge br0, br1, commands are same with above,
> please note that br1 is linked to eth1(the forward nic).
>
> 2.Qemu-ifup
> We need a script to bring up the TAP interface.
> You can find this info from http://en.wikibooks.org/wiki/QEMU/Networking.
> Master:
> address@hidden cat /etc/qemu-ifup
> #!/bin/sh
> switch=br0
> if [ -n "$1" ]; then
> ip link set $1 up
> brctl addif ${switch} $1
> fi
> Slave:
> address@hidden # cat /etc/qemu-ifup
> #!/bin/sh
> switch=br1 #in primary, switch is br0. in secondary switch is br1
> if [ -n "$1" ]; then
> ip link set $1 up
> brctl addif ${switch} $1
> fi
>
> 3. Prepare host kernel
> colo-proxy kernel module need cooperate with linux kernel.
> You should put a kernel patch 'colo-patch-for-kernel.patch'
> (It's based on linux kernel-3.19) which you can get from
> https://github.com/gao-feng/colo-proxy.git
> and then compile kernel and intall the new kernel.
>
> 4. Proxy module
> proxy module is used for network packets compare, you can also get the lastest
> version from: https://github.com/gao-feng/colo-proxy.git.
> You can compile and install it by using command 'make' && 'make install'.
>
> 5. Modified iptables
> We have add a new rule to iptables command, so please get the patch from
> https://github.com/gao-feng/colo-proxy/blob/master/COLO-library_for_iptables-1.4.21.patch
> It is based on version 1.4.21.
I'm getting closer but I don't think I'm getting packets from the secondary
to the primary yet; it looks like the primary is holding onto it's packets
until the end of the 10second checkpoint.
Still debugging that but I'd take any tips.
> 2. startup qemu
> master:
> # qemu-system-x86_64 -enable-kvm -netdev
> tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1
> -device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive
> driver=quorum,read-pattern=first,children.0.file.filename=suse11_3.img,children.0.driver=raw,children.1.file.driver=nbd+colo,children.1.file.host=192.168.2.88,children.1.file.port=8889,children.1.file.export=colo1,children.1.driver=raw,if=virtio
> -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor
> stdio -S
> slave:
> # qemu-system-x86_64 -enable-kvm -netdev
> tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1
> -device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive
> driver=blkcolo,export=colo1,backing.file.filename=suse11_3.img,backing.driver=raw,if=virtio
> -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor
> stdio -incoming tcp:0:8888
>
> 3. On Secondary VM's QEMU monitor, run
> (qemu) nbd_server_start 192.168.2.88:8889
>
> 4.on Primary VM's QEMU monitor, run following command:
> (qemu) migrate_set_capability colo on
> (qemu) migrate tcp:192.168.2.88:8888
>
> 5. done
> You will see two runing VMs, whenever you make changes to PVM, SVM
> will be synced to PVM's state.
>
> 6. failover test:
> You can kill SVM (PVM) and run 'colo_lost_heartbeat' in PVM's (SVM's) monitor
> at the same time, then PVM (SVM) will failover and client will not feel this
> change.
I've not got that to work yet; so far if I kill the PVM the SVM quits
shortly after. (I've tried both 'q' in the PVMs monitor and also kill -9).
Dave
> It is still a framework, far away from commercial use,
> so any comments/feedbacks are warmly welcomed ;)
>
> PS:
> We (huawei) have cooperated with fujitsu on COLO work,
> and we work mainly on COLO frame and fujitsu will focus on COLO block.
>
> TODO list:
> 1) Optimize the process of checkpoint, shorten the time-consuming
> 2) Add more debug/stat info
> 3) Strengthen failover
> 4) The capability of continuous FT
>
> v3:
> - use proxy instead of colo agent to compare network packets
> - add block replication
> - Optimize failover disposal
> - handle shutdown
>
> v2:
> - use QEMUSizedBuffer/QEMUFile as COLO buffer
> - colo support is enabled by default
> - add nic replication support
> - addressed comments from Eric Blake and Dr. David Alan Gilbert
>
> v1:
> - implement the frame of colo
>
>
> zhanghailiang (27):
> configure: Add parameter for configure to enable/disable COLO support
> migration: Introduce capability 'colo' to migration
> COLO: migrate colo related info to slave
> migration: Integrate COLO checkpoint process into migration
> migration: Integrate COLO checkpoint process into loadvm
> migration: Don't send vm description in COLO mode
> COLO: Implement colo checkpoint protocol
> COLO: Add a new RunState RUN_STATE_COLO
> QEMUSizedBuffer: Introduce two help functions for qsb
> COLO: Save VM state to slave when do checkpoint
> COLO RAM: Load PVM's dirty page into SVM's RAM cache temporarily
> COLO VMstate: Load VM state into qsb before restore it
> COLO RAM: Flush cached RAM into SVM's memory
> COLO failover: Introduce a new command to trigger a failover
> COLO failover: Implement COLO master/slave failover work
> COLO failover: Don't do failover during loading VM's state
> COLO: Add new command parameter 'colo_nicname' 'colo_script' for net
> COLO NIC: Init/remove colo nic devices when add/cleanup tap devices
> COLO NIC: Implement colo nic device interface configure()
> COLO NIC : Implement colo nic init/destroy function
> COLO NIC: Some init work related with proxy module
> COLO: Do checkpoint according to the result of net packets comparing
> COLO: Improve checkpoint efficiency by do additional periodic
> checkpoint
> COLO NIC: Implement NIC checkpoint and failover
> COLO: Disable qdev hotplug when VM is in COLO mode
> COLO: Implement shutdown checkpoint
> COLO: Add block replication into colo process
>
> arch_init.c | 196 ++++++++-
> configure | 14 +
> hmp-commands.hx | 15 +
> hmp.c | 7 +
> hmp.h | 1 +
> include/exec/cpu-all.h | 1 +
> include/migration/migration-colo.h | 57 +++
> include/migration/migration-failover.h | 22 +
> include/migration/migration.h | 14 +
> include/migration/qemu-file.h | 3 +-
> include/net/colo-nic.h | 25 ++
> include/net/net.h | 4 +
> include/sysemu/sysemu.h | 3 +
> migration/Makefile.objs | 2 +
> migration/colo-comm.c | 81 ++++
> migration/colo-failover.c | 48 +++
> migration/colo.c | 743
> +++++++++++++++++++++++++++++++++
> migration/migration.c | 74 +++-
> migration/qemu-file-buf.c | 57 +++
> net/Makefile.objs | 1 +
> net/colo-nic.c | 438 +++++++++++++++++++
> net/tap.c | 45 +-
> qapi-schema.json | 27 +-
> qemu-options.hx | 10 +-
> qmp-commands.hx | 19 +
> savevm.c | 10 +-
> scripts/colo-proxy-script.sh | 88 ++++
> stubs/Makefile.objs | 1 +
> stubs/migration-colo.c | 49 +++
> vl.c | 36 +-
> 30 files changed, 2047 insertions(+), 44 deletions(-)
> create mode 100644 include/migration/migration-colo.h
> create mode 100644 include/migration/migration-failover.h
> create mode 100644 include/net/colo-nic.h
> create mode 100644 migration/colo-comm.c
> create mode 100644 migration/colo-failover.c
> create mode 100644 migration/colo.c
> create mode 100644 net/colo-nic.c
> create mode 100755 scripts/colo-proxy-script.sh
> create mode 100644 stubs/migration-colo.c
>
> --
> 1.7.12.4
>
>
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK
- [Qemu-devel] [PATCH RFC v3 05/27] migration: Integrate COLO checkpoint process into loadvm, (continued)
- [Qemu-devel] [PATCH RFC v3 05/27] migration: Integrate COLO checkpoint process into loadvm, zhanghailiang, 2015/02/11
- [Qemu-devel] [PATCH RFC v3 11/27] COLO RAM: Load PVM's dirty page into SVM's RAM cache temporarily, zhanghailiang, 2015/02/11
- [Qemu-devel] [PATCH RFC v3 14/27] COLO failover: Introduce a new command to trigger a failover, zhanghailiang, 2015/02/11
- [Qemu-devel] [PATCH RFC v3 06/27] migration: Don't send vm description in COLO mode, zhanghailiang, 2015/02/11
- [Qemu-devel] [PATCH RFC v3 23/27] COLO: Improve checkpoint efficiency by do additional periodic checkpoint, zhanghailiang, 2015/02/11
- [Qemu-devel] [PATCH RFC v3 03/27] COLO: migrate colo related info to slave, zhanghailiang, 2015/02/11
- Re: [Qemu-devel] [PATCH RFC v3 00/27] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service, Dr. David Alan Gilbert, 2015/02/16
- Re: [Qemu-devel] [PATCH RFC v3 00/27] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service, Dr. David Alan Gilbert, 2015/02/24
- Re: [Qemu-devel] [PATCH RFC v3 00/27] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service,
Dr. David Alan Gilbert <=