[Qemu-devel] [POC] colo-proxy in qemu

[Li Zhijian]

Hi, all

We are planning to implement colo-proxy in qemu to cache and compare packets.
This module is one of the important component of COLO project and now it is
still in early stage, so any comments and feedback are warmly welcomed,
thanks in advance.

## Background
COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on demand)
is conducted.
Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking called
colo-proxy.

This document describes the design of the colo-proxy module

## Glossary
  PVM - Primary VM, which provides services to clients.
  SVM - Secondary VM, a hot standby and replication of PVM.
  PN - Primary Node, the host which PVM runs on
  SN - Secondary Node, the host which SVM runs on

## Workflow ##
The following image show the qemu networking packet datapath between
guest's NIC and qemu's backend in colo-proxy.

+---+                                        +---+
|PN |                                        |SN |
+---+--------------------------+             +------------------------------+
|               +-------+      |             |   +-------+                  |
+--------+      |chkpoint<--------[socket]------->chkpoint         +--------+
|PVM     |      +---^---+      |             |   +---+---+         |SVM     |
|        |  +proxy--v--------+ |             |       |             |        |
|        |  |                | |             |       |             |        |
| +---+  |  | +TCP/IP stack+ | |             | +-----v-------proxy | +---+  |
+-|NIC|--+  | |            | | |             | |                 | +-|NIC|--+
| +^-++     | | +--------+ | | |             | | +TCP/IP stack-+ |   +^--+  |
|  | +------> | | compare| | <-[socket]-forward- | +--------+  | |    |     |
|  |        | | +---+----+ | | |             | | | |seq&ack |  | <----+     |
|  |        | +-----|------+ | |             | | | |adjust  |  | |          |
|  |        |       |        | |             | | | +--------+  | |          |
|  +-----------<+>-----copy&forward-[socket]---> +-------------+ |          |
|           +---|---|--------+ |             | +------------^----+          |
|               |   |          |             |              |               |
|               |   |          |             |              x               |
|            +--+---v----+     |             |            +-v---------+     |
| QEMU       |  backend  |     |             | QEMU       |  backend  |     |
+------------+  (tap)    +-----+             +------------+  (tap)    +-----+
             +-----------+                                +-----------+

## Our Idea ##

### Net filter
In current QEMU, a packet is transported between networking backend(tap) and
qemu network adapter(NIC) directly. Backend and adapter is linked by
NetClientState->peer in qemu as following
       +----------------------------------------+
       v                                        |
+NetClientState+   +------->+NetClientState+    |
|info->type=TAP|   |        |info->type=NIC|    |
+--------------+   |        +--------------+    |
|   *peer      +---+        |   *peer      +----+
+--------------+            +--------------+
|name="tap0"   |            |name="e1000"  |
+--------------+            +--------------+
| ...          |            | ...          |
+--------------+            +--------------+

In COLO QEMU, we insert a net filter named colo-proxy between backend and
adapter like below:
typedef struct COLOState {
    NetClientState nc;
    NetClientState *peer;
} COLOState;
   +------->+NetClientState+            +NetClientState+<--------+
   |        |info->type=TAP|            |info->type=NIC|         |
   |        +--------------+            +--------------+         |
+-----------+   *peer      |            |   *peer      +------------+
|  |        +--------------+            +--------------+         |  |
|  |        |name="tap0"   |            |name="e1000"  |         |  |
|  |        +--------------+            +--------------+         |  |
|  |        | ...          |            | ...          |         |  |
|  |        +--------------+            +--------------+         |  |
|  |                                                             |  |
|  |   +-COLOState------------+       +-COLOState------------+   |  |
+--------->+NetClientState+<- - +   +---->+NetClientState+<---------+
   |   |   |info->type=COLO   | |   | |   |info->type=COLO   |   |
   |   |   +--------------+   | |   | |   +--------------+   |   |
   +-------+   *peer      |   | |   | |   |   *peer      +-------+
       |   +--------------+   | |   | |   +--------------+   |
       |   |name="colo1"  |   | |   | |   |name="colo2"  |   |
       |   +--------------+   | |   | |   +--------------+   |
       |                      | |   | |                      |
       |   +--------------+---------+ |   +--------------+   |
       |   |   *peer      |   | |     |   |   *peer      |   |
       |   +--------------+   | +---------+--------------+   |
       +----------------------+       +----------------------+

After we insert colo-proxy filter, all packets will pass by this filter
and more important thing is that we can analysis packet by ourselves.

### QEMU space TCP/IP stack(re-use SLIRP) ###
We need a QEMU space TCP/IP stack to help us to analysis packet. After looking
into QEMU, we found that SLIRP
http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it can
help use to handle the packet written to/read from backend(tap) device which is
just like a link layer(L2) packet.

### packet enqueue and compare ###
Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM and
SVM on Primary QEMU, and then compare the packet payload for each connection.

### Net filter Usage ###
On both Primary/Secondary host, invoke QEMU with the following parameters to 
insert
a net filter(colo-proxy):
  "-netdev tap,id=hn0 -device e1000,netdev=hn0 \
   -netdev colo,id=colo,backend=hn0"