Re: [Qemu-devel] [RFC PATCH V10 1/7] colo-compare: introduce colo compare initialization

[Zhang Chen]

On 08/02/2016 02:26 PM, Jason Wang wrote:
> On 2016年07月26日 09:49, Zhang Chen wrote:
> > This a COLO net ascii figure:
> >
> >   Primary qemu Secondary qemu
> > +----------------------------------------------------------+ 
> > +----------------------------------------------------------------+
> > | +-----------------------------------------------------+ |           
> > | +-----------------------------------------------------------+ |
> > | |                                                     | |           
> > | |                                                           | |
> > | |                        guest                        | |           
> > |  | guest                              | |
> > | |                                                     | |           
> > | |                                                           | |
> > | +-------^------------------+--------------------------+ |           
> > | +---------------------+--------+----------------------------+ |
> > |         |                  | |           |                        ^ 
> > |                              |
> > |         |                  | |           |                        | 
> > |                              |
> > |         | +------------------------------------------------------+ 
> > |                        |        | |
> > |netfilter|  |               | |        |  |   netfilter            | 
> > |                              |
> > | +----------+ -----------------------+ |        |  | 
> > +-----------------------------------------------------------+ |
> > | |       |  |               |        | |        |  |  
> > |                     |        |  filter excute order       | |
> > | |       |  |               |        | |        |  |  
> > |                     |        | +------------------->      | |
> > | |       |  |               |        | |        |  |  
> > |                     |        | TCP                      | |
> > | | +-----+--+--+     +------v-----+  | +------------+ |        |  |  
> > | +------------+  +---+----+---v+rewriter++ +------------+ | |
> > | | |           |     |            |  | |            | |        |  |  
> > | |            |  |        |              | |            | | |
> > | | |  filter   |     |   filter   +---->   colo <--------+     
> > +-------->  filter   +--> adjust | adjust     +-->   filter   | | |
> > | | |  mirror   |     | redirector |  | |  compare   |     | |        
> > |  | | redirector |  | ack    |   seq        |  | redirector | | |
> > | | |           |     |            |  | |            |     | |        
> > |  | |            |  |        |              | |            | | |
> > | | +----^------+     +------------+  | +-----+------+     | |        
> > |  | +------------+  +--------+--------------+ +---+--------+ | |
> > | |      |     tx                 rx  |       |            | |        
> > |  |            tx                        all       | rx      | |
> > | |      |                            |       |            | |        
> > | +-----------------------------------------------------------+ |
> > | |      |                            |       |            | |        
> > | |            |
> > | |      |   filter excute order      |       |            | |        
> > | |            |
> > | |      |  +------------------->     |       |            | 
> > +------------------------------------------------------------+ |
> > | +-----------------------------------+       | | | |
> > |        |                                    | | | |
> > +----------------------------------------------------------+ 
> > +----------------------------------------------------------------+
> >           |guest receive                       |guest send
> >           |                                    |
> > +--------+------------------------------------v------------+
> > | |                              NOTE: filter direction is rx/tx/all
> > |                         tap |                              
> > rx:receive packets sent to the netdev
> > | |                              tx:receive packets sent by the netdev
> > +----------------------------------------------------------+
> >
> > In COLO-compare.
> > Packets coming from the primary char indev will be sent to outdev
> > Packets coming from the secondary char dev will be dropped
> > colo-comapre need two input chardev and one output chardev:
> > primary_in=chardev1-id
> > secondary_in=chardev2-id
> > outdev=chardev3-id
>
> Though it has 'compare' in its name, this description needs some key 
> information still. e.g what it did (packet comparing). And then you 
> can describe where were the data sources from.

OK, I will add some comments for this.

> > usage:
> >
> > primary:
> > -netdev 
> > tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
> > -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
> > -chardev socket,id=mirror0,host=3.3.3.3,port=9003,server,nowait
> > -chardev socket,id=compare1,host=3.3.3.3,port=9004,server,nowait
> > -chardev socket,id=compare0,host=3.3.3.3,port=9001,server,nowait
> > -chardev socket,id=compare0-0,host=3.3.3.3,port=9001
> > -chardev socket,id=compare_out,host=3.3.3.3,port=9005,server,nowait
> > -chardev socket,id=compare_out0,host=3.3.3.3,port=9005
> > -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
> > -object 
> > filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out
>
> Looks like this filter were missed in the above figure.

Yes,I will update figure in next version.

> > -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0
> > -object 
> > colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0
> >
> > secondary:
> > -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down 
> > script=/etc/qemu-ifdown
> > -device e1000,netdev=hn0,mac=52:a4:00:12:78:66
> > -chardev socket,id=red0,host=3.3.3.3,port=9003
> > -chardev socket,id=red1,host=3.3.3.3,port=9004
> > -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
> > -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
> >
> > Signed-off-by: Zhang Chen <address@hidden>
> > Signed-off-by: Li Zhijian <address@hidden>
> > Signed-off-by: Wen Congyang <address@hidden>
> > ---
> >   net/Makefile.objs  |   1 +
> >   net/colo-compare.c | 222 
> > +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >   qemu-options.hx    |  38 +++++++++
> >   vl.c               |   3 +-
> >   4 files changed, 263 insertions(+), 1 deletion(-)
> >   create mode 100644 net/colo-compare.c
> >
> > diff --git a/net/Makefile.objs b/net/Makefile.objs
> > index b7c22fd..ba92f73 100644
> > --- a/net/Makefile.objs
> > +++ b/net/Makefile.objs
> > @@ -16,3 +16,4 @@ common-obj-$(CONFIG_NETMAP) += netmap.o
> >   common-obj-y += filter.o
> >   common-obj-y += filter-buffer.o
> >   common-obj-y += filter-mirror.o
> > +common-obj-y += colo-compare.o
> > diff --git a/net/colo-compare.c b/net/colo-compare.c
> > new file mode 100644
> > index 0000000..0402958
> > --- /dev/null
> > +++ b/net/colo-compare.c
> > @@ -0,0 +1,222 @@
> > +/*
> > + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service 
> > (COLO)
> > + * (a.k.a. Fault Tolerance or Continuous Replication)
> > + *
> > + * Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD.
> > + * Copyright (c) 2016 FUJITSU LIMITED
> > + * Copyright (c) 2016 Intel Corporation
> > + *
> > + * Author: Zhang Chen <address@hidden>
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or
> > + * later.  See the COPYING file in the top-level directory.
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +#include "qemu/error-report.h"
> > +#include "qemu-common.h"
> > +#include "qapi/qmp/qerror.h"
> > +#include "qapi/error.h"
> > +#include "net/net.h"
> > +#include "net/vhost_net.h"
> > +#include "qom/object_interfaces.h"
> > +#include "qemu/iov.h"
> > +#include "qom/object.h"
> > +#include "qemu/typedefs.h"
> > +#include "net/queue.h"
> > +#include "sysemu/char.h"
> > +#include "qemu/sockets.h"
> > +#include "qapi-visit.h"
> > +
> > +#define TYPE_COLO_COMPARE "colo-compare"
> > +#define COLO_COMPARE(obj) \
> > +    OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE)
> > +
> > +#define COMPARE_READ_LEN_MAX NET_BUFSIZE
> > +
> > +typedef struct CompareState {
> > +    Object parent;
> > +
> > +    char *pri_indev;
> > +    char *sec_indev;
> > +    char *outdev;
> > +    CharDriverState *chr_pri_in;
> > +    CharDriverState *chr_sec_in;
> > +    CharDriverState *chr_out;
> > +    QTAILQ_ENTRY(CompareState) next;
> > +    SocketReadState pri_rs;
> > +    SocketReadState sec_rs;
> > +} CompareState;
> > +
> > +typedef struct CompareClass {
> > +    ObjectClass parent_class;
> > +} CompareClass;
> > +
> > +static char *compare_get_pri_indev(Object *obj, Error **errp)
> > +{
> > +    CompareState *s = COLO_COMPARE(obj);
> > +
> > +    return g_strdup(s->pri_indev);
> > +}
> > +
> > +static void compare_set_pri_indev(Object *obj, const char *value, 
> > Error **errp)
> > +{
> > +    CompareState *s = COLO_COMPARE(obj);
> > +
> > +    g_free(s->pri_indev);
> > +    s->pri_indev = g_strdup(value);
> > +}
> > +
> > +static char *compare_get_sec_indev(Object *obj, Error **errp)
> > +{
> > +    CompareState *s = COLO_COMPARE(obj);
> > +
> > +    return g_strdup(s->sec_indev);
> > +}
> > +
> > +static void compare_set_sec_indev(Object *obj, const char *value, 
> > Error **errp)
> > +{
> > +    CompareState *s = COLO_COMPARE(obj);
> > +
> > +    g_free(s->sec_indev);
> > +    s->sec_indev = g_strdup(value);
> > +}
> > +
> > +static char *compare_get_outdev(Object *obj, Error **errp)
> > +{
> > +    CompareState *s = COLO_COMPARE(obj);
> > +
> > +    return g_strdup(s->outdev);
> > +}
> > +
> > +static void compare_set_outdev(Object *obj, const char *value, Error 
> > **errp)
> > +{
> > +    CompareState *s = COLO_COMPARE(obj);
> > +
> > +    g_free(s->outdev);
> > +    s->outdev = g_strdup(value);
> > +}
> > +
> > +static void compare_pri_rs_finalize(SocketReadState *pri_rs)
> > +{
> > +    /* if packet_enqueue pri pkt failed we will send unsupported 
> > packet */
> > +}
> > +
> > +static void compare_sec_rs_finalize(SocketReadState *sec_rs)
> > +{
> > +    /* if packet_enqueue sec pkt failed we will notify trace */
> > +}
> > +
> > +/*
> > + * called from the main thread on the primary
> > + * to setup colo-compare.
> > + */
> > +static void colo_compare_complete(UserCreatable *uc, Error **errp)
> > +{
> > +    CompareState *s = COLO_COMPARE(uc);
> > +
> > +    if (!s->pri_indev || !s->sec_indev || !s->outdev) {
> > +        error_setg(errp, "colo compare needs 'primary_in' ,"
> > +                   "'secondary_in','outdev' property set");
> > +        return;
> > +    } else if (!strcmp(s->pri_indev, s->outdev) ||
> > +               !strcmp(s->sec_indev, s->outdev) ||
> > +               !strcmp(s->pri_indev, s->sec_indev)) {
> > +        error_setg(errp, "'indev' and 'outdev' could not be same "
> > +                   "for compare module");
> > +        return;
>
> Can relax this by reusing the socket chardev? Looks we only use one 
> direction during packet passing.

We can't qemu_chr_fe_claim_no_fail(xxx) same one chardev twice.
Do you mean we should use -chardev socket,id=xx,....mux=on,... ?

> > +    }
> > +
> > +    s->chr_pri_in = qemu_chr_find(s->pri_indev);
> > +    if (s->chr_pri_in == NULL) {
> > +        error_setg(errp, "Primary IN Device '%s' not found",
> > +                   s->pri_indev);
> > +        return;
> > +    }
> > +
> > +    s->chr_sec_in = qemu_chr_find(s->sec_indev);
> > +    if (s->chr_sec_in == NULL) {
> > +        error_setg(errp, "Secondary IN Device '%s' not found",
> > +                   s->sec_indev);
> > +        return;
> > +    }
> > +
> > +    s->chr_out = qemu_chr_find(s->outdev);
> > +    if (s->chr_out == NULL) {
> > +        error_setg(errp, "OUT Device '%s' not found", s->outdev);
> > +        return;
> > +    }
>
> Like I point out in the past iterations, we probably need some 
> inspection here and fail if it was socket chardev. (Similar to what 
> vhost-use did for udp socket).

OK,I got it.

> > +
> > +    qemu_chr_fe_claim_no_fail(s->chr_pri_in);
> > +
> > +    qemu_chr_fe_claim_no_fail(s->chr_sec_in);
> > +
> > +    qemu_chr_fe_claim_no_fail(s->chr_out);
> > +
> > +    net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize);
> > +    net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize);
> > +
>
> Why chr_out is left here?

Because chr_out needn't read data from chardev, just output.

> > +    return;
> > +}
> > +
> > +static void colo_compare_class_init(ObjectClass *oc, void *data)
> > +{
> > +    UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc);
> > +
> > +    ucc->complete = colo_compare_complete;
> > +}
> > +
> > +static void colo_compare_init(Object *obj)
> > +{
> > +    object_property_add_str(obj, "primary_in",
> > +                            compare_get_pri_indev, 
> > compare_set_pri_indev,
> > +                            NULL);
> > +    object_property_add_str(obj, "secondary_in",
> > +                            compare_get_sec_indev, 
> > compare_set_sec_indev,
> > +                            NULL);
> > +    object_property_add_str(obj, "outdev",
> > +                            compare_get_outdev, compare_set_outdev,
> > +                            NULL);
> > +}
> > +
> > +static void colo_compare_finalize(Object *obj)
> > +{
> > +    CompareState *s = COLO_COMPARE(obj);
> > +
> > +    if (s->chr_pri_in) {
> > +        qemu_chr_add_handlers(s->chr_pri_in, NULL, NULL, NULL, NULL);
> > +        qemu_chr_fe_release(s->chr_pri_in);
> > +    }
> > +    if (s->chr_sec_in) {
> > +        qemu_chr_add_handlers(s->chr_sec_in, NULL, NULL, NULL, NULL);
> > +        qemu_chr_fe_release(s->chr_sec_in);
> > +    }
> > +    if (s->chr_out) {
> > +        qemu_chr_fe_release(s->chr_out);
> > +    }
> > +
> > +    g_free(s->pri_indev);
> > +    g_free(s->sec_indev);
> > +    g_free(s->outdev);
> > +}
> > +
> > +static const TypeInfo colo_compare_info = {
> > +    .name = TYPE_COLO_COMPARE,
> > +    .parent = TYPE_OBJECT,
> > +    .instance_size = sizeof(CompareState),
> > +    .instance_init = colo_compare_init,
> > +    .instance_finalize = colo_compare_finalize,
> > +    .class_size = sizeof(CompareClass),
> > +    .class_init = colo_compare_class_init,
> > +    .interfaces = (InterfaceInfo[]) {
> > +        { TYPE_USER_CREATABLE },
> > +        { }
> > +    }
> > +};
> > +
> > +static void register_types(void)
> > +{
> > +    type_register_static(&colo_compare_info);
> > +}
> > +
> > +type_init(register_types);
> > diff --git a/qemu-options.hx b/qemu-options.hx
> > index 587de8f..79e5896 100644
> > --- a/qemu-options.hx
> > +++ b/qemu-options.hx
> > @@ -3866,6 +3866,44 @@ Dump the network traffic on netdev @var{dev} 
> > to the file specified by
> >   The file format is libpcap, so it can be analyzed with tools such 
> > as tcpdump
> >   or Wireshark.
> >   address@hidden -object 
> > colo-compare,address@hidden,address@hidden,address@hidden,
> > address@hidden
> > +
> > +Colo-compare gets packet from address@hidden and 
> > address@hidden,than compare primary packet with
> > +secondary packet. if the packet same,we will output primary
>
> s/if/If/ and miss a blank before ','

will fix~

> > +packet to address@hidden,else we will notify colo-frame
>
> miss a blank before ','

will fix it.

> > +do checkpoint and send primary packet to address@hidden
> > +
> > +we can use it with the help of filter-mirror and filter-redirector.
> > +
> > +The simple usage:
>
> This line could be removed since we have example below.

will remove it.

> > +
> > address@hidden
> > +
> > +primary:
> > +-netdev 
> > tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
> > +-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
> > +-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server,nowait
> > +-chardev socket,id=compare1,host=3.3.3.3,port=9004,server,nowait
> > +-chardev socket,id=compare0,host=3.3.3.3,port=9001,server,nowait
> > +-chardev socket,id=compare0-0,host=3.3.3.3,port=9001
> > +-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server,nowait
> > +-chardev socket,id=compare_out0,host=3.3.3.3,port=9005
> > +-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
> > +-object 
> > filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out
> > +-object 
> > filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0
> > +-object 
> > colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0
> > +
> > +secondary:
> > +-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down 
> > script=/etc/qemu-ifdown
> > +-device e1000,netdev=hn0,mac=52:a4:00:12:78:66
> > +-chardev socket,id=red0,host=3.3.3.3,port=9003
> > +-chardev socket,id=red1,host=3.3.3.3,port=9004
> > +-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
> > +-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
> > +
> > address@hidden example
> > +
>
> Better add detail description on the above command line since it was 
> rather complex for starters.

We have add detail description in commit log, need we repeat it again here?


Thanks
Zhang Chen

> >   @item -object 
> > secret,address@hidden,address@hidden,address@hidden|base64}[,address@hidden,address@hidden
> >   @item -object 
> > secret,address@hidden,address@hidden,address@hidden|base64}[,address@hidden,address@hidden
> >   diff --git a/vl.c b/vl.c
> > index cbe51ac..c6b9a6f 100644
> > --- a/vl.c
> > +++ b/vl.c
> > @@ -2865,7 +2865,8 @@ static bool object_create_initial(const char 
> > *type)
> >       if (g_str_equal(type, "filter-buffer") ||
> >           g_str_equal(type, "filter-dump") ||
> >           g_str_equal(type, "filter-mirror") ||
> > -        g_str_equal(type, "filter-redirector")) {
> > +        g_str_equal(type, "filter-redirector") ||
> > +        g_str_equal(type, "colo-compare")) {
> >           return false;
> >       }
>
>
>
> .

--
Thanks
zhangchen