Re: [Qemu-devel] [PATCH RFC v3 19/27] COLO NIC: Implement colo nic device interface configure()

[Dr. David Alan Gilbert]

* zhanghailiang (address@hidden) wrote:
> On 2015/2/25 17:08, Dr. David Alan Gilbert wrote:
> >* zhanghailiang (address@hidden) wrote:
> >>On 2015/2/16 20:03, Dr. David Alan Gilbert wrote:
> >>>* zhanghailiang (address@hidden) wrote:
> >>>>Implement colo nic device interface configure()
> >>>>add a script to configure nic devices:
> >>>>${QEMU_SCRIPT_DIR}/colo-proxy-script.sh
> >>>
> >>>Do you have some more documentation of the new colo-proxy?  I've
> >>
> >>Yes, gaofeng is writing it now...
> >
> >Great.
> >
> >>>been reading the kernel module source and I can see that it's
> >>>a nice idea to do the sequence number adjustment on the host,
> >>>that reduces the need to modify the guest kernel; I was trying to
> >>>figure out how you synchronise the master/slave idea of sequence numbers -
> >>>is that purely from the 'ack' that's duplicated back to the secondary?
> >>
> >>Yes, you've got it :)
> >>
> >>>If you were unlucky and the 'ack' packet was lost on the duplicated
> >>>link from the primary to secondary how would you recover?
> >>
> >>The 'ack' packet will be consider to be lost, because the primary will not
> >>respond to this 'ack' packet until it got secondary's response,
> >>and client will resend it ('ack' packet).
> >>
> >>>What about TCP connections setup before colo was activated?
> >>>
> >>
> >>Actually, now, we only support activate colo before guest is startup (for 
> >>test procedure,
> >>'-S' is needed for qemu command line).
> >
> >Consider this:
> >      1) Start primary
> >      2) Start secondary
> >      3) Start the colo pairing
> >      4) Primary fails
> >      5) Colo failover to secondary
> >
> >Now we have only the old secondary running; we'd really like to get back to
> >having a pair of fault-tolerant hosts, so it would be good to be able to:
> >
> >      6) Make the old secondary the new primary
> >      7) Add a new secondary
> >      8) Start colo-pairing to the new secondary
> >
> 
> Er, what you described is continuous FT, yes, it is in our TODO list.

Ah, OK, I wondered what that meant.

Dave

> >You could theoretically do this with colo-agent, but not with colo-proxy.
> >
> 
> We have decided using colo-proxy, it has more advantages.
> 
> >>>The other thought is that passing the 'sec_dev' as a module parameter
> >>>gives you an artificial limitation; it forces all of the pairs
> >>>to be between the same pair of hosts.   If the 'sec_dev' was a parameter
> >>>to the connection then you could have different slaves associated with
> >>>each guest on the primary host.
> >>>
> >>
> >>Hmm, do you mean we should pass this 'sec_dev' as a parameter from qemu to 
> >>proxy module by
> >>maybe ioctl ?
> >
> >Yes, ioctl or tc or whatever; and make it per-guest.
> >
> 
> OK. Thanks.
> 
> >>Yes, it is ugly to pass this 'sec_dev' directly to module as parameter.
> >>We will consider this, thanks ;)
> >
> >Thanks!
> >
> >>>Dave
> >>>P.S. You probably need to clean the debug messages up in the kernel module!
> >>>
> >>
> >>OK, will do that.
> >
> >Thanks.
> >
> >Dave
> >
> >>
> >>>>Signed-off-by: zhanghailiang <address@hidden>
> >>>>Signed-off-by: Gao feng <address@hidden>
> >>>>Signed-off-by: Li Zhijian <address@hidden>
> >>>>---
> >>>>  net/colo-nic.c               | 56 +++++++++++++++++++++++++++-
> >>>>  scripts/colo-proxy-script.sh | 88 
> >>>> ++++++++++++++++++++++++++++++++++++++++++++
> >>>>  2 files changed, 143 insertions(+), 1 deletion(-)
> >>>>  create mode 100755 scripts/colo-proxy-script.sh
> >>>>
> >>>>diff --git a/net/colo-nic.c b/net/colo-nic.c
> >>>>index 965af49..f8fc35d 100644
> >>>>--- a/net/colo-nic.c
> >>>>+++ b/net/colo-nic.c
> >>>>@@ -39,12 +39,66 @@ static bool colo_nic_support(NetClientState *nc)
> >>>>      return nc && nc->colo_script[0] && nc->colo_nicname[0];
> >>>>  }
> >>>>
> >>>>+static int launch_colo_script(char *argv[])
> >>>>+{
> >>>>+    int pid, status;
> >>>>+    char *script = argv[0];
> >>>>+
> >>>>+    /* try to launch network script */
> >>>>+    pid = fork();
> >>>>+    if (pid == 0) {
> >>>>+        execv(script, argv);
> >>>>+        _exit(1);
> >>>>+    } else if (pid > 0) {
> >>>>+        while (waitpid(pid, &status, 0) != pid) {
> >>>>+            /* loop */
> >>>>+        }
> >>>>+
> >>>>+        if (WIFEXITED(status) && WEXITSTATUS(status) == 0) {
> >>>>+            return 0;
> >>>>+        }
> >>>>+    }
> >>>>+    return -1;
> >>>>+}
> >>>>+
> >>>>+static int colo_nic_configure(NetClientState *nc,
> >>>>+            bool up, int side, int index)
> >>>>+{
> >>>>+    int i, argc = 6;
> >>>>+    char *argv[7], index_str[32];
> >>>>+    char **parg;
> >>>>+
> >>>>+    if (!nc && index <= 0) {
> >>>>+        error_report("Can not parse colo_script or colo_nicname");
> >>>>+        return -1;
> >>>>+    }
> >>>>+
> >>>>+    parg = argv;
> >>>>+    *parg++ = nc->colo_script;
> >>>>+    *parg++ = (char *)(side == COLO_SECONDARY_MODE ? "slave" : "master");
> >>>>+    *parg++ = (char *)(up ? "install" : "uninstall");
> >>>>+    *parg++ = nc->colo_nicname;
> >>>>+    *parg++ = nc->ifname;
> >>>>+    sprintf(index_str, "%d", index);
> >>>>+    *parg++ = index_str;
> >>>>+    *parg = NULL;
> >>>>+
> >>>>+    for (i = 0; i < argc; i++) {
> >>>>+        if (!argv[i][0]) {
> >>>>+            error_report("Can not get colo_script argument");
> >>>>+            return -1;
> >>>>+        }
> >>>>+    }
> >>>>+
> >>>>+    return launch_colo_script(argv);
> >>>>+}
> >>>>+
> >>>>  void colo_add_nic_devices(NetClientState *nc)
> >>>>  {
> >>>>      struct nic_device *nic = g_malloc0(sizeof(*nic));
> >>>>
> >>>>      nic->support_colo = colo_nic_support;
> >>>>-    nic->configure = NULL;
> >>>>+    nic->configure = colo_nic_configure;
> >>>>      /*
> >>>>       * TODO
> >>>>       * only support "-netdev tap,colo_scripte..."  options
> >>>>diff --git a/scripts/colo-proxy-script.sh b/scripts/colo-proxy-script.sh
> >>>>new file mode 100755
> >>>>index 0000000..c7aa53f
> >>>>--- /dev/null
> >>>>+++ b/scripts/colo-proxy-script.sh
> >>>>@@ -0,0 +1,88 @@
> >>>>+#!/bin/sh
> >>>>+#usage: ./colo-proxy-script.sh master/slave install/uninstall phy_if 
> >>>>virt_if index
> >>>>+#.e.g ./colo-proxy-script.sh master install eth2 tap0 1
> >>>>+
> >>>>+side=$1
> >>>>+action=$2
> >>>>+phy_if=$3
> >>>>+virt_if=$4
> >>>>+index=$5
> >>>>+br=br1
> >>>>+failover_br=br0
> >>>>+
> >>>>+script_usage()
> >>>>+{
> >>>>+    echo -n "usage: ./colo-proxy-script.sh master/slave "
> >>>>+    echo -e "install/uninstall phy_if virt_if index\n"
> >>>>+}
> >>>>+
> >>>>+master_install()
> >>>>+{
> >>>>+    tc qdisc add dev $virt_if root handle 1: prio
> >>>>+    tc filter add dev $virt_if parent 1: protocol ip prio 10 u32 match 
> >>>>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>>>+    tc filter add dev $virt_if parent 1: protocol arp prio 11 u32 match 
> >>>>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>>>+    tc filter add dev $virt_if parent 1: protocol ipv6 prio 12 u32 match 
> >>>>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>>>+
> >>>>+    modprobe nf_conntrack_ipv4
> >>>>+    modprobe xt_PMYCOLO sec_dev=$phy_if
> >>>>+
> >>>>+    /usr/local/sbin/iptables -t mangle -I PREROUTING -m physdev 
> >>>>--physdev-in $virt_if -j PMYCOLO --index $index
> >>>>+    /usr/local/sbin/ip6tables -t mangle -I PREROUTING -m physdev 
> >>>>--physdev-in $virt_if -j PMYCOLO --index $index
> >>>>+    /usr/local/sbin/arptables -I INPUT -i $phy_if -j MARK --set-mark 
> >>>>$index
> >>>>+}
> >>>>+
> >>>>+master_uninstall()
> >>>>+{
> >>>>+    tc filter del dev $virt_if parent 1: protocol ip prio 10 u32 match 
> >>>>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>>>+    tc filter del dev $virt_if parent 1: protocol arp prio 11 u32 match 
> >>>>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>>>+    tc filter del dev $virt_if parent 1: protocol ipv6 prio 12 u32 match 
> >>>>u32 0 0 flowid 1:2 action mirred egress mirror dev $phy_if
> >>>>+    tc qdisc del dev $virt_if root handle 1: prio
> >>>>+
> >>>>+    /usr/local/sbin/iptables -t mangle -F
> >>>>+    /usr/local/sbin/ip6tables -t mangle -F
> >>>>+    /usr/local/sbin/arptables -F
> >>>>+    rmmod xt_PMYCOLO
> >>>>+}
> >>>>+
> >>>>+slave_install()
> >>>>+{
> >>>>+    brctl addif $br $phy_if
> >>>>+    modprobe xt_SECCOLO
> >>>>+
> >>>>+    /usr/local/sbin/iptables -t mangle -I PREROUTING -m physdev 
> >>>>--physdev-in $virt_if -j SECCOLO --index $index
> >>>>+    /usr/local/sbin/ip6tables -t mangle -I PREROUTING -m physdev 
> >>>>--physdev-in $virt_if -j SECCOLO --index $index
> >>>>+}
> >>>>+
> >>>>+
> >>>>+slave_uninstall()
> >>>>+{
> >>>>+    brctl delif $br $phy_if
> >>>>+    brctl delif $br $virt_if
> >>>>+    brctl addif $failover_br $virt_if
> >>>>+
> >>>>+    /usr/local/sbin/iptables -t mangle -F
> >>>>+    /usr/local/sbin/ip6tables -t mangle -F
> >>>>+    rmmod xt_SECCOLO
> >>>>+}
> >>>>+
> >>>>+if [ $# -ne 5 ]; then
> >>>>+    script_usage
> >>>>+    exit 1
> >>>>+fi
> >>>>+
> >>>>+if [ "x$side" != "xmaster" ] && [ "x$side" != "xslave" ]; then
> >>>>+    script_usage
> >>>>+    exit 2
> >>>>+fi
> >>>>+
> >>>>+if [ "x$action" != "xinstall" ] && [ "x$action" != "xuninstall" ]; then
> >>>>+    script_usage
> >>>>+    exit 3
> >>>>+fi
> >>>>+
> >>>>+if [ $index -lt 0 ] || [ $index -gt 100 ]; then
> >>>>+    echo "index overflow"
> >>>>+    exit 4
> >>>>+fi
> >>>>+
> >>>>+${side}_${action}
> >>>>--
> >>>>1.7.12.4
> >>>>
> >>>>
> >>>--
> >>>Dr. David Alan Gilbert / address@hidden / Manchester, UK
> >>>
> >>>.
> >>>
> >>
> >>
> >--
> >Dr. David Alan Gilbert / address@hidden / Manchester, UK
> >
> >.
> >
> 
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK