Re: [Qemu-devel] [PATCH RFC v3 00/27] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service

[Dr. David Alan Gilbert]

* zhanghailiang (address@hidden) wrote:
> This is the 3th version of COLO, it is only COLO frame part, include: VM 
> checkpoint,
> failover, proxy API, block replication API, not include block replication.
> The block part has been sent by wencongyang:
> '[RFC PATCH 00/14] Block replication for continuous checkpoints'
> 
> You can get the integrated qemu colo patches from github:
> https://github.com/coloft/qemu/commits/colo-v1.0
> 
> Compared with the previous version, we have realized all parts of COLO frame, 
> and it is works now.
> 
> The main change since last version is, we use colo proxy mode instead of
> colo agent, they are all used for network packets compare, but proxy is more
> efficient, it is based on netfilter.
> Another modification is we implement new block replication scheme, 
> you can get more info from wencongyang's block patch series 
> 
> If you don't know about COLO, please refer to below link for detailed 
> information.
> 
> The idea is presented in Xen summit 2012, and 2013,
> and academia paper in SOCC 2013. It's also presented in KVM forum in 2013:
> http://www.linux-kvm.org/wiki/images/1/1d/Kvm-forum-2013-COLO.pdf
> 
> Previous posted RFC proposal:
> http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg05567.html
> http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg04459.html
> 
> The below is the detail about test COLO, you can also get this info
> from http://wiki.qemu.org/Features/COLO.
> * Hardware requirements
> There is at least one directly connected nic to forward the network requests
> from client to secondary VM. The directly connected nic must not be used by
> any other purpose.
> 
> * Network link topology
> =================================normal ======================================
>                                 +--------+
>                                 |client  |
>    master                       +----+---+                    slave
> -------------------------+           |            + -------------------------+
>    PVM                   |           +            |                          |
> +-------+         +----[eth0]-----[switch]-----[eth0]---------+              |
> |guest  |     +---+-+    |                        |       +---+-+            |
> |     [tap0]--+ br0 |    |                        |       | br0 |            |
> |       |     +-----+  [eth1]-----[forward]----[eth1]--+  +-----+      SVM   |
> +-------+                |                        |    |            +-------+|
>                          |                        |    |  +-----+   | guest ||
>                        [eth2]---[checkpoint]---[eth2]  +--+br1  |-[tap0]    ||
>                          |                        |       +-----+   |       ||
>                          |                        |                 +-------+|
> -------------------------+                        +--------------------------+
> e.g.
> master:
> br0: 192.168.0.33
> eth1: 192.168.1.33
> eth2: 192.168.2.33
> 
> slave:
> br0: 192.168.0.88
> br1: no ip address
> eth1: 192.168.1.88
> eth2: 192.168.2.88
> (Actually, you can also use eth0 as checkpoint channel)
> Note: in normal, SVM will always be linked to br1 like above until
> failover.

Why does eth1 need IP addresses?  Isn't the traffic on eth1 just a copy of the
traffic on eth0 for the proxy modules to compare/forward?
Wouldn't any ARP traffic or the like generated from having IPs on those
interfaces confuse the comparison process?
(Similarly for the bridges, is it best to turn off STP and the like
to stop the bridges adding extra packets on eth1/eth0 ?)

Dave

> * Test environment prepare:
> 1. Set Up the Bridge and network environment
> You must setup you network environment like above picture,
> In master, setup a bridge br0, using command brctl, like:
> # ifconfig eth0 down
> # ifconfig eth0 0.0.0.0
> # brctl addbr br0
> # brctl addif br0 eth0
> # ifconfig br0 192.168.0.33 netmask 255.255.255.0
> # ifconfig eth0 up
> In slave, setup two bridge br0, br1, commands are same with above,
> please note that br1 is linked to eth1(the forward nic).
> 
> 2.Qemu-ifup
> We need a script to bring up the TAP interface.
> You can find this info from http://en.wikibooks.org/wiki/QEMU/Networking.
> Master:
> address@hidden cat /etc/qemu-ifup
> #!/bin/sh
> switch=br0
> if [ -n "$1" ]; then
>         ip link set $1 up
>         brctl addif ${switch} $1
> fi
> Slave:
> address@hidden # cat /etc/qemu-ifup
> #!/bin/sh
> switch=br1  #in primary, switch is br0. in secondary switch is br1
> if [ -n "$1" ]; then
>         ip link set $1 up
>         brctl addif ${switch} $1
> fi 
> 
> 3. Prepare host kernel
> colo-proxy kernel module need cooperate with linux kernel.
> You should put a kernel patch 'colo-patch-for-kernel.patch'
> (It's based on linux kernel-3.19) which you can get from 
> https://github.com/gao-feng/colo-proxy.git
> and then compile kernel and intall the new kernel.
> 
> 4. Proxy module
> proxy module is used for network packets compare, you can also get the lastest
> version from: https://github.com/gao-feng/colo-proxy.git.
> You can compile and install it by using command 'make' && 'make install'.
> 
> 5. Modified iptables
> We have add a new rule to iptables command, so please get the patch from
> https://github.com/gao-feng/colo-proxy/blob/master/COLO-library_for_iptables-1.4.21.patch
> It is based on version 1.4.21.
> 
> 6. Qemu colo
> Checkout the latest colo branch from
> https://github.com/coloft/qemu/commits/colo-v1.0
> configure and make: 
> # ./configure --target-list=x86_64-softmmu --enable-colo --enable-quorum 
> # make
> 
> * Test steps:
> 1. load module
> # modprobe nf_conntrack_colo (Other colo module will be automatically loaded 
> by
> script colo-proxy-script.sh)
> # modprobe xt_mark
> # modprobe kvm-intel
> 
> 2. startup qemu
> master:
> # qemu-system-x86_64 -enable-kvm -netdev 
> tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1 
> -device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive 
> driver=quorum,read-pattern=first,children.0.file.filename=suse11_3.img,children.0.driver=raw,children.1.file.driver=nbd+colo,children.1.file.host=192.168.2.88,children.1.file.port=8889,children.1.file.export=colo1,children.1.driver=raw,if=virtio
>  -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor 
> stdio -S
> slave:
> # qemu-system-x86_64 -enable-kvm -netdev 
> tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1 
> -device virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive 
> driver=blkcolo,export=colo1,backing.file.filename=suse11_3.img,backing.driver=raw,if=virtio
>  -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor 
> stdio -incoming tcp:0:8888
> 
> 3. On Secondary VM's QEMU monitor, run
> (qemu) nbd_server_start 192.168.2.88:8889 
> 
> 4.on Primary VM's QEMU monitor, run following command:
> (qemu) migrate_set_capability colo on
> (qemu) migrate tcp:192.168.2.88:8888
> 
> 5. done
> You will see two runing VMs, whenever you make changes to PVM, SVM
> will be synced to PVM's state.
> 
> 6. failover test:
> You can kill SVM (PVM) and run 'colo_lost_heartbeat' in PVM's (SVM's) monitor
> at the same time, then PVM (SVM) will failover and client will not feel this
> change.
> 
> It is still a framework, far away from commercial use,
> so any comments/feedbacks are warmly welcomed ;)
> 
> PS: 
> We (huawei) have cooperated with fujitsu on COLO work,
> and we work mainly on COLO frame and fujitsu will focus on COLO block.
> 
> TODO list:
> 1) Optimize the process of checkpoint, shorten the time-consuming
> 2) Add more debug/stat info 
> 3) Strengthen failover 
> 4) The capability of continuous FT
> 
> v3:
> - use proxy instead of colo agent to compare network packets
> - add block replication
> - Optimize failover disposal
> - handle shutdown
> 
> v2:
> - use QEMUSizedBuffer/QEMUFile as COLO buffer
> - colo support is enabled by default
> - add nic replication support
> - addressed comments from Eric Blake and Dr. David Alan Gilbert
> 
> v1:
> - implement the frame of colo
> 
> 
> zhanghailiang (27):
>   configure: Add parameter for configure to enable/disable COLO support
>   migration: Introduce capability 'colo' to migration
>   COLO: migrate colo related info to slave
>   migration: Integrate COLO checkpoint process into migration
>   migration: Integrate COLO checkpoint process into loadvm
>   migration: Don't send vm description in COLO mode
>   COLO: Implement colo checkpoint protocol
>   COLO: Add a new RunState RUN_STATE_COLO
>   QEMUSizedBuffer: Introduce two help functions for qsb
>   COLO: Save VM state to slave when do checkpoint
>   COLO RAM: Load PVM's dirty page into SVM's RAM cache temporarily
>   COLO VMstate: Load VM state into qsb before restore it
>   COLO RAM: Flush cached RAM into SVM's memory
>   COLO failover: Introduce a new command to trigger a failover
>   COLO failover: Implement COLO master/slave failover work
>   COLO failover: Don't do failover during loading VM's state
>   COLO: Add new command parameter 'colo_nicname' 'colo_script' for net
>   COLO NIC: Init/remove colo nic devices when add/cleanup tap devices
>   COLO NIC: Implement colo nic device interface configure()
>   COLO NIC : Implement colo nic init/destroy function
>   COLO NIC: Some init work related with proxy module
>   COLO: Do checkpoint according to the result of net packets comparing
>   COLO: Improve checkpoint efficiency by do additional periodic
>     checkpoint
>   COLO NIC: Implement NIC checkpoint and failover
>   COLO: Disable qdev hotplug when VM is in COLO mode
>   COLO: Implement shutdown checkpoint
>   COLO: Add block replication into colo process
> 
>  arch_init.c                            | 196 ++++++++-
>  configure                              |  14 +
>  hmp-commands.hx                        |  15 +
>  hmp.c                                  |   7 +
>  hmp.h                                  |   1 +
>  include/exec/cpu-all.h                 |   1 +
>  include/migration/migration-colo.h     |  57 +++
>  include/migration/migration-failover.h |  22 +
>  include/migration/migration.h          |  14 +
>  include/migration/qemu-file.h          |   3 +-
>  include/net/colo-nic.h                 |  25 ++
>  include/net/net.h                      |   4 +
>  include/sysemu/sysemu.h                |   3 +
>  migration/Makefile.objs                |   2 +
>  migration/colo-comm.c                  |  81 ++++
>  migration/colo-failover.c              |  48 +++
>  migration/colo.c                       | 743 
> +++++++++++++++++++++++++++++++++
>  migration/migration.c                  |  74 +++-
>  migration/qemu-file-buf.c              |  57 +++
>  net/Makefile.objs                      |   1 +
>  net/colo-nic.c                         | 438 +++++++++++++++++++
>  net/tap.c                              |  45 +-
>  qapi-schema.json                       |  27 +-
>  qemu-options.hx                        |  10 +-
>  qmp-commands.hx                        |  19 +
>  savevm.c                               |  10 +-
>  scripts/colo-proxy-script.sh           |  88 ++++
>  stubs/Makefile.objs                    |   1 +
>  stubs/migration-colo.c                 |  49 +++
>  vl.c                                   |  36 +-
>  30 files changed, 2047 insertions(+), 44 deletions(-)
>  create mode 100644 include/migration/migration-colo.h
>  create mode 100644 include/migration/migration-failover.h
>  create mode 100644 include/net/colo-nic.h
>  create mode 100644 migration/colo-comm.c
>  create mode 100644 migration/colo-failover.c
>  create mode 100644 migration/colo.c
>  create mode 100644 net/colo-nic.c
>  create mode 100755 scripts/colo-proxy-script.sh
>  create mode 100644 stubs/migration-colo.c
> 
> -- 
> 1.7.12.4
> 
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK