[Qemu-devel] [PATCH RFC v3 00/27] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service

[zhanghailiang]

This is the 3th version of COLO, it is only COLO frame part, include: VM 
checkpoint,
failover, proxy API, block replication API, not include block replication.
The block part has been sent by wencongyang:
'[RFC PATCH 00/14] Block replication for continuous checkpoints'

You can get the integrated qemu colo patches from github:
https://github.com/coloft/qemu/commits/colo-v1.0

Compared with the previous version, we have realized all parts of COLO frame, 
and it is works now.

The main change since last version is, we use colo proxy mode instead of
colo agent, they are all used for network packets compare, but proxy is more
efficient, it is based on netfilter.
Another modification is we implement new block replication scheme, 
you can get more info from wencongyang's block patch series 

If you don't know about COLO, please refer to below link for detailed 
information.

The idea is presented in Xen summit 2012, and 2013,
and academia paper in SOCC 2013. It's also presented in KVM forum in 2013:
http://www.linux-kvm.org/wiki/images/1/1d/Kvm-forum-2013-COLO.pdf

Previous posted RFC proposal:
http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg05567.html
http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg04459.html

The below is the detail about test COLO, you can also get this info
from http://wiki.qemu.org/Features/COLO.
* Hardware requirements
There is at least one directly connected nic to forward the network requests
from client to secondary VM. The directly connected nic must not be used by
any other purpose.

* Network link topology
=================================normal ======================================
                                +--------+
                                |client  |
   master                       +----+---+                    slave
-------------------------+           |            + -------------------------+
   PVM                   |           +            |                          |
+-------+         +----[eth0]-----[switch]-----[eth0]---------+              |
|guest  |     +---+-+    |                        |       +---+-+            |
|     [tap0]--+ br0 |    |                        |       | br0 |            |
|       |     +-----+  [eth1]-----[forward]----[eth1]--+  +-----+      SVM   |
+-------+                |                        |    |            +-------+|
                         |                        |    |  +-----+   | guest ||
                       [eth2]---[checkpoint]---[eth2]  +--+br1  |-[tap0]    ||
                         |                        |       +-----+   |       ||
                         |                        |                 +-------+|
-------------------------+                        +--------------------------+
e.g.
master:
br0: 192.168.0.33
eth1: 192.168.1.33
eth2: 192.168.2.33

slave:
br0: 192.168.0.88
br1: no ip address
eth1: 192.168.1.88
eth2: 192.168.2.88
(Actually, you can also use eth0 as checkpoint channel)
Note: in normal, SVM will always be linked to br1 like above until
failover.

* Test environment prepare:
1. Set Up the Bridge and network environment
You must setup you network environment like above picture,
In master, setup a bridge br0, using command brctl, like:
# ifconfig eth0 down
# ifconfig eth0 0.0.0.0
# brctl addbr br0
# brctl addif br0 eth0
# ifconfig br0 192.168.0.33 netmask 255.255.255.0
# ifconfig eth0 up
In slave, setup two bridge br0, br1, commands are same with above,
please note that br1 is linked to eth1(the forward nic).

2.Qemu-ifup
We need a script to bring up the TAP interface.
You can find this info from http://en.wikibooks.org/wiki/QEMU/Networking.
Master:
address@hidden cat /etc/qemu-ifup
#!/bin/sh
switch=br0
if [ -n "$1" ]; then
        ip link set $1 up
        brctl addif ${switch} $1
fi
Slave:
address@hidden # cat /etc/qemu-ifup
#!/bin/sh
switch=br1  #in primary, switch is br0. in secondary switch is br1
if [ -n "$1" ]; then
        ip link set $1 up
        brctl addif ${switch} $1
fi 

3. Prepare host kernel
colo-proxy kernel module need cooperate with linux kernel.
You should put a kernel patch 'colo-patch-for-kernel.patch'
(It's based on linux kernel-3.19) which you can get from 
https://github.com/gao-feng/colo-proxy.git
and then compile kernel and intall the new kernel.

4. Proxy module
proxy module is used for network packets compare, you can also get the lastest
version from: https://github.com/gao-feng/colo-proxy.git.
You can compile and install it by using command 'make' && 'make install'.

5. Modified iptables
We have add a new rule to iptables command, so please get the patch from
https://github.com/gao-feng/colo-proxy/blob/master/COLO-library_for_iptables-1.4.21.patch
It is based on version 1.4.21.

6. Qemu colo
Checkout the latest colo branch from
https://github.com/coloft/qemu/commits/colo-v1.0
configure and make: 
# ./configure --target-list=x86_64-softmmu --enable-colo --enable-quorum 
# make

* Test steps:
1. load module
# modprobe nf_conntrack_colo (Other colo module will be automatically loaded by
script colo-proxy-script.sh)
# modprobe xt_mark
# modprobe kvm-intel

2. startup qemu
master:
# qemu-system-x86_64 -enable-kvm -netdev 
tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1 -device 
virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive 
driver=quorum,read-pattern=first,children.0.file.filename=suse11_3.img,children.0.driver=raw,children.1.file.driver=nbd+colo,children.1.file.host=192.168.2.88,children.1.file.port=8889,children.1.file.export=colo1,children.1.driver=raw,if=virtio
 -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor 
stdio -S
slave:
# qemu-system-x86_64 -enable-kvm -netdev 
tap,id=hn0,colo_script=./scripts/colo-proxy-script.sh,colo_nicname=eth1 -device 
virtio-net-pci,id=net-pci0,netdev=hn0 -boot c -drive 
driver=blkcolo,export=colo1,backing.file.filename=suse11_3.img,backing.driver=raw,if=virtio
 -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor 
stdio -incoming tcp:0:8888

3. On Secondary VM's QEMU monitor, run
(qemu) nbd_server_start 192.168.2.88:8889 

4.on Primary VM's QEMU monitor, run following command:
(qemu) migrate_set_capability colo on
(qemu) migrate tcp:192.168.2.88:8888

5. done
You will see two runing VMs, whenever you make changes to PVM, SVM
will be synced to PVM's state.

6. failover test:
You can kill SVM (PVM) and run 'colo_lost_heartbeat' in PVM's (SVM's) monitor
at the same time, then PVM (SVM) will failover and client will not feel this
change.

It is still a framework, far away from commercial use,
so any comments/feedbacks are warmly welcomed ;)

PS: 
We (huawei) have cooperated with fujitsu on COLO work,
and we work mainly on COLO frame and fujitsu will focus on COLO block.

TODO list:
1) Optimize the process of checkpoint, shorten the time-consuming
2) Add more debug/stat info 
3) Strengthen failover 
4) The capability of continuous FT

v3:
- use proxy instead of colo agent to compare network packets
- add block replication
- Optimize failover disposal
- handle shutdown

v2:
- use QEMUSizedBuffer/QEMUFile as COLO buffer
- colo support is enabled by default
- add nic replication support
- addressed comments from Eric Blake and Dr. David Alan Gilbert

v1:
- implement the frame of colo


zhanghailiang (27):
  configure: Add parameter for configure to enable/disable COLO support
  migration: Introduce capability 'colo' to migration
  COLO: migrate colo related info to slave
  migration: Integrate COLO checkpoint process into migration
  migration: Integrate COLO checkpoint process into loadvm
  migration: Don't send vm description in COLO mode
  COLO: Implement colo checkpoint protocol
  COLO: Add a new RunState RUN_STATE_COLO
  QEMUSizedBuffer: Introduce two help functions for qsb
  COLO: Save VM state to slave when do checkpoint
  COLO RAM: Load PVM's dirty page into SVM's RAM cache temporarily
  COLO VMstate: Load VM state into qsb before restore it
  COLO RAM: Flush cached RAM into SVM's memory
  COLO failover: Introduce a new command to trigger a failover
  COLO failover: Implement COLO master/slave failover work
  COLO failover: Don't do failover during loading VM's state
  COLO: Add new command parameter 'colo_nicname' 'colo_script' for net
  COLO NIC: Init/remove colo nic devices when add/cleanup tap devices
  COLO NIC: Implement colo nic device interface configure()
  COLO NIC : Implement colo nic init/destroy function
  COLO NIC: Some init work related with proxy module
  COLO: Do checkpoint according to the result of net packets comparing
  COLO: Improve checkpoint efficiency by do additional periodic
    checkpoint
  COLO NIC: Implement NIC checkpoint and failover
  COLO: Disable qdev hotplug when VM is in COLO mode
  COLO: Implement shutdown checkpoint
  COLO: Add block replication into colo process

 arch_init.c                            | 196 ++++++++-
 configure                              |  14 +
 hmp-commands.hx                        |  15 +
 hmp.c                                  |   7 +
 hmp.h                                  |   1 +
 include/exec/cpu-all.h                 |   1 +
 include/migration/migration-colo.h     |  57 +++
 include/migration/migration-failover.h |  22 +
 include/migration/migration.h          |  14 +
 include/migration/qemu-file.h          |   3 +-
 include/net/colo-nic.h                 |  25 ++
 include/net/net.h                      |   4 +
 include/sysemu/sysemu.h                |   3 +
 migration/Makefile.objs                |   2 +
 migration/colo-comm.c                  |  81 ++++
 migration/colo-failover.c              |  48 +++
 migration/colo.c                       | 743 +++++++++++++++++++++++++++++++++
 migration/migration.c                  |  74 +++-
 migration/qemu-file-buf.c              |  57 +++
 net/Makefile.objs                      |   1 +
 net/colo-nic.c                         | 438 +++++++++++++++++++
 net/tap.c                              |  45 +-
 qapi-schema.json                       |  27 +-
 qemu-options.hx                        |  10 +-
 qmp-commands.hx                        |  19 +
 savevm.c                               |  10 +-
 scripts/colo-proxy-script.sh           |  88 ++++
 stubs/Makefile.objs                    |   1 +
 stubs/migration-colo.c                 |  49 +++
 vl.c                                   |  36 +-
 30 files changed, 2047 insertions(+), 44 deletions(-)
 create mode 100644 include/migration/migration-colo.h
 create mode 100644 include/migration/migration-failover.h
 create mode 100644 include/net/colo-nic.h
 create mode 100644 migration/colo-comm.c
 create mode 100644 migration/colo-failover.c
 create mode 100644 migration/colo.c
 create mode 100644 net/colo-nic.c
 create mode 100755 scripts/colo-proxy-script.sh
 create mode 100644 stubs/migration-colo.c

-- 
1.7.12.4