[Qemu-devel] [PATCH V3 00/20] Multiqueue virtio-net

[Jason Wang]

Hello all:

This seires is an update of last version of multiqueue virtio-net support.

This series tries to brings multiqueue support to virtio-net through a
multiqueue support tap backend and multiple vhost threads.

To support this, multiqueue nic support were added to qemu. This is done by
introducing an array of NetClientStates in NICState, and make each pair of peers
to be an queue of the nic. This is done in patch 1-7.

Tap were also converted to be able to create a multiple queue
backend. Currently, only linux support this by issuing TUNSETIFF N times with
the same device name to create N queues. Each fd returned by TUNSETIFF were a
queue supported by kernel. Three new command lines were introduced, "queues"
were used to tell how many queues will be created by qemu; "fds" were used to
pass multiple pre-created tap file descriptors to qemu; "vhostfds" were used to
pass multiple pre-created vhost descriptors to qemu. This is done in patch 8-13.

A method of deleting a queue and queue_index were also introduce for virtio,
this is done in patch 14-15.

Vhost were also changed to support multiqueue by introducing a start vq index
which tracks the first virtqueue that will be used by vhost instead of the
assumption that the vhost always use virtqueue from index 0. This is done in
patch 16.

The last part is the multiqueue userspace changes, this is done in patch 17-20.

With this changes, user could start a multiqueue virtio-net device through

./qemu -netdev tap,id=hn0,queues=2,vhost=on -device virtio-net-pci,netdev=hn0

Management tools such as libvirt can pass multiple pre-created fds/vhostfds 
through

./qemu -netdev tap,id=hn0,fds=X:Y,vhostfds=M:N -device virtio-net-pci,netdev=hn0

For the one who wants to try, a git tree is available at:
git://github.com/jasowang/qemu.git

Changes from V2:
- Don't start/stop vhost threads when changing queues and simplify the interface
  between virtio-net and vhost further.

Changes from V1:
- silent checkpatch (Blue)
- use fds/vhostfds instead of fd/vhostfd (Stefan)
- use fds="X:Y:Z" instead of fd=X,fd=Y,fd=Z (Anthony)
- split patches (Stefan)
- typos in commit log (Stefan)
- Warn 'queues=' when fds/vhostfds is used (Stefan)
- rename __net_init_tap to net_init_tap_one (Stefan)
- check the consistency of vnet_hdr of multiple tap fds (Stefan)
- disable multiqueue support for bridge-helper (Stefan)
- rename tap_attach()/tap_detach() to tap_enable()/tap_disable() (Stefan)
- fix booting with legacy guest (WanLong)
- don't bump the version when doing migration (Michael)
- simplify the interface between virtio-net and multiqueue vhost_net (Michael)
- rebase the patches to latest
- re-order the patches that let the net part comes first to simplify the
  reviewing
- simplify the interface between virtio-net and multiqueue vhost_net
- move the guest notifiers setup from vhost to vhost_net
- fix a build issue of hw/mcf_fce.c

Changes from RFC v2:
- rebase the codes to latest qemu
- align the multiqueue virtio-net implementation to virtio spec
- split the patches into more smaller patches
- set_link and hotplug support

Changes from RFC V1:
- rebase to the latest
- fix memory leak in parse_netdev
- fix guest notifiers assignment/de-assignment
- changes the command lines to:
   qemu -netdev tap,queues=2 -device virtio-net-pci,queues=2

Reference:
V1: http://lists.nongnu.org/archive/html/qemu-devel/2012-12/msg03558.html
RFC v2: http://lists.gnu.org/archive/html/qemu-devel/2012-06/msg04108.html
RFC v1: http://comments.gmane.org/gmane.comp.emulators.qemu/100481

Perf Numbers:
- norm is short for normalize result
- trans.rate is short for transaction rate

Two Intel Xeon 5620 with direct connected intel 82599EB
Host/Guest kernel: David net tree
vhost enabled

- lots of improvents of both latency and cpu utilization in request-reponse test
- get regression of guest sending small packets which because TCP tends to batch
  less when the latency were improved

1q/2q/4q
TCP_RR
 size #sessions trans.rate  norm trans.rate  norm trans.rate  norm
1 1     9393.26   595.64  9408.18   597.34  9375.19   584.12
1 20    72162.1   2214.24 129880.22 2456.13 196949.81 2298.13
1 50    107513.38 2653.99 139721.93 2490.58 259713.82 2873.57
1 100   126734.63 2676.54 145553.5  2406.63 265252.68 2943
64 1    9453.42   632.33  9371.37   616.13  9338.19   615.97
64 20   70620.03  2093.68 125155.75 2409.15 191239.91 2253.32
64 50   106966    2448.29 146518.67 2514.47 242134.07 2720.91
64 100  117046.35 2394.56 190153.09 2696.82 238881.29 2704.41
256 1   8733.29   736.36  8701.07   680.83  8608.92   530.1
256 20  69279.89  2274.45 115103.07 2299.76 144555.16 1963.53
256 50  97676.02  2296.09 150719.57 2522.92 254510.5  3028.44
256 100 150221.55 2949.56 197569.3  2790.92 300695.78 3494.83
TCP_CRR
 size #sessions trans.rate  norm trans.rate  norm trans.rate  norm
1 1     2848.37  163.41 2230.39  130.89 2013.09  120.47
1 20    23434.5  562.11 31057.43 531.07 49488.28 564.41
1 50    28514.88 582.17 40494.23 605.92 60113.35 654.97
1 100   28827.22 584.73 48813.25 661.6  61783.62 676.56
64 1    2780.08  159.4  2201.07  127.96 2006.8   117.63
64 20   23318.51 564.47 30982.44 530.24 49734.95 566.13
64 50   28585.72 582.54 40576.7  610.08 60167.89 656.56
64 100  28747.37 584.17 49081.87 667.87 60612.94 662
256 1   2772.08  160.51 2231.84  131.05 2003.62  113.45
256 20  23086.35 559.8  30929.09 528.16 48454.9  555.22
256 50  28354.7  579.85 40578.31 607    60261.71 657.87
256 100 28844.55 585.67 48541.86 659.08 61941.07 676.72
TCP_STREAM guest receiving
 size #sessions throughput  norm throughput  norm throughput  norm
1 1     16.27   1.33   16.1    1.12   16.13   0.99
1 2     33.04   2.08   32.96   2.19   32.75   1.98
1 4     66.62   6.83   68.3    5.56   66.14   2.65
64 1    896.55  56.67  914.02  58.14  898.9   61.56
64 2    1830.46 91.02  1812.02 64.59  1835.57 66.26
64 4    3626.61 142.55 3636.25 100.64 3607.46 75.03
256 1   2619.49 131.23 2543.19 129.03 2618.69 132.39
256 2   5136.58 203.02 5163.31 141.11 5236.51 149.4
256 4   7063.99 242.83 9365.4  208.49 9421.03 159.94
512 1   3592.43 165.24 3603.12 167.19 3552.5  169.57
512 2   7042.62 246.59 7068.46 180.87 7258.52 186.3
512 4   6996.08 241.49 9298.34 206.12 9418.52 159.33
1024 1  4339.54 192.95 4370.2  191.92 4211.72 192.49
1024 2  7439.45 254.77 9403.99 215.24 9120.82 222.67
1024 4  7953.86 272.11 9403.87 208.23 9366.98 159.49
4096 1  7696.28 272.04 7611.41 270.38 7778.71 267.76
4096 2  7530.35 261.1  8905.43 246.27 8990.18 267.57
4096 4  7121.6  247.02 9411.75 206.71 9654.96 184.67
16384 1 7795.73 268.54 7780.94 267.2  7634.26 260.73
16384 2 7436.57 255.81 9381.86 220.85 9392    220.36
16384 4 7199.07 247.81 9420.96 205.87 9373.69 159.57
TCP_MAERTS guest sending
 size #sessions throughput  norm throughput  norm throughput  norm
1 1     15.94   0.62   15.55   0.61   15.13   0.59
1 2     36.11   0.83   32.46   0.69   32.28   0.69
1 4     71.59   1      68.91   0.94   61.52   0.77
64 1    630.71  22.52  622.11  22.35  605.09  21.84
64 2    1442.36 30.57  1292.15 25.82  1282.67 25.55
64 4    3186.79 42.59  2844.96 36.03  2529.69 30.06
256 1   1760.96 58.07  1738.44 57.43  1695.99 56.19
256 2   4834.23 95.19  3524.85 64.21  3511.94 64.45
256 4   9324.63 145.74 8956.49 116.39 6720.17 73.86
512 1   2678.03 84.1   2630.68 82.93  2636.54 82.57
512 2   9368.17 195.61 9408.82 204.53 5316.3  92.99
512 4   9186.34 209.68 9358.72 183.82 9489.29 160.42
1024 1  3620.71 109.88 3625.54 109.83 3606.61 112.35
1024 2  9429    258.32 7082.79 120.55 7403.53 134.78
1024 4  9430.66 290.44 9499.29 232.31 9414.6  190.92
4096 1  9339.28 296.48 9374.23 372.88 9348.76 298.49
4096 2  9410.53 378.69 9412.61 286.18 9409.75 278.31
4096 4  9487.35 374.1  9556.91 288.81 9441.94 221.64
16384 1 9380.43 403.8  9379.78 399.13 9382.42 393.55
16384 2 9367.69 406.93 9415.04 312.68 9409.29 300.9
16384 4 9391.96 405.17 9695.12 310.54 9423.76 223.47

Jason Wang (20):
  net: introduce qemu_get_queue()
  net: introduce qemu_get_nic()
  net: intorduce qemu_del_nic()
  net: introduce qemu_find_net_clients_except()
  net: introduce qemu_net_client_setup()
  net: introduce NetClientState destructor
  net: multiqueue support
  tap: import linux multiqueue constants
  tap: factor out common tap initialization
  tap: add Linux multiqueue support
  tap: support enabling or disabling a queue
  tap: introduce a helper to get the name of an interface
  tap: multiqueue support
  vhost: multiqueue support
  virtio: introduce virtio_del_queue()
  virtio: add a queue_index to VirtQueue
  virtio-net: separate virtqueue from VirtIONet
  virtio-net: multiqueue support
  virtio-net: migration support for multiqueue
  virtio-net: compat multiqueue support

 hw/cadence_gem.c            |   17 +-
 hw/dp8393x.c                |   17 +-
 hw/e1000.c                  |   32 ++--
 hw/eepro100.c               |   18 +-
 hw/etraxfs_eth.c            |   10 +-
 hw/lan9118.c                |   16 +-
 hw/lance.c                  |    2 +-
 hw/mcf_fec.c                |   12 +-
 hw/milkymist-minimac2.c     |   10 +-
 hw/mipsnet.c                |   10 +-
 hw/musicpal.c               |    6 +-
 hw/ne2000-isa.c             |    4 +-
 hw/ne2000.c                 |   13 +-
 hw/opencores_eth.c          |   12 +-
 hw/pc_piix.c                |    4 +
 hw/pcnet-pci.c              |    4 +-
 hw/pcnet.c                  |   13 +-
 hw/qdev-properties-system.c |   46 ++++-
 hw/qdev-properties.h        |    6 +-
 hw/rtl8139.c                |   22 +-
 hw/smc91c111.c              |   10 +-
 hw/spapr_llan.c             |    8 +-
 hw/stellaris_enet.c         |   11 +-
 hw/usb/dev-network.c        |   16 +-
 hw/vhost.c                  |   82 +++----
 hw/vhost.h                  |    2 +
 hw/vhost_net.c              |   86 +++++++-
 hw/vhost_net.h              |    4 +-
 hw/virtio-net.c             |  509 +++++++++++++++++++++++++++++++------------
 hw/virtio-net.h             |   28 +++-
 hw/virtio.c                 |   17 ++
 hw/virtio.h                 |    3 +
 hw/xen_nic.c                |   17 +-
 hw/xgmac.c                  |   10 +-
 hw/xilinx_axienet.c         |   10 +-
 hw/xilinx_ethlite.c         |   10 +-
 include/net/net.h           |   26 ++-
 include/net/tap.h           |    2 +
 net/net.c                   |  198 +++++++++++++----
 net/tap-aix.c               |   19 ++-
 net/tap-bsd.c               |   18 ++-
 net/tap-haiku.c             |   18 ++-
 net/tap-linux.c             |   67 ++++++-
 net/tap-linux.h             |    4 +
 net/tap-solaris.c           |   18 ++-
 net/tap-win32.c             |   10 +
 net/tap.c                   |  295 ++++++++++++++++++-------
 net/tap_int.h               |    6 +-
 qapi-schema.json            |    5 +-
 savevm.c                    |    2 +-
 50 files changed, 1303 insertions(+), 482 deletions(-)