Re: [Qemu-devel] Performance Profiling 2 VMs

[kalyan tata]

Thanks a lot for the quick reply Stefan

Following from problem VM:

18:56:29     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s

18:56:44       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.20      0.00

18:56:49       1    0.00    0.00    0.00    0.00    3.21   10.22    0.00   79.56    908.22

18:56:54       1    0.00    0.00    0.00    0.00   11.47   54.93    0.00    2.82   5527.77

18:56:59       1    0.00    0.00    0.00    0.00   10.04   66.06    0.00    2.21   7160.64

18:57:04       1    0.00    0.00    0.00    0.00   10.42   65.13    0.00    2.00   7295.99

18:57:09       1    0.00    0.00    0.00    0.00   12.53   50.51    0.00    4.04   5700.20

18:57:14       1    0.00    0.00    0.00    0.00   16.43   65.53    0.00    8.62   9572.34

18:57:19       1    0.00    0.00    0.00    0.00   11.45   60.64    0.00    4.02   5798.19

18:57:24       1    0.00    0.00    0.00    0.00   11.45   81.33    0.00    0.80   6064.26

18:57:29       1    0.00    0.00    0.00    0.00    7.65   85.11    0.00    0.80   7578.27

18:57:34       1    0.00    0.00    0.00    0.00    9.42   84.17    0.00    1.40   9083.97

18:57:39       1    0.00    0.00    0.00    0.00    7.78   82.83    0.00    1.60   7264.87

18:57:44       1    0.00    0.00    0.00    0.00    8.62   87.78    0.00    0.60   8597.80

18:57:49       1    0.00    0.00    0.00    0.00   10.02   82.16    0.00    2.40   7750.90

18:57:54       1    0.00    0.00    0.00    0.00    8.42   81.76    0.00    1.00   6303.41

18:57:59       1    0.00    0.00    0.00    0.00    7.63   87.35    0.00    1.20   9422.49

18:58:04       1    0.00    0.00    0.00    0.00   10.44   80.32    0.00    2.21   7496.79

18:58:09       1    0.00    0.00    0.00    0.00    6.43   59.84    0.00   26.91   5019.28

18:58:14       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      1.00

18:58:19       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00

I set the affinity of both tx and rx interfaces to cpu 1 so just showing cpu1.

NAPI weight is 128 in this version, I changed to 64 just to see. This version of the code seems to be changing quota and budget (which i did not see in newer versions) I am thinking of playing around with that.

I also see that this version kicks for every packet on the tx side. 

Any other pointers would be really helpful.

Thanks

Kal

On Wed, Mar 2, 2016 at 3:28 AM, Stefan Hajnoczi <address@hidden> wrote:

On Tue, Mar 01, 2016 at 04:06:16PM -0800, kalyan tata wrote:
> Hi All,
>
> I am new to qemu development.
> Sorry If this is not the correct forum for this question, it would be great
> if you could direct me to correct forum.
>
> I am seeing very low virtio network throughput on an older (2.6.18) linux
> guest  vs another newer guest (3.10) both running on the same host. (same
> config 2 vcpus, no multi Q etc.)  I see very high CPU usage on the 2.6.18
> guest at very low network throughput and want to profile to find
> bottleneck.
>
> I tried to use "perf kvm" but the analysis  shows overhead as  max .25 %
> where as top in VM shows 100% cpu. (I used following as a guide
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Virtualization_Tuning_and_Optimization_Guide/index.html#sect-Virtualization_Tuning_Optimization_Guide-Monitoring_Tools-perf_kvm
> )
>
>      0.25%  :5235    [uhci_hcd]        [g] 0xffffffff80182236
>      0.24%  :5235    [uhci_hcd]        [g] 0xffffffff8018226a
>      0.23%  :5235    [virtio_ring]     [g] vring_new_virtqueue
>      0.20%  :5236    [uhci_hcd]        [g] 0xffffffff80182236
>      0.18%  :5236    [uhci_hcd]        [g] 0xffffffff8018226a
>      0.18%  :5235    [uhci_hcd]        [g] 0xffffffff8016f385
>      0.14%  :5236    [uhci_hcd]        [g] 0xffffffff802fbe0f
>      0.14%  :5235    [uhci_hcd]        [g] 0xffffffff8001161a
>      0.14%  :5235    [virtio_ring]     [g] virtqueue_is_broken
>
>
> My basic question is - Is there a way I can profile the older version of
> linux guest so i can see the bottleneck (where the guest is spending CPU
> cycles) My aim is to see if i can patch the older version in the critical
> path with improvements made in newer version

What is the output of "mpstat 5" in the guest and on the host?  mpstat
is part of the "sysstat" package.

mpstat is similar to vmstat but also shows "guest time" and "steal
time".  Both are relevant to virtualization and will help show which
component is using so much CPU time.

Stefan