Re: [Qemu-devel] live migration vs device assignment (motivation)

[Alexander Duyck]

On Tue, Dec 29, 2015 at 9:15 AM, Michael S. Tsirkin <address@hidden> wrote:
> On Tue, Dec 29, 2015 at 09:04:51AM -0800, Alexander Duyck wrote:
>> On Tue, Dec 29, 2015 at 8:46 AM, Michael S. Tsirkin <address@hidden> wrote:
>> > On Tue, Dec 29, 2015 at 01:42:14AM +0800, Lan, Tianyu wrote:
>> >>
>> >>
>> >> On 12/25/2015 8:11 PM, Michael S. Tsirkin wrote:
>> >> >As long as you keep up this vague talk about performance during
>> >> >migration, without even bothering with any measurements, this patchset
>> >> >will keep going nowhere.
>> >> >
>> >>
>> >> I measured network service downtime for "keep device alive"(RFC patch V1
>> >> presented) and "put down and up network interface"(RFC patch V2 presented)
>> >> during migration with some optimizations.
>> >>
>> >> The former is around 140ms and the later is around 240ms.
>> >>
>> >> My patchset relies on the maibox irq which doesn't work in the suspend 
>> >> state
>> >> and so can't get downtime for suspend/resume cases. Will try to get the
>> >> result later.
>> >
>> >
>> > Interesting. So you sare saying merely ifdown/ifup is 100ms?
>> > This does not sound reasonable.
>> > Is there a chance you are e.g. getting IP from dhcp?
>>
>>
>> Actually it wouldn't surprise me if that is due to a reset logic in
>> the driver.  For starters there is a 10 msec delay in the call
>> ixgbevf_reset_hw_vf which I believe is present to allow the PF time to
>> clear registers after the VF has requested a reset.  There is also a
>> 10 to 20 msec sleep in ixgbevf_down which occurs after the Rx queues
>> were disabled.  That is in addition to the fact that the function that
>> disables the queues does so serially and polls each queue until the
>> hardware acknowledges that the queues are actually disabled.  The
>> driver also does the serial enable with poll logic on re-enabling the
>> queues which likely doesn't help things.
>>
>> Really this driver is probably in need of a refactor to clean the
>> cruft out of the reset and initialization logic.  I suspect we have
>> far more delays than we really need and that is the source of much of
>> the slow down.
>
> For ifdown, why is there any need to reset the device at all?
> Is it so buffers can be reclaimed?
>

I believe it is mostly historical.  All the Intel drivers are derived
from e1000.  The e1000 has a 10ms sleep to allow outstanding PCI
transactions to complete before resetting and it looks like they ended
up inheriting that in the ixgbevf driver.  I suppose it does allow for
the buffers to be reclaimed which is something we may need, though the
VF driver should have already verified that it disabled the queues
when it was doing the polling on the bits being cleared in the
individual queue control registers.  Likely the 10ms sleep is
redundant as a result.

- Alex