Re: [Qemu-devel] Fwd: Re: Tunneled Migration with Non-Shared Storage

[Dr. David Alan Gilbert]

* Gary R Hook (address@hidden) wrote:
> Ugh, I wish I could teach Thunderbird to understand how to reply to a
> newsgroup.
> 
> Apologies to Paolo for the direct note.
> 
> On 11/19/14 4:19 AM, Paolo Bonzini wrote:
> >
> >
> >On 19/11/2014 10:35, Dr. David Alan Gilbert wrote:
> >>* Paolo Bonzini (address@hidden) wrote:
> >>>
> >>>
> >>>On 18/11/2014 21:28, Dr. David Alan Gilbert wrote:
> >>>>This seems odd, since as far as I know the tunneling code is quite 
> >>>>separate
> >>>>to the migration code; I thought the only thing that the migration
> >>>>code sees different is the file descriptors it gets past.
> >>>>(Having said that, again I don't know storage stuff, so if this
> >>>>is a storage special there may be something there...)
> >>>
> >>>Tunnelled migration uses the old block-migration.c code.  Non-tunnelled
> >>>migration uses the NBD server and block/mirror.c.
> >>
> >>OK, that explains that.  Is that because the tunneling code can't
> >>deal with tunneling the NBD server connection?
> >>
> >>>The main problem with
> >>>the old code is that uses a possibly unbounded amount of memory in
> >>>mig_save_device_dirty and can have huge jitter if any serious workload
> >>>is running in the guest.
> >>
> >>So that's sending dirty blocks iteratively? Not that I can see
> >>when the allocations get freed; but is the amount allocated there
> >>related to total disk size (as Gary suggested) or to the amount
> >>of dirty blocks?
> >
> >It should be related to the maximum rate limit (which can be set to
> >arbitrarily high values, however).
> 
> This makes no sense. The code in block_save_iterate() specifically
> attempts to control the rate of transfer. But when
> qemu_file_get_rate_limit() returns a number like 922337203685372723
> (0xCCCCCCCCCCB3333) I'm under the impression that no bandwidth
> constraints are being imposed at this layer. Why, then, would that
> transfer be occurring at 20MB/sec (simple, under-utilized 1 gigE
> connection) with no clear bottleneck in CPU or network? What other
> relation might exist?

Disk IO on the disk that you're trying to transfer?

> >The reads are started, then the ones that are ready are sent and the
> >blocks are freed in flush_blks.  The jitter happens when the guest reads
> >a lot but only writes a few blocks.  In that case, the bdrv_drain_all in
> >mig_save_device_dirty can be called relatively often and it can be
> >expensive because it also waits for all guest-initiated reads to complete.
> 
> Pardon my ignorance, but this does not match my observations. What I am
> seeing is the process size of the source qemu grow steadily until the
> COR completes; during this time the backing file on the destination
> system does not change/grow at all, which implies that no blocks are
> being transferred. (I have tested this with a 25GB VM disk, and larger;
> no network activity occurs during this period.) Once the COR is done and
> the in-memory copy ready (marked by a "Completed 100%" message from
> blk_mig_save_builked_block()) the transfer begins. At an abysmally slow
> rate, I'll add, per the above. Another problem to be investigated.

Odd thought; can you try dropping your migration bandwidth limit
(migrate_set_speed) - try something low, like 10M - does the behaviour
stay the same, or does it start transmitting disk data before it's read
the lot?

> >The bulk phase is similar, just with different functions (the reads are
> >done in mig_save_device_bulk).  With a high rate limit, the total
> >allocated memory can reach a few gigabytes indeed.
> 
> Much, much more than that. It's definitely dependent upon the disk file
> size. Tiny VM disks are a nit; big VM disks are a problem.

Well, if as you say it's not starting transmitting for some reason until
it's read the lot then that would make sense.

> >Depending on the scenario, a possible disadvantage of NBD migration is
> >that it can only throttle each disk separately, while the old code will
> >apply a single limit to all migrations.
> 
> How about no throttling at all? And just to be very clear, the goal is
> fast (NBD-based) migrations of VMs using non-shared storage over an
> encrypted channel. Safest, worst-case scenario. Aside from gaining an
> understanding of this code.

There are vague plans to add TLS support for encrypting these streams
internally to qemu; but they're just thoughts at the moment.

> Thank you for your attention.

Dave

> 
> -- 
> Gary R Hook
> Senior Kernel Engineer
> NIMBOXX, Inc
> 
> 
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK