Re: [PATCH v1] target/i386: kvm: Block migration when enfore_cpuid is set to false

[Daniel P . Berrangé]

On Thu, Jul 11, 2024 at 12:10:34PM +0000, Wang, Wei W wrote:
> On Thursday, July 11, 2024 7:48 PM, Daniel P. Berrangé wrote:
> > On Wed, Jul 03, 2024 at 10:49:12PM +0800, Wei Wang wrote:
> > > When enforce_cpuid is set to false, the guest is launched with a
> > > filtered set of features, meaning that unsupported features by the
> > > host are removed from the guest's vCPU model. This could cause issues for
> > live migration.
> > > For example, a guest on the source is running with features A and B.
> > > If the destination host does not support feature B, the stub guest can
> > > still be launched on the destination with feature A only if 
> > > enforce_cpuid=false.
> > > Live migration can start in this case, though it may fail later when
> > > the states of feature B are put to the destination side. This failure
> > > occurs in the late stage (i.e., stop&copy phase) of the migration
> > > flow, where the source guest has already been paused. Tests show that
> > > in such cases the source guest does not recover, and the destination
> > > is unable to resume to run.
> > >
> > > Make "enfore_cpuid=true" a hard requirement for a guest to be
> > > migratable, and change the default value of "enforce_cpuid" to true,
> > > making the guest vCPUs migratable by default. If the destination stub
> > > guest has inconsistent CPUIDs (i.e., destination host cannot support
> > > the features defined by the guest's vCPU model), it fails to boot
> > > (with enfore_cpuid=true by default), thereby preventing migration from
> > > occuring. If enfore_cpuid=false is explicitly added for the guest, the
> > > guest is deemed as non-migratable (via the migration blocker), so the
> > > above issue won't occur as the guest won't be migrated.
> > 
> > Blocking migration when enforce=false is making an assumption that users of
> > that setting are inherantly broken. This is NOT the case if the user/app has
> > already validated compatibility in some manner outside QEMU. Blocking
> > migration in this case will break valid working use cases.
> 
> It's just an enforcement to ensure a safe migration. Without this (i.e., the 
> current
> QEMU code) is making an assumption that users always have validated
> compatibility in a good manner outside QEMU, which is risky to some degree?

QEMU configurations must never be assumed to be migratable by default.
There is a huge set of things that a user must do with QEMU configuration
to guarantee migratability beyond CPU features. All aspects of guest HW
device topology must be set explicitly.

> Do you see how this would break valid working use cases (any examples)?
> This is actually what we are looking for. Please be aware that "enforce" is
> changed to be true by default to make the guest to be migratable by default
> under the enforcement.

Setting "enforce" will break existing use of QEMU. It is valid to launch
QEMU with a CPU model that is not fully supported by the host, allowing
QEMU to disable unsupported features automatically.

> > IMHO this patch doesn't need to exist. If users of QEMU want strong 
> > protection
> > they can already opt-in to that with enforce=true.
> 
> AFAIK, many users are not aware of this, and also we couldn't assume everybody
> knows it. That's why we want to add the enforcement.

Users who directly launch QEMU are expected to know about QEMU config
details for migration. If they don't, then they ought to be using a
higher level tool like libvirt, which ensures the configuration is
migration compatible.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|