[Qemu-devel] Re: Live migration protocol, device features, ABIs and other beasts

[Juan Quintela]

Dor Laor <address@hidden> wrote:
> In the last couple of days we discovered some issues regarding stable
> ABI and the robustness of the live migration protocol. Let's just jump
> right into it, ordered by complexity:
>
> 1. Control *every* feature exposed to the guest by qemu cmdline:
>
>    While thinking on cross version migration, and reviewing some
>    patches, I noticed that there are many times that we use feature bits
>    in order to expose functionality for the guest driver - example:
>    VIRTIO_BLK_F_BARRIER, but we do not control it from qemu cmdline.

In my opinion this is madness, qemu command line is already too
complicated.  I agree with anthony to put it in the command line.
I will go further, and think that this kind of issues should be put into
the machine type.

If you start qemu with -M pc-0.10, it should save the state in a 0.10
compatible way (that don't happens at the moment, but it should work
that way).

>    The solution here is simpler - Every guest visible change should
>    translate into cmdline option. This is part of the machine type and
>    in addition should be configurable.
>    It's an issue we all should keep in the back of our heads and popup
>    when a new capability/change are introduced.

I think this creates again a exponential posibilities, when we are only
interested in some small part of the combinations.

Upstream: It only cares about what was is 0.10, 0.11 and 0.12, three
combinations, not all possible combinations of all devices.

Downstream: (we for instance) care about RHEL5.4 RHEL5.5 or similar, not
all other possible combinations.

Notice that this is important, we change devices constantly, but we need
new machine types ... once each X months, i.e. they are much less.

> 2. Live migration inherent problem.
>
>    Currently, even with VMState, the protocol is not flexible enough.
>    We run into problem when we needed to fix pvclock migration issue.
>    The fix included 2 additional fields in save/load state and thus
>    needed a new version number.
>    The trouble is that the load function does not accept sections with
>    versions greater than the one it supports.

This is a feature :)
We need te _save function to decide if it can save in an old enough
format.  As we are now, there are no infrastructure of doing different
things on save, but we should be able.

>    IMHO we should 'specify' the migration protocol and introduce
>    capabilities, feature bits, etc. This way we'll have a robust,
>    extensible protocol that will withstand any potential issue. Both
>    Michael Tisrkin and I suggest it at the time vmstate was introduced.
>    Vmstate is good for the code but it's not a protocol.

A very small machine, has:
- "ide"
- "cpu"
- "e1000"
- "acpi"
- "apic"
- "cirrus_vga"
- "usb"
- "pckbd"
- "ps2mouse"
- ....

(no machine has so few devices), took that each device only has 2
features/capabilities/..., that is: 2^9 combinations.  Start to think
about test cases for going forward <-> backward, etc.

Now change it again, and decide that you support 2 kind of machines:
- pc-0.10
- pc-0.11

(that is what you are going to be interested), and now, just in case, it
becames easy to know what things you need to support/test.

i.e. old acpi and new apic is not a valid combination, ...

> Which protocol should we use? You're smarter than me, please suggest
> one.
> wrt the above guest abi issue, we should write a qemu spec with clear
> definitions for devices, drivers, versions, etc.
>
> Looking forward interesting fruitful discussion,

My idea here is that we need to have further use of machine
descriptions, once that is done, we need something like a new property
for qdev (version?).  Once there, each device could do:
- if version != last_version -> die (what it happens now)
- do someting sensible, not use the "new" features not existing on that
  version
- edit the savevm format in an easy way.

With respect to VMState, we have the information of _when_ a field was
introduced.  With VMState it is not difficult to save state in an old
format  (we just need to add some infrastructure, but nothing too
dificult),  What is needed is to test/check that for each particular
device, saving the subset of fields is enough.  And this is a though job.

Later, Juan.