Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids

[Jaggi, Manish]

> On 23-Aug-2018, at 7:59 PM, Juan Quintela <address@hidden> wrote:
> 
> External Email
> 
> "Jaggi, Manish" <address@hidden> wrote:
>> Hi,
> 
> Hi
> 
> [Note that I was confused about what do you mean with problems with
> processorID.  There is no processorID on the migration stream, so I
> didn't understood what you were talking about.  Until I realized that
> you were trying to migrate from different cpu types]
> 
>> Posting again with my cavium ID and CCing relevant folks
> 
> It will be good to give What architecture are we talking about?  MIPS,
> ARM, anything else?
> 
arm64

> Why?  Because we do this continously on x86_64 world.  How do we do
> this?  We emulate the _processor_ capabilities, so "in general" you can
> always migrate from a processor to another with a superset of the
> features.  If you look at the ouput of:
> 
>    qemu-system-x86_64 -cpu ?
> 
> You can see that we have lots of cpu types that we emulate and cpuid
> (features really).  Migration intel<->amd is tricky.  But from "intel
> with less features" to "intel with more features" (or the same with AMD)
> it is a common thing to do.  Once told that, it is a lot of work, simple
> things like that processors run at different clock speeds imply that you
> need to be careful during migration with timers and anything that
> depends on frequencies.
> 
> I don't know enough about other architectures to know how to do it, or
> how feasible is.

For arm64 qemu/kvm throws an error when processorID does not match. 
> 
>> Live Migration between machines with different processorIds
>> 
>> VM Migration between machines with different processorId values throws
>> error in qemu/kvm. Though this check is appropriate but is overkill where
>> two machines are of same SoC/arch family and have same core/gic but
>> delta could be in other parts of Soc which have no effect on VM
>> operation.
> 
> Then you need to do the whole process of:
> 
> Lets call both processors A1 and A2.  You need to do the whole process
> of:
> 
> a- defining cpu A1
> b- make sure that when you run qemu/kvm on processor A2, the
>  features/behaviours that the guest sees.  This is not trivial at
>  all.
> c- when migration comes, you can see that you need to adjust to whatever
>  is the architecture of the destination.
> 
>> There could be two ways to address this issue by ignoring the
>> comparison of processorIDs and so need feedback from the
>> community on this.
>> 
>> a) Maintain a whitelist in qemu:
>> 
>> This will be a set of all processorIds which are compatible and migration can
>> happen between any of the machines with the Ids from this set. This set can
>> be statically built within qemu binary.
> 
> In general, I preffer whitelists over blacklists.
> 
>> b) Provide an extra option with migrate command
>> 
>>  migrate tcp:<ip>:<port>:<dest_processor_id>
>> 
>> This is to fake the src_processor_id as dest_processor_id, so the qemu 
>> running
>> on destination machine will not complain. The overhead with this approach is
>> that the destination machines Id need to be known beforehand.
> 
> Please, don't even think about this:
> a- migration commands are architecture agnostic
> b- in general it is _much_, _much_ easier to fix things on destination
>  that on source.
> 
>> If there is some better way… please suggest.
> 
> Look at how it is done on x86_64.  But be aware that "doing it right"
> takes a lot of work.  To give you one idea:
> - upstream, i.e. qemu, "warantee" that migration of:
> qemu-X -M machine-type-X -> qemu-Y -M machine-type-X
> works when X < Y.
> 
> - downstream (i.e. redhat on my case, but I am sure that others also
> "suffer" this)  allow also:
> 
> qemu-Y -M machine-type-X -> qemu-X -M machine-type-X (Y > X)
> 
> in general it is a very complicated problem, so we limit _what_ you
> can do.  Basically we only support our machine-types, do a lot of
> testing, and are very careful when we add new features.  I.e. be
> preparred to do a lot of testing and a lot of fixing.

At this point I am targeting a simpler case where Machine A1 and A2 has a core 
from the same SoC family.
For example Cavium ThunderX2 Core incremental versions which has identical core 
and GIC and may have some errata fixes.
In that case Y=X since migration only takes care of PV devices.

In that case a whitelist could be an easier option?

How to provide the whitelist to qemu in a platform agnostic way?
- I will look into intel model as you have suggested, does intel keeps a 
whitelist or masks off some bits of processorID
How does intel does it

- is providing a -mirate-compat-whitelist <file> option for arm only looks good?
this option can be added in A1/A2 qemu command, so it would be upstream / 
downstream agnostic.
> 
> I am sorry to not be able to tell you that this is an easy problem.
> 
> Later, Juan.