Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids

[Juan Quintela]

"Jaggi, Manish" <address@hidden> wrote:
> Hi,

Hi

[Note that I was confused about what do you mean with problems with
processorID.  There is no processorID on the migration stream, so I
didn't understood what you were talking about.  Until I realized that
you were trying to migrate from different cpu types]

> Posting again with my cavium ID and CCing relevant folks

It will be good to give What architecture are we talking about?  MIPS,
ARM, anything else?

Why?  Because we do this continously on x86_64 world.  How do we do
this?  We emulate the _processor_ capabilities, so "in general" you can
always migrate from a processor to another with a superset of the
features.  If you look at the ouput of:

     qemu-system-x86_64 -cpu ?

You can see that we have lots of cpu types that we emulate and cpuid
(features really).  Migration intel<->amd is tricky.  But from "intel
with less features" to "intel with more features" (or the same with AMD)
it is a common thing to do.  Once told that, it is a lot of work, simple
things like that processors run at different clock speeds imply that you
need to be careful during migration with timers and anything that
depends on frequencies.

I don't know enough about other architectures to know how to do it, or
how feasible is.

> Live Migration between machines with different processorIds
>
> VM Migration between machines with different processorId values throws
> error in qemu/kvm. Though this check is appropriate but is overkill where
> two machines are of same SoC/arch family and have same core/gic but
> delta could be in other parts of Soc which have no effect on VM
> operation.

Then you need to do the whole process of:

Lets call both processors A1 and A2.  You need to do the whole process
of:

a- defining cpu A1
b- make sure that when you run qemu/kvm on processor A2, the
   features/behaviours that the guest sees.  This is not trivial at
   all.
c- when migration comes, you can see that you need to adjust to whatever
   is the architecture of the destination.

> There could be two ways to address this issue by ignoring the
> comparison of processorIDs and so need feedback from the
> community on this.
>
> a) Maintain a whitelist in qemu:
>
> This will be a set of all processorIds which are compatible and migration can
> happen between any of the machines with the Ids from this set. This set can
> be statically built within qemu binary.

In general, I preffer whitelists over blacklists.

> b) Provide an extra option with migrate command
>
>    migrate tcp:<ip>:<port>:<dest_processor_id>
>
> This is to fake the src_processor_id as dest_processor_id, so the qemu running
> on destination machine will not complain. The overhead with this approach is
> that the destination machines Id need to be known beforehand.

Please, don't even think about this:
a- migration commands are architecture agnostic
b- in general it is _much_, _much_ easier to fix things on destination
   that on source.

> If there is some better way… please suggest.

Look at how it is done on x86_64.  But be aware that "doing it right"
takes a lot of work.  To give you one idea:
- upstream, i.e. qemu, "warantee" that migration of:
  qemu-X -M machine-type-X -> qemu-Y -M machine-type-X
  works when X < Y.

- downstream (i.e. redhat on my case, but I am sure that others also
  "suffer" this)  allow also:

  qemu-Y -M machine-type-X -> qemu-X -M machine-type-X (Y > X)

  in general it is a very complicated problem, so we limit _what_ you
  can do.  Basically we only support our machine-types, do a lot of
  testing, and are very careful when we add new features.  I.e. be
  preparred to do a lot of testing and a lot of fixing.

I am sorry to not be able to tell you that this is an easy problem.

Later, Juan.