Re: [Qemu-devel] CPUID feature bits not saved with migration

[Jamie Lokier]

Andre Przywara wrote:
> Jamie Lokier wrote:
> >Anthony Liguori wrote:
> >>It's unclear what to do about -cpu host.  If we did migrate cpuid 
> >>values, then -cpu would effectively be ignored after an incoming 
> >>migration.
> >
> >The new host might not support all the cpuid features of the old host,
> >whether by -cpu host or explicit cpuid.  What happens then?
> 
> If you plan to migrate, you should think of this in advance.

In my experience so far, for small sites, you don't plan migration for
2-3 years after starting the VM, because it's only then you realise
your host hardware is quite old, you buy a replacement to consolidate,
and you are still running the VM that you didn't know would still be
mission critical years later.

At least, that's been my experience so far.  I've "cold migrated" a
few VMs, and in some cases from non-virtual machines to VMs.  None of
these could be planned when the guest was first installed, especially
the ones where it wasn't realised the guest would outlive the host
hardware.

I have to say, unfortunately hot migration has never been an option
because the version of KVM running on the new host is invariably
incompatible with the KVM running on the old host.

> I have a rough version of a tool to compute the least common
> denominator CPUID bits given either processor (code)names or host
> names. In the latter case it will login into the box and query the
> host's CPUID. The tool then generates a QEMU command line (like -cpu
> qemu64,-mwait,-popcnt) with which the guest should be started.  This
> should ensure that the guest always sees the same subset of the CPUs
> capabilities.

I wonder how that would be useful.  Don't you usually migrate only
when you've acquired new hardware, who's specs you don't know at the
time you'd want to compute the initial CPUID?

> >For changing cpuid when migrating, as you might like to do with -cpu
> >host for performance, is reboot-during-migrate useful?  It would make
> >sure all disk state is committed to the image files asynchronously
> >while the machine continues to run (just like normal migration), and
> >at the last moment transfers control and the machine sees a reboot,
> >permitting devices changes including cpuid change.
>
> Is that really useful? After all the sexy part of live migration is the 
> "live" component...

The other sexy part is being easy and asynchronous: not stopping the
guest for a long time during the migration.

Easy: Now, at the moment you have to give all the right guest
configuration on the destination command line, so I take your point.

But if guest configuration is ever included in the saved state for
migration, migration will really easy.  I hope it's just as easy to do
"cold migration".

Async: Do we save RAM state across reboots normally?  I know of OSes
which expect at least some part of RAM to survive reboots, so killing
the VM and restarting on another host would change the behaviour,
compared with rebooting locally; that's not transparent migration,
it's a subtle, unexpected behaviour change.  Unfortunately doing the
right thing involves savev, which pauses the guests for a long time.
The pause comes from saving and loading RAM, something which migration
handles well.

There's also the small matter of migration having a totally different
interface compared with savevm right now, with savevm requiring a
dummy qcow2 disk while migration transfers control across a network
with no temporary file.

Guess which one is nicer for the user wanting "move my VM to host FOO
(which doesn't support SSE4) with minimal downtime".

> >CPU hotplug could be used for cpuid change in theory, but I doubt if
> >any guests or guest apps would handle it well.
> Hotplugging could work for secondary processors, but hotplugging the BSP 
> is kind of tricky. And this does not solve the userspace issues, where 
> libraries detect CPU capabilities during startup and use optimized code 
> paths. AFAIK there is no mechanism of informing those libraries about a 
> CPUID change.

I agree, and it's pointless to spend much time discussion hotplug for this.
Most guests wouldn't handle CPUs with mixed CPUIDs anyway.

In theory, sometimes it'd be ok to push that problem to the user: they
can stop and start specific apps under user control without bringing
down a whole machine, and most apps don't use cpuid-dependent
features, especially on servers.

Btw, why can't hotplugging the main processor work?  For (real)
high-reliability systems, all processors are hotpluggable, afaik.

-- Jamie