Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Paolo Bonzini]

Il 19/07/2014 10:43, Alex Bligh ha scritto:
> Paolo,
> 
> On 19 Jul 2014, at 08:30, Paolo Bonzini <address@hidden> wrote:
> 
>> Il 19/07/2014 09:10, Alex Bligh ha scritto:
>>>>> Looks like your source and destination machines have different
>>>>> iPXE ROMs.  This could be a packaging problem in your distro.
>>>
>>> This is Ubuntu 12.04 to Ubuntu 14.04. I think the 14.04 one is
>>> correct, and don't want to run with a non-standard qemu there.
>>
>> No, the old one is _always_ the correct one.
>>
>> The pxe-* ROMs must be 128k, the efi-* ROMs must be 256k.
> 
> On 12.04
> # ls -la /usr/share/qemu/pxe-virtio.rom
> -rw-r--r-- 1 root root 63488 Jun 12 23:23 /usr/share/qemu/pxe-virtio.rom
> 
> On 14.04 (after resolving a pile of symlinks)
> # ls -la /usr/lib/ipxe/qemu/pxe-virtio.rom
> -rw-r--r-- 1 root root 83968 Jan  6  2014 /usr/lib/ipxe/qemu/pxe-virtio.rom
> 
> I'm guessing the ROM size gets rounded up to the next power of 2, hence
> the ROM size on 12.04 is 64k, and on 14.04 is 128k, right?
> 
> So it would appear that if the length is actually coming from the
> length of the file on disk, that on 12.04 the pxe-ROM is not 128k
> but 64k. So if 'pxe-* ROMs must be 128k' is correct, I'm not sure
> that's incompatible with  'the old one is _always_ the correct
> one'.

Sorry, pxe-virtio.rom must be 64k.  It's pxe-e1000.rom that is 128k
because it exceeds 64k by a little:

-rw-rw-r-- 1 pbonzini users   67072  2 dic  2013 pxe-e1000.rom
-rw-rw-r-- 1 pbonzini users   61440  2 dic  2013 pxe-eepro100.rom
-rw-rw-r-- 1 pbonzini users   61440  2 dic  2013 pxe-ne2k_pci.rom
-rw-rw-r-- 1 pbonzini users   61440  2 dic  2013 pxe-pcnet.rom
-rw-rw-r-- 1 pbonzini users   61440  2 dic  2013 pxe-rtl8139.rom
-rw-rw-r-- 1 pbonzini users   60416  2 dic  2013 pxe-virtio.rom
-rw-r--r-- 1 pbonzini users  194560 16 lug 13.16 efi-e1000.rom
-rw-r--r-- 1 pbonzini users  196096 16 lug 13.16 efi-eepro100.rom
-rw-r--r-- 1 pbonzini users  194560 16 lug 13.16 efi-ne2k_pci.rom
-rw-r--r-- 1 pbonzini users  194560 16 lug 13.16 efi-pcnet.rom
-rw-r--r-- 1 pbonzini users  198144 16 lug 13.16 efi-rtl8139.rom
-rw-r--r-- 1 pbonzini users  192000 16 lug 13.16 efi-virtio.rom

> Whilst (per below) your workaround gets past the issue for this ROM,
> I'm confused about the numbers in the error message, as 64k and
> 128k do not equal 10,000 or 20,000:
>  Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000

These are hexadecimal.

>>> Can I use a different ROM (e.g. the old one) with a command line
>>> option just for the migrated machines? Obviously when they reboot,
>>> they'l get a hardware change (which is fine).
>>
>> Yes, "-global foo.romfile" on the destination QEMU.  You can just pass
>> an empty 128k file to the destination since ROM contents are migrated
>> properly, and hardware will only change when QEMU restarts.
> 
> Thanks.
> 
> So, copying the old ROM file across (out of an abundance of caution)
> plus the magic incantation:
> 
> -machine pc-1.0 -global cirrus-vga.vgamem_mb=10 -global 
> virtio-net-pci.romfile=pxe-virtio.rom.12.04

It's really 16 (again hex).

> would appear to be sufficient to permit the migration to get past the
> initial stages and commence the block migration.

Ok, at least the memory map is now the same.

> However, after the block migration is finished, I now see:
> 
> Receiving block device images
> Completed 100 %
> Unknown savevm section type 255
> load of migration failed

This could be another incompatibility between qemu-kvm and qemu.  I know
we had a couple in Fedora.

> My guess is this might be the bogus inclusion of PITCommonState
> per the last hunk of
>  
> http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20
> 
> Is there a technique to debug this sort of thing?

You can try using Amit's vmstate checker.

>> I don't know of any distro that actively tests migration except RHEL,
>> SLES.  I'm not sure about Debian.  Since there is no free-beer SLES,
>> you'd better switch to CentOS 7 than keeping Ubuntu 14.04.
> 
> Switching is not an option here; we have way too much other stuff dependent
> on Ubuntu.

You could try rebuilding the patched QEMU sources from CentOS.  CentOS
7's rhel6.1.0 or rhel6.2.0 machine types are comparable to pc-1.0, with
some luck they might even be compatible.

> Ubuntu is about to become tested (and a patch or workaround generated
> at least for migrations from 12.04 - the previous LTS version);
> whether or not it is upstreamed won't be up to me though.

Yes, because Ubuntu hardly introduces any new feature during the LTS
support period.  But in 2 years you will have to endure roughly the same
pain.

> To be fair, I think the issue is that they are supporting migration from
> stock qemu.git pc-1.0 (which is what I think they shipped with the
> subsequent non-LTS releases, which means they can't (easily) also
> support migration from qemu-kvm pc-1.0. That's the bit I want to fix.

Yes, Ubuntu didn't bother testing migration from 12.04 to non-LTS
releases or 14.04.  They don't have the manpower to do so anyway.

Paolo