Re: Bitlocker-tripping hardware change with 5.2.0?

[Stefan Berger]

On 12/20/20 8:25 AM, Michael Weiser wrote:
> Hi Stefan,
>
> On Sat, Dec 19, 2020 at 11:02:29PM -0500, Stefan Berger wrote:
>
> > The whole purpose of measured/trusted boot is to reflect some known
> > measurement values of a known BIOS in the TPM PCRs. Unfortunately this bites
> > with sealing to those values and the rather fast development of QEMU.
> > Updates of QEMU on the host platform then become a recovery event inside the
> > VM, which is unfortunate, but this is when measured/trusted boot actually
> > 'does its job'.
> Thank you! This was a cruicial hint at exactly the right time: I was at
> a point of severe head-scratching where I had gotten git bisect to work
> but it was telling me that stock v5.1.0 compiled from git source also
> showed the problem while my system-installed version wasn't.
>
> The difference indeed is in the firmware blobs used: Compiled from
> source, qemu uses the firmware files in the pc-bios subdirectory of the
> build directory while my system's qemu looks into /usr/share/qemu where
> bios{,-256k}.bin and vgabios-qxl.bin are provided by a different package
> (seabios).
>
> With that knowledge, I was able to make the git-compiled v5.1.0 qemu
> behave the same as the one installed in the system by providing -L
> /usr/share/qemu to it.
>
> qemu v5.2.0 compiled from git source also does not drop into BitLocker
> recovery when pointed at the unchanged system-installed qemu-5.1.0
> /usr/share/qemu directory. This negates the suspicion that some change
> in qemu proper carries over into the VM.

Good to know...


> strace shows the following files being opened by qemu when starting my
> testing VM:
>
> bios.bin
> bios-256k.bin
> efi-e1000e.rom
> vgabios-qxl.bin
> kvmvapic.bin
>
> Since bios{,-256k}.bin and vgabios-qxl.bin are provided by the seabios
> package and therefore (in my particular case) stable across an update
> from qemu-5.1.0 to 5.2.0 and kvmvapic.bin is identical in 5.1.0 and
> 5.2.0, this left only efi-e1000e.rom as reason for Bitlocker recovery
> upon upgrade.
>
> And indeed, reverting efi-e1000e.rom to the 5.1.0 version after
> upgrading my system qemu made the Bitlocker recovery go away.


Ah, this is the option ROM that's being measured.


> Armed with that knowledge, I was able to provide qemu with a VM-specific
> firmware path from within libvirtd xml following documentation at
> https://libvirt.org/drvqemu.html#qemucommand:
>
> --- win10-bitlocker.xml.bak     2020-12-20 13:28:09.361141043 +0100
> +++ win10-bitlocker.xml 2020-12-20 13:29:22.098750216 +0100
> @@ -5,7 +5,7 @@
>   or other application using the libvirt API.
>   -->
>
> -<domain type='kvm'>
> +<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>     <name>win10-bitlocker</name>
>     <uuid>bf566263-35e3-4dba-af8c-8ca85dba6a85</uuid>
>     <metadata>
> @@ -141,4 +141,8 @@
>         <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
>       </memballoon>
>     </devices>
> +  <qemu:commandline>
> +    <qemu:arg value='-L'/>
> +    <qemu:arg 
> value='/home/user/.config/libvirt/qemu/firmware/bf566263-35e3-4dba-af8c-8ca85dba6a85'/>
> +  </qemu:commandline>
>   </domain>
>
> # ls -la 
> /home/user/.config/libvirt/qemu/firmware/bf566263-35e3-4dba-af8c-8ca85dba6a85
> total 680
> drwxr-xr-x 2 user group   4096 Dec 20 13:26 .
> drwxr-xr-x 3 user group   4096 Dec 20 13:25 ..
> -rw-r--r-- 1 user group 262144 Dec 20 13:25 bios-256k.bin
> -rw-r--r-- 1 user group 131072 Dec 20 13:25 bios.bin
> -rw-r--r-- 1 user group 240128 Dec 20 13:25 efi-e1000e.rom
> -rw-r--r-- 1 user group   9216 Dec 20 13:26 kvmvapic.bin
> -rw-r--r-- 1 user group  38912 Dec 20 13:25 vgabios-qxl.bin
>
> Assuming I keep the virtual hardware specification stable
> also (e.g. pc-q35-5.1) and don't add/remove devices, this should work
> almost indefinitly, shouldn't it?


'almost' yes, but there are other reasons for why bitlocker may go into 
recovery mode: 
https://docs.microsoft.com/en-us/windows/security/information-protection/bitlocker/bitlocker-recovery-guide-plan#what-causes-bitlocker-recovery

We will occasionally update the TPM 2 code via libtpms and that could 
qualify as a TPM firmware upgrade , which again may cause a recovery. 
Though the TPM 2 should be a trusted device with a trusted update 
process and what other choice does a user have than to accept the TPM 2 
firmware update?


> > >      For giggles I did enter the recovery key of the testing VM when
> > >      prompted. It did boot up and showed Bitlocker enabled. After rebooting
> > >      it prompted for the recovery key again. I entered it again, it booted
> > >      again and I turned off Bitlocker (decrypted the disk). After
> > >      re-enabling
> > >      Bitlocker (re-encrypting the disk) and rebooting again it now does not
> > >      prompt for the recovery key again.
> > Does Bitlocker not allow you to accept a configuration change (= PCR value
> > change) and it internally (presumably) then seals the secreted against those
> > new values so upon the next reboot it works again? Do you really need to go
> > through the whole re-encryption process? It sounds like unattractive even
> > for a physical machine firmware update...
> Indeed, upon retesting multiple times it seems that entering the
> recovery key does actually also accept the firmware change. So there
> must have been some user error on my part on the first try.


Also good to know. But it's a pain...

> On closer look there is also the option of suspending Bitlocker
> protection for a number of boots[1]. The data stays encrypted but
> encryption keys are made available to the bootloader in the clear. Upon
> resumption, Bitlocker silently re-measures the system and thus accepts
> changes. When using the Control Panel applet[2], there's no option to
> enter the number of boots and it's reactivated immediately after the
> next boot.
>
> I did a quick test of that: A change is accepted on the next boot and
> recovery stays silent on following boots but an additional change at a
> later time drops me back into recovery again. So it seems to work as
> advertised.
>
> So, suspending Bitlocker in every "bitlockered" VM before a qemu
> upgrade, the user should never see any recovery screen. I would be
> certain to forget about that every second time and end up needing to
> downgrade qemu (and in my case possibly also the seabios package) again.

That's when you know when the updates are happening but in a cloud where 
the infrastructure admins may not communicate all their actions to the 
tenants that may become an 'uncoordinated event'.

Thanks for reporting the issue and trying all these things!


   Stefan

> [1] 
> https://docs.microsoft.com/en-us/powershell/module/bitlocker/suspend-bitlocker
> [2] 
> https://www.top-password.com/blog/wp-content/uploads/2019/02/suspend-bitlocker-protection.png
>
> @Marc-Andre: Re: git bisect: It seems "git bisect v5.2.0 v5.1.0" does
> something different than "git checkout v5.2.0 && git bisect start && git
> bisect bad && git bisect good v5.1.0". With the latter command sequence
> (and switching back to an out-of-source build), git bisect worked just
> fine, including parallel build. I'm not too happy with that explanation
> myself and would rather write this off to some additional layer 8
> problem in the process.