Re: [Qemu-devel] Image probing: how it can be insecure, and what we could do about it

[Markus Armbruster]

Kevin Wolf <address@hidden> writes:

> Am 04.11.2014 um 19:45 hat Markus Armbruster geschrieben:
>> I'll try to explain all solutions fairly.  Isn't easy when you're as
>> biased towards one of them as I am.  Please bear with me.
>> 
>> 
>> = The trust boundary between image contents and meta-data =
>> 
>> A disk image consists of image contents and meta-data.
>> 
>> Example: all of a raw image's contents is image contents.  Leaves just
>> file name and attributes for meta-data.
>
> Better: Leaves only protocol-specific metadata (e.g. file name and
> attributes for raw-posix).

Can you give examples for other protocols?

>> Example: QCOW2 meta-data includes header, header extensions, L1 table,
>> L2 tables, ...  The meta-data defines where in the image the actual
>> contents is stored.
>> 
>> A guest can access the image contents, not the meta-data.
>> 
>> Image contents you've let an untrusted guest write is untrusted.
>> 
>> Therefore, there's a trust boundary between image contents and
>> meta-data.  QEMU has to trust image meta-data, but shouldn't trust image
>> contents.  The exact location of the trust boundary depends on the image
>> format.
>
> Trust metadata to a certain degree - the block layer audit was started
> because we noticed that it might not be all that trustworthy in
> practice. Different problem, though, it just shows that there are hardly
> any clear borders between "always completely trusted" and "always
> completely untrusted".

Yes.  Similar point made by Dave, I think.

Say I receive a QCOW2 image from an untrusted source.  Things I'd like
to be able to do securely with it:

* Examine it with qemu-img info.

* Open it with BDRV_O_NO_BACKING.

* If the backing file is innocent, open it without BDRV_O_NO_BACKING.

Assuming I do all this with trusted tools, of course.

However, here I talk about something else: the trust boundary between
image contents and meta-data.

>> = How we instruct QEMU what to trust =
>> 
>> By configuring QEMU to use an image, the user instructs QEMU to trust
>> the image's meta-data.
>> 
>> When the user's configuration specifies the image format explicitly, the
>> trust boundary is clear.
>> 
>> Else, the trust boundary is ambigous when more than one format is
>> possible.
>> 
>> QEMU resolves this ambiguity by picking the first format with the
>> highest "score".  Raw format is always possible, and always has the
>> lowest score.
>
> You used the term "untrusted guest" before. Are there any trusted guests,
> or should we assume that guests are untrusted by definition?

I'm trying to allow for users saying "guest subverting this trust
boundary is not an issue for me, now get off my lawn!"

> If you use virtualisation for isolation, then the answer is probably
> that guests are always untrusted. Other users may know exactly what
> their guest is doing and are using qemu for other reasons. The former
> would probably want to disable probing completely, the latter don't care
> about it and prefer convenience.

Yes.

The third group is unsophisticated users blissfully unaware of the
problem.

> My guess is that the share of those with trusted guests is higher among
> direct qemu users than libvirt users, but it's just that, a guess. It
> also doesn't mean that they are the majority of direct qemu users (they
> might be, but I honestly don't know).

Me neither.

If secure usage was easy, it wouldn't really matter.

> If there are trusted and untrusted guests, does this section need some
> thoughts about instructing qemu whether to trust the guest or not?

I don't know.

>> = How this lets the guest escape isolation =
>> 
>> Unfortunately, this lets the guest shift the trust boundary and escape
>> isolation, as follows:
>> 
>> * Expose a raw image to the guest (whether you specify the format=raw or
>>   let QEMU guess it doesn't matter).  The complete contents becomes
>>   untrusted.
>> 
>> * Reuse the image *without* specifying the raw format.  QEMU guesses the
>>   format based on untrusted image contents.  Now QEMU guesses a format
>>   chosen by the guest, with meta-data chosen by the guest.  By
>>   controlling image meta-data, the malicious guest can access arbitrary
>>   files as QEMU, enlarge its storage, and more.  A non-malicious guest
>>   can accidentally DoS itself, by writing a pattern probing recognizes.
>> 
>> This is CVE-2008-2004.
>> 
>> 
>> = Aside: other trust boundaries =
>> 
>> Of course, this is not the only trust boundary that matters.  For
>> instance, there's normally one between your host and somebody else's
>> computers.  Telling QEMU to trust meta-data of some image you got "from
>> the internet" violates it.  There's nothing QEMU can do about that.
>
> Okay, this addresses what I commented above.

Good :)

>> = Insecure usage is easy, secure usage is hard =
>> 
>> The oldest stratum of user interfaces doesn't let you specify the image
>> format.  Use of raw images with these is insecure by design.  These
>> interfaces are still recommended for human users.
>> 
>> Example of insecure usage: -hda foo.img, where foo.img is raw.
>> 
>> With the next generation of interfaces, specifying the image format is
>> optional.  Use of raw images with these is insecure by default.
>> 
>> Example of insecure usage: -drive file=foo.img,index=0,media=cdrom,
>> where foo.img is raw.  The -hda above is actually sugar for this.
>> 
>> Equivalent secure usage: add format=raw.
>> 
>> Note that specifying just the top image's format is not enough, you also
>> have to specify any backing images' formats.  QCOW2 can optionally store
>> the backing image format in the image.  The other COW formats can't.
>> 
>> Example of insecure usage: -hda bar.vmdk, where bar.vmdk is a VMDK image
>> with a raw backing file.
>
> Usually this is mitigated by the fact that backing files are read-only.
> Trouble is starting when you use things like commit.

Yes.

>> Equivalent secure usage: Beats me.  Maybe there's a funky -drive
>> backing.whatever to specify the backing image's format.
>
> Yes, you can override the backing file driver (backing.driver=raw should
> do the trick). Not really user-friendly, especially with long backing
> file chains, but it happens to be there.
>
> And of course, libvirt should be using it for non-qcow2 or qcow2 without
> the backing format header extension (but doesn't yet).

I'm glad it's there.  Too bad libvirt doesn't use it, yet.  Supports my
point that secure usage is too hard now.

>> With the latest interface blockdev-add, specifying the format is
>> mandatory.  Secure, but not really suitable for humans.
>> 
>> Example of secure usage:
>> 
>>     { "execute": "blockdev-add",
>>       "arguments": {'options': {
>>           'driver': 'raw', 'id':'foo',
>>           'file': { 'driver': 'file', 'filename': 'foo.img' } } } }
>
> Backing file probing can still happen with blockdev-add. As for secure
> usage, -drive should be equivalent with blockdev-add.

More support for my point that secure usage is too hard now: even I
can't do it.

>> Insecure usage is easy, secure usage is *hard*.  Even for sophisticated
>> users like libvirt developers.  Evidence: libvirt CVE-2010-2237,
>> CVE-2010-2238, CVE-2010-2239, and more that didn't get a CVE, like the
>> recent accidental probing in drive-mirror.
>> 
>> 
>> = How can we better guard the trust boundary in QEMU? =
>> 
>> The guest can violate the trust boundary only because
>> 
>> (a) QEMU supports both raw images and image formats, and
>> 
>> (b) QEMU guesses image format from raw image contents, and
>> 
>> (c) given a raw image, guests can change its contents and control a
>> future QEMU's format guess.
>> 
>> We can attack one ore more of these three conditions:
>> 
>> (a) Outlaw raw images
>> 
>> (b) Don't guess format from untrusted image contents
>> 
>> (c) Prevent "bad" guest writes
>> 
>> Nobody is seriously suggesting we do (a).  It's clearly too late for
>> that.  Let's explore the other two in more detail.
>> 
>> == Don't guess format from untrusted image contents ==
>> 
>> Several variations of the theme.
>> 
>> Guessing only happens when the user doesn't specify a format, so the
>> simplest way to avoid it would be requiring users to always specify the
>> format.
>> 
>> PRO: Simple, plugs the hole.
>> 
>> CON: Breaks a lot of existing usage.  Bye -hda, hello extra typing.
>
> Not only extra typing, but affects existing scripts, too.

Yes, that's "breaks a lot of existing usage".

>> Variation: command line option to opt out of probing completely.
>> 
>> PRO: Simple, plugs the hole.
>> 
>> CON: Insecure by default.
>> 
>> CON: In addition to opting out, you have to update your usage
>> accordingly.  I guess libvirt would do it anyway, to guard against
>> accidental probing once and for all.
>
> This seems to come back to my comment above whether we need to tell qemu
> whether the guest is trusted or not. It's an interesting thought that we
> haven't discussed enough yet.

Yes.

>> Variation: Stefan proposed to make format= mandatory just for -drive.  I
>> guess he rather meant mandatory for anything but -hda and other selected
>> convenience interfaces.
>> 
>> PRO: No extra typing in the cases where it makes the most difference.
>> 
>> CON: Breaks existing usage in the other cases, extra typing.
>> 
>> CON: Doesn't fully plug the hole.  Users of convenience interfaces may
>>     remain insecure out of ignorance.  We could add a warning to guide
>>     them to more secure usage, but then that warning would annoy users
>>     who don't care for security (sometimes with reason), and we can't
>>     have that.  So we silently leave the users who would care if they
>>     knew insecure.
>
> I do think that we should allow those with trusted guests to use a
> convenient interface, but their being annoyed by a warning shouldn't be
> a reason not to print warnings. Ignoring the warning isn't any
> considerable extra work.

I stand corrected on warnings.

CON: Doesn't fully plug the hole.  Users of convenience interfaces may
    remain insecure out of ignorance.  We could add a warning to guide
    them to more secure usage.

>> I proposed something less radical, namely to keep guessing the image
>> format, but base the guess on trusted meta-data only: file name and
>> attributes.  Block and character special files are raw.  For other
>> files, find the file name extension, and look up the format claiming it.
>> 
>> PRO: Plugs the hole.
>> 
>> CON: Breaks existing usage when the new guess differs from the old
>>     guess.  Common usage should be fine:
>> 
>>     * -hda test.qcow2
>> 
>>       Fine as long as test.qcow2 is really QCOW2 (as it should!), and
>>       either specifies a backing format (as it arguably should), or the
>>       backing file name is sane.
>> 
>>     * -hda disk.img
>> 
>>       Fine as long as disk.img is really a disk image (as it should).
>
> .img is not as clear, I've seen people using it for other formats. It's
> still a disk image, but not a raw one.

Is this usage common?

>>     * -hda /dev/mapper/vg0-virtdisk
>> 
>>       Fine as long as the logical volume is raw.
>> 
>>     Less common usage can break:
>> 
>>     * -hda nbd://localhost
>> 
>>       Socket provides no clue, so no guess.
>> 
>>     Weird usage can conceivably break hard:
>> 
>>     * -hdd disk.img
>> 
>>       Breaks hard when disk.img is actually QCOW2, the guest boots
>>       anyway from another drive, then proceeds to overwrite this one.
>> 
>> Mitigation: lengthy transition period where we warn "this usage is
>> insecure, and we'll eventually break it; here's a hint on secure usage".
>> 
>> CON: We delay plugging the hole one more time.  But at least we no
>> longer expose our users to it silently.
>
> CON: Relies on metadata that is protocol-specific. Each protocol that
>      should support probing needs extra code. Essentially means that
>      probing will be disabled on anything except raw-posix (and if we're
>      lucky enough that someone pays attention during review, raw-win32)

Terminology: I use "probing" and "guessing from trusted meta-data".  The
former is for probing raw image contents.  The latter may only examine
trusted meta-data.

I suspect you mean "each protocol that should support guessing from
trusted meta-data needs extra code".  Do you?

>> Jeff pointed out that we want to retain probing in things like qemu-img
>> info.
>> 
>> Richard asked for a way for users to ask for insecure probing, say
>> format=insecure-probe.  I have no problem with giving users rope when
>> they ask for it.
>
> Should actually fall out naturally from my bdrv_open() refactorings.
> I'll need at least "probe-format" and "probe-protocol" for internal use,
> and there's nothing that stops us from allowing it in the schema.
>
>> Variation: if file name and attributes are unavailable or provide no
>> clue, guess raw.  Same PRO and CON as above, only it avoids breaking a
>> few more cases.  For instance, "-hda nbd://localhost" keeps working as
>> long as the server serves a raw image.  Smells a bit like too much magic
>> to me.
>
> Can break in the same way as disk.img being qcow2. If we can tolerate
> one instance of the problem (can we?), we can probably tolerate the
> other one as well.
>
>> My proposal replaces probing wholesale.  I like that because it results
>> in simple, predictable guessing.  Here's a hybrid approach: first guess
>> raw vs. non-raw based on trusted meta-data only, and if non-raw, probe.
>> 
>> Nothing the guest writes can affect the raw vs. non-raw decision.  Once
>> an image is raw, only the user can make it non-raw, by changing its name
>> or attributes.
>> 
>> Two variations: 1. guess raw without a clue, and 2. guess non-raw then.
>> 
>> Again, same PRO and CON as above, only it doesn't break when users give
>> their non-raw images weird names.
>> 
>> == Prevent "bad" guest writes ==
>> 
>> Again, several variations, but this time, only the last one is serious,
>> the others are just for illustration.
>> 
>> Fail guest writes to those parts of the image that probing may examine
>> Can fail only writes to the first few sectors (at worst) of raw images.
>> 
>> PRO: Plugs the hole.
>> 
>> CON: The virtual hardware is defective.  Breaks common guest software
>> that writes to the first few sectors, such as boot loaders and
>> partitioning tools.  Breaks guest software using the whole device, which
>> isn't common, but certainly not unheard of.
>> 
>> Variation: fail only writes of patterns that actually can make probing
>> guess something other than raw.
>> 
>> PRO: Still plugs the hole.
>> 
>> CON: Except when you upgrade to a version that recognizes more patterns.
>> 
>> CON: The virtual hardware is still defective, but the defects are
>> minimized.  We can hope that partition tables, boot sectors and such
>> won't match the patterns, so common guest software hopefully works.
>> Guest software using the whole device still breaks, only now it breaks
>> later rather than sooner.
>> 
>> Variation: fail writes only on *probed* raw images.
>> 
>> CON: Doesn't fully plug the hole: mixing probed usage (user doesn't
>> specify format) with non-probed usage (user specifies format) remains
>> insecure.  The guest's write succeeds in non-probed usage, and the guest
>> escapes isolation in the next probed usage.
>> 
>> CON: The virtual hardware is still defective, but it now comes with a
>> "defective on/off" switch, factory default "defective on".  We could add
>> a warning to guide users to switch defective off but then that warning
>> would annoy people who don't care to switch it off (sometimes with
>> reason), and we can't have that.  So we leave users who would care if
>> they knew in the dark.

Replace by

CON: The virtual hardware is still defective, but it now comes with a
"defective on/off" switch, factory default "defective on".  We could add
a warning to guide users to switch defective off.

>> The two variations can be combined.  This is Kevin's proposal.
>> 
>> CON: Doesn't fully plug the hole: union of both variations' flaws.
>> 
>> CON: The virtual hardware is still defective: interesection of both
>> variations' defects.
>
> I like how you took care to avoid finding any PROs. :-)

Come on, this line 281 of 327, cut me some slack :)

> I'll leave commenting on this section to others for now. I feel I have
> already said enough about it in the other threads, and defending it here
> at this point wouldn't help the discussion.

The purpose of this document is to summarize our thoughts.  Need yours
to achieve it.  Can you give me your concise PROs?

>> = Conclusion =
>> 
>> This is *my* conclusion.  Yours may be different, and that's okay.  It's
>> business, not personal ;)
>> 
>> Secure usage should not be hard.
>> 
>> If we permit insecure usage for convenience or compatibility, we should
>> warn the user, unless he clearly asked for it.  Even if that warning
>> annoys Kevin and Max.
>
> It doesn't. I couldn't care less about a warning on my stderr.

Good.

> (Practically, though, it causes a bit of work because qemu-iotests runs
> into many of these warnings, changing the output. I have a patch that
> adds a -f to qemu-io, now I need to teach qemu-iotests to make use of
> it.)
>
>> If you want to suppress it with configure
>> --reckless or something, no objections.
>> 
>> Same for defective virtual hardware.
>> 
>> Kevin's proposal to prevent "bad" guest writes has relatively small
>> compatibility issues.  It improves things from "insecure by default" to
>> "slightly defective by default" as long as you use raw images either
>> always probed or always non-probed.  It doesn't help at all when you
>> alternate probed and non-probed usage.
>
> Why would you do that?

Carelessness?

Why should it be dangerous?

> And yes, it also doesn't help when you accidentally type format=qcow2
> instead of format=raw. When you have images of both types, things like
> this happen with manual typing.

I consider mixing probed and non-probed usage a more plausible bad habit
than accidental use of format=qcow2, because the former works just fine,
but the latter fails (unless your guest has "helpfully" written a QCOW2
header).

>> Improvement of sorts, but it
>> still fails my "secure usage should not be hard" requirement, and that
>> requirement is a hard one for me.
>> 
>> My proposal to ditch image contents probing entirely has more serious
>> compatibility issues.  In particular, we'd have to forgo sugared
>> convenience syntax for a number of less common things.  It definitely
>> needs a grace period where all usage we're going to break warns.  On the
>> up side, it will actually be secure by default when it's done.
>> 
>> If this is not acceptable, my second choice is actually the command line
>> option to opt out of probing completely.  This doesn't address "insecure
>> by default" (sadly), but it does at least satisfy my "secure usage
>> should not be hard" requirement.
>> 
>> If we should adopt Kevin's proposal against my objections, then I very
>> badly want the opt out option on top of it, opting out of both probing
>> and "bad" write prevention.
>
> This sounds quite reasonable. And opting out of probing automatically
> opts out of "bad write prevention" because it only happens on probed
> images.
>
> To take this though a bit further: Do we have similar problems in other
> areas in qemu? We could introduce a -[no-]strict option that forces
> the user to be explicit in all places and return errors if they aren't.
> I could even live with a configure switch that makes -strict the
> default, as long as that configure switch remains disabled by default.

Open-ended question.  Good one, though.  Ideas, anyone?