Re: [Qemu-devel] "Using Python to investigate EFI and ACPI"

[Laszlo Ersek]

On 09/03/15 16:50, Josh Triplett wrote:
> On Thu, Sep 03, 2015 at 11:16:40AM +0200, Laszlo Ersek wrote:
>> Hi Josh,
>>
>> I just found the LWN.net article with the title in the subject:
>>
>>   http://lwn.net/SubscriberLink/655992/04701db2bbb7e716/
>>
>> ("Thank you LWN", of course. :))
>>
>> This got me kind of excited. In your presentation, according to the
>> article, you were using KVM and OVMF to showcase the ACPI features of
>> "BITS". Further, in
>>
>>   http://biosbits.org/scripting/
> 
> (Note: that page is badly out of date, by a couple of years, and we need
> to drop it in favor of the various more recent presentations.  All of
> that functionality still exists, but we have an interactive Python REPL
> now, and the ACPI methods have moved to the "acpi" module and become
> easier to use.)
> 
>> you write, "You can also find arbitrary ACPI objects by path, and
>> evaluate arbitrary ACPI methods, with the return value decoded into a
>> Python object".
> 
> Right.  In particular, you can call acpi.get_objpaths to list the ACPI
> tree, and acpi.evaluate to evaluate an arbitrary method.
> 
>> I'm excited because for virtualization we might have the reverse use
>> case: use BITS to test / debug / develop the ACPI generator of QEMU. :)
> 
> Awesome.  Unit testing and CI for QEMU sounds like an ideal use case for
> BITS.
> 
>> As you may now, QEMU generates almost all of its ACPI tables
>> dynamically, providing the guest with an "ACPI payload" that describes
>> the actual virtual hardware as accurately as possible.
> 
> Many BIOSes do that, since the hardware configuration may vary at
> runtime.
> 
>> Then this payload is passed to the guest firmware (SeaBIOS or OVMF) over
>> "fw_cfg" (which is a simple protocol, comprising, at this point, one
>> selector and one data register, which are IO ports or MMIO locations --
>> see "docs/specs/fw_cfg.txt" in QEMU and
>> "Documentation/devicetree/bindings/arm/fw-cfg.txt" in the kernel).
> 
> Interesting; I hadn't seen that protocol before.
> 
> Do you virtualize those I/O ports by CPU, to make them thread-safe, or
> does the last address written to 0x510 get saved system-wide, making it
> unsafe for concurrent access?

I think fw_cfg is not meant to be accessed by several CPUs concurrently.
The protocol is stateful (selected key, offset within blob associated
with selected key, etc), and "accessing CPU" is not part of that state.

> 
>> The guest firmware downloads the "ACPI payload" over fw_cfg, does some
>> generic processing on them (we call this the "ACPI linker/loader
>> script"), and ultimately installs the ACPI tables in a way that is
>> appropriate for the firmware in question.
>>
>> To be more concrete, the tables you investigated in your presentation
>> with BITS were generated (very roughly) by the following files in QEMU:
>>
>>   hw/acpi/aml-build.c
>>   hw/acpi/bios-linker-loader.c
>>   hw/acpi/...
>>   hw/i386/acpi-build.c
>>
>> and processed and installed by the following source file in edk2:
>>
>>   OvmfPkg/AcpiPlatformDxe/QemuFwCfgAcpi.c
>>
>> With this background, you can probably see where I'm going with this. It
>> is not really easy to *test* the AML methods that QEMU generates
>> (piecing them together, with helper functions, from AML primitives),
>> without dedicated guest kernel drivers. I think the only method that I
>> know of is the following:
>>
>> - in the Linux guest, dump the ACPI tables with acpidump, or save them
>> from sysfs directly (/sys/firmware/acpi/tables)
>> - pass the DSDT and the SSDTs (and any data tables referenced by them?)
>> to AcpiExec from the ACPICA suite
>> - work with AcpiExec
>>
>> But, for simple testing, can we maybe run your tool within the guest,
>> before the runtime OS boots?
> 
> Yes, absolutely.  We have a batch-mode testing mechanism based on a
> config file; you'd probably want to make use of that.  With some
> extensions, it could dump results either to an emulated serial port or
> some other interface that you can read from outside qemu.  We also need
> to work on making the results more machine-parseable for automation.

While I certainly don't discount automation, my primary use case is
interactive development / testing. :) (Although, I can see myself
canning some commands in a script or config file, and invoking *that*
interactively. Which was your point, probably.)

> 
>> Although AcpiDump is being ported to edk2:
>>
>>   http://thread.gmane.org/gmane.comp.bios.edk2.devel/1109
>>
>> I'm unsure if it will be able to evaluate / execute AML methods. (If it
>> parallels the "acpidump" utility we have on GNU/Linux distros, it won't.)
> 
> Probably not.  And I haven't heard of any plans to port acpiexec to
> EDK2.
> 
>> Thus it would be awesome if we had some AcpiExec-like functionality
>> early on in the guest (for example in the form of a UEFI Shell
>> application, or as a python tool that runs within the edk2 Python port,
>> or even in grub).
>>
>> For example, assume your runtime guest OS is Windows (with its picky,
>> closed-source ACPI interpreter); you make a change in QEMU's ACPI
>> generator, rebuild QEMU, reboot the guest, drop to the UEFI shell to
>> verify the change "by eye", exit the shell, and *then* continue booting
>> Windows. (Which will hopefully not BSOD at that point, after the
>> verification with BITS / AcpiExec etc.)
>>
>> So, I have three questions:
>>
>> (1) What is the relationship between the ACPI facility of BITS, and ACPICA?
> 
> BITS links in ACPICA and uses it to evaluate ACPI.  We pull in ACPICA as
> a git submodule and build it as part of BITS.  acpi.evaluate uses the
> ACPICA interpreter.

Awesome! :)

Another question: when you execute an AML method that does, say, IO port
access, does the AML interpreter of ACPICA actually *perform* that IO
port access? Because, the one that is embedded in Linux obviously does,
and the one that is embedded in the userspace ACPICA command line
utility "acpiexec" obviously doesn't.

I assume (and very much hope) that the IO port access *is* performed
from BITS, simply because you developed it for physical machines, and it
wouldn't make much sense to avoid actual hardware access that was
implemented by the BIOS vendor for that platform.

If that is the case, then this tool could become the killer ACPI tester
for QEMU developers -- the hardware accesses in the AML methods
generated by QEMU would actually poke QEMU devices! (Unlike the
userspace "acpiexec" utility.) It would completely detach Linux guest
driver development from host side / firmware development. \o/

> 
>> (2) Is there a bit more comprehensive documentation about the ACPI
>> module of BITS? AcpiExec and the ACPICA Debugger have quite indulged me
>> with their incredible documentation (documents/acpica-reference.pdf). It
>> would be great if BITS' ACPI module had a list of commands, to see what
>> is there to play with.
> 
> We don't come close to the level of documentation for ACPICA, but we do
> have pydoc documentation for the modules in BITS, including acpi.  You
> can run help("acpi") within BITS, or read acpi.py.  We've tried to make
> sure all of the methods considered "API" have docstrings.

I'll have to digest this some, and play with it.

>> (3) I should mention that QEMU generates ACPI also for arm/aarch64
>> virtual machines ("hw/arm/virt-acpi-build.c" in QEMU), and the same edk2
>> module as noted above (built for arm/aarch64) does the guest side
>> processing. Do you think it's possible to use BITS in arm/aarch64 VMs?
> 
> Some folks from Linaro started looking into a BITS arm64 port.

Great!

>> ... I apologize if tools / documentation already exist for this kind of
>> development work; everyone please educate me then. I hope my questions
>> make at least some sense; I realize this email isn't well organized.
> 
> Makes perfect sense, and thanks for your mail!  I love the idea of using
> BITS to test qemu's own ACPI.

Thank you very much! :)

(I must say, I have found the LWN article at just the right time. I
intend to start implementing a VMGenID device for QEMU, and it's all
ACPI based. Here's our design for that:
<http://thread.gmane.org/gmane.comp.emulators.qemu/357940>. I've been
already dreading the need for a Linux guest driver, in order to
white-box test the device & the ACPI stuff from the guest side. :))

Cheers!
Laszlo