Re: [PATCH 00/12] Introduce new acpi/smbios qtests using biosbits

[Ani Sinha]

On Mon, Jun 27, 2022 at 1:02 PM Ani Sinha <ani@anisinha.ca> wrote:
>
> On Mon, Jun 27, 2022 at 12:25 PM Ani Sinha <ani@anisinha.ca> wrote:
> >
> > On Mon, Jun 27, 2022 at 12:22 PM Ani Sinha <ani@anisinha.ca> wrote:
> > >
> > > On Mon, Jun 27, 2022 at 12:14 PM Ani Sinha <ani@anisinha.ca> wrote:
> > > >
> > > > Biosbits is a software written by Josh Triplett that can be downloaded 
> > > > by
> > > > visiting https://biosbits.org/. The github codebase can be found here:
> > > > https://github.com/biosbits/bits/tree/master. It is a software that 
> > > > exercizes
> > > > the bios components such as acpi and smbios tables directly through 
> > > > acpica
> > > > bios interpreter (a freely available C based library written by Intel,
> > > > downloadable from https://acpica.org/ and is included with biosbits) 
> > > > without an
> > > > operating system getting involved in between.
> > > > There are several advantages to directly testing the bios in a real 
> > > > physical
> > > > machine or VM as opposed to indirectly discovering bios issues through 
> > > > the
> > > > operating system. For one thing, the OSes tend to hide bios problems 
> > > > from the
> > > > end user. The other is that we have more control of what we wanted to 
> > > > test
> > > > and how by directly using acpica interpreter on top of the bios on a 
> > > > running
> > > > system. More details on the inspiration for developing biosbits and its 
> > > > real
> > > > life uses can be found in (a) and (b).
> > > > This patchset contains QEMU qtests written in python that exercizes the 
> > > > QEMU
> > > > bios components using biosbits and reports test failures.
> > > >
> > > > Details of each of the files added by this patchset are provided in the 
> > > > README
> > > > file which is part of Patch 11. Every effort to contact Josh, through 
> > > > various
> > > > means including email, twitter, linkedIn etc has failed. Hence, the 
> > > > changes to
> > > > build biosbits with the newer compiler, upgrade acpica and other 
> > > > changes are
> > > > currently maintained in a forked project in my personal github. We may 
> > > > want to
> > > > maintain bits in a separate fork in a stable repository that is 
> > > > accessible by
> > > > QEMU developers.
> > > >
> > > > The newly introduced qtest currently only run for x86_64 platform. They 
> > > > pass
> > > > both when running make check on a baremetal box as well as from inside 
> > > > a vm.
> > > >
> > > > Thanks to Igor M for pointing me to this work.
> > > >
> > > > (a) 
> > > > https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> > > > (b) https://www.youtube.com/watch?v=36QIepyUuhg
> > >
> > > Doh, sending the patch series failed since google won't allow file
> > > sizes larger than 25 MB. The bits prebuilt binaries have sizes 30 MB
> > > and 40 MB in two separate patches :(
> >
> > I have pushed the patchset here:
> > https://gitlab.com/anisinha/qemu/-/commits/acpi-bits .
>
> I have re-sent all the patches except patches 2 and 3 that contain
> pre-built biosbits archives. For those two patches, please see the
> gitlab repo above.

I took a peek at our avocado framework. Seems binary artefacts like
these are stored in github. For example,
https://github.com/philmd/qemu-testing-blob/tree/malta_i6400/mips/malta/mips64el
.

Maybe in this case, we can do the same - store this in my github and
the test wgets it. The flip side is that this will introduce external
dependencies for this test. I will wait for further comments from
reviewers on this.

>
> >
> > >
> > > >
> > > > Ani Sinha (12):
> > > >   qtest: meson.build changes required to integrate python based qtests
> > > >   acpi/tests/bits: add prebuilt bios bits zip archive
> > > >   acpi/tests/bits: add prebuilt bits generated grub modules and scripts
> > > >   acpi/tests/bits: initial commit of test scripts that are run by
> > > >     biosbits
> > > >   acpi/tests/bits: disable acpi PSS tests that are failing in biosbits
> > > >   acpi/tests/bits: add smilatency test suite from bits in order to
> > > >     disable it
> > > >   acpi/tests/bits: disable smilatency test since it does not pass
> > > >     everytime
> > > >   acpi/tests/bits: add biosbits config file for running bios tests
> > > >   acpi/tests/bits: add acpi and smbios python tests that uses biosbits
> > > >   acpi/tests/bits: add acpi bits qtest directory in meson for running
> > > >     tests
> > > >   acpi/tests/bits: add README file for bits qtests
> > > >   MAINTAINERS: add myself as the maintainer for acpi biosbits qtests
> > > >
> > > >  MAINTAINERS                                   |    5 +
> > > >  tests/qtest/acpi-bits/README                  |  168 ++
> > > >  tests/qtest/acpi-bits/acpi-bits-test-venv.sh  |   59 +
> > > >  tests/qtest/acpi-bits/acpi-bits-test.py       |  327 +++
> > > >  .../qtest/acpi-bits/bits-config/bits-cfg.txt  |   18 +
> > > >  tests/qtest/acpi-bits/bits-config/meson.build |   11 +
> > > >  tests/qtest/acpi-bits/bits-tests/meson.build  |   11 +
> > > >  tests/qtest/acpi-bits/bits-tests/smbios.py    | 2430 +++++++++++++++++
> > > >  .../qtest/acpi-bits/bits-tests/smilatency.py  |  103 +
> > > >  tests/qtest/acpi-bits/bits-tests/testacpi.py  |  283 ++
> > > >  tests/qtest/acpi-bits/bits-tests/testcpuid.py |   83 +
> > > >  tests/qtest/acpi-bits/meson.build             |   39 +
> > > >  .../acpi-bits/prebuilt/bits-2095-grub.tar.gz  |  Bin 0 -> 41416278 
> > > > bytes
> > > >  tests/qtest/acpi-bits/prebuilt/bits-2095.zip  |  Bin 0 -> 31922898 
> > > > bytes
> > > >  tests/qtest/acpi-bits/prebuilt/meson.build    |   11 +
> > > >  tests/qtest/acpi-bits/requirements.txt        |    1 +
> > > >  tests/qtest/meson.build                       |    7 +-
> > > >  17 files changed, 3555 insertions(+), 1 deletion(-)
> > > >  create mode 100644 tests/qtest/acpi-bits/README
> > > >  create mode 100644 tests/qtest/acpi-bits/acpi-bits-test-venv.sh
> > > >  create mode 100644 tests/qtest/acpi-bits/acpi-bits-test.py
> > > >  create mode 100644 tests/qtest/acpi-bits/bits-config/bits-cfg.txt
> > > >  create mode 100644 tests/qtest/acpi-bits/bits-config/meson.build
> > > >  create mode 100644 tests/qtest/acpi-bits/bits-tests/meson.build
> > > >  create mode 100644 tests/qtest/acpi-bits/bits-tests/smbios.py
> > > >  create mode 100644 tests/qtest/acpi-bits/bits-tests/smilatency.py
> > > >  create mode 100644 tests/qtest/acpi-bits/bits-tests/testacpi.py
> > > >  create mode 100644 tests/qtest/acpi-bits/bits-tests/testcpuid.py
> > > >  create mode 100644 tests/qtest/acpi-bits/meson.build
> > > >  create mode 100644 tests/qtest/acpi-bits/prebuilt/bits-2095-grub.tar.gz
> > > >  create mode 100644 tests/qtest/acpi-bits/prebuilt/bits-2095.zip
> > > >  create mode 100644 tests/qtest/acpi-bits/prebuilt/meson.build
> > > >  create mode 100644 tests/qtest/acpi-bits/requirements.txt
> > > >
> > > > --
> > > > 2.25.1
> > > >