Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits

[Ani Sinha]

On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote:
> >
> > Ani Sinha <ani@anisinha.ca> writes:
> >
> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >>
> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote:
> > >> > And have multiple platform specific branches in bits that have fixes 
> > >> > for those
> > >> > platforms so that bits can run there. Plus the existing test can be 
> > >> > enhanced to
> > >> > pull in binaries from those branches based on the platform on which it 
> > >> > is being
> > >> > run.
> > >> >
> > >>
> > >> What a mess.
> > >> Who is going to be testing all these million platforms?
> > >
> > > I am not talking about branches in QEMU but branches in bits.
> > > If you are going to test multiple platforms, you do need to build bits
> > > binaries for them. There is no way around it.
> > > bits is not all platform independent python. It does have binary 
> > > executables.
> > >
> > > Currently bits is built only for the x86 platform. Other platforms are
> > > not tested. I doubt if anyone even tried building bits for arm or
> > > mips.
> >
> > I'm not worried about test bits on other targets, but we do run x86
> > targets on a number of hosts. The current reliance on a special patched
> > host build tool for only one architecture is the problem. If  we just
> > download the iso that problem goes away.
>
> 👍what he said.

Yes, in that case the problem is that upstream bits does not pass all
the test out of the box. Hence we are taking this approach of keeping
some test scripts in QEMU repo and modifying them. Then generating the
iso with the modified scripts. It also helps developers who want to
write new tests or make enhancements to existing tests.
If modifications need to be made to tests, they need to be versioned.
We have gone through the route of not using submodules and I am not
going to open that can of worms again.
We also have no consensus on where to keep the one time built iso that
we can download for this test you are proposing.

So let's figure out the above first. Programmatically downloading an
iso and running tests within a VM would be a much simpler test than
the one I wrote. We can add a subtest or a brand new test anytime if
we can figure out the above logistics.

>
> > > It makes sense to try things incrementally once we have something going.
> > >
> > > Lets discuss this on a separate thread.
> > >
> > >> All this does nothing at all to help developers avoid
> > >> bugs and when they do trigger debug the issue. Which is
> > >> after all why we have testing.
> > >> Yes once in a very long while we are going to tweak
> > >> something in the tests, and for that rare occurence
> > >> it makes sense to periodically rebuild everything,
> > >> otherwise code bitrots.
> > >>
> > >> But the test is supposed to run within a VM anyway, let's
> > >> have an image and be done with it.
> > >>
> > >> --
> > >> MST
> > >>
> >
> >
> > --
> > Alex Bennée
>