Re: biosbits test failing on origin/master

[Ani Sinha]

On Wed, Nov 16, 2022 at 12:18 AM John Snow <jsnow@redhat.com> wrote:
>
> On Tue, Nov 15, 2022 at 9:31 AM Ani Sinha <ani@anisinha.ca> wrote:
> >
> > On Tue, Nov 15, 2022 at 3:36 PM Ani Sinha <ani@anisinha.ca> wrote:
> > >
> > > On Tue, Nov 15, 2022 at 9:07 AM Ani Sinha <ani@anisinha.ca> wrote:
> > > >
> > > > On Tue, Nov 15, 2022 at 5:13 AM John Snow <jsnow@redhat.com> wrote:
> > > > >
> > > > > On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha <ani@anisinha.ca> wrote:
> > > > > >
> > > > > > On Thu, Nov 10, 2022 at 11:37 PM John Snow <jsnow@redhat.com> wrote:
> > > > > > >
> > > > > > > Hiya, on today's origin/master
> > > > > > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make
> > > > > > > check-avocado" is failing on the new biosbits test on my local
> > > > > > > development machine:
> > > > > > >
> > > > > > >  (001/193) 
> > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits:
> > > > > > > FAIL: True is not false : The VM seems to have failed to shutdown 
> > > > > > > in
> > > > > > > time (83.65 s)
> > > > > > >
> > > > > > > Is this a known issue, or should I begin to investigate it?
> > > > > >
> > > > > > In my test environment it does pass.
> > > > > >
> > > > > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado
> > > > > > Fetching asset from
> > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> > > > > > JOB ID     : 35726df7d3c2e0f41847822620c78195ba45b9b9
> > > > > > JOB LOG    : 
> > > > > > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log
> > > > > >  (1/1) 
> > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits:
> > > > > > PASS (57.57 s)
> > > > > > RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | 
> > > > > > INTERRUPT 0
> > > > > > | CANCEL 0
> > > > > > JOB TIME   : 63.82 s
> > > > > >
> > > > > > However, I have seen that on certain slower test machines or when 
> > > > > > run
> > > > > > within a virtual machine, the test can take longer to complete and 
> > > > > > 60
> > > > > > secs may not always be enough. In those cases raising the maximum
> > > > > > completion time to 90 secs helps. Perhaps you can try this and let 
> > > > > > me
> > > > > > know if it helps:
> > > > >
> > > > > Hmm - I'm running on a fairly modern machine and not in a VM. Do you
> > > > > have an invocation to share that exists outside of the avocado
> > > > > machinery
> > > >
> > > > If you pass V=1 in the environment then it dumps the QEMU command line
> > > > that was used to run the test. You also need to comment out the line
> > > > > shutil.rmtree(self._workDir)
> > > > in tearDown() so that the iso is not cleaned up.
> > >
> > > Maybe I will send out this patch once we have figured out what is
> > > going on with your environment:
> > > https://gitlab.com/anisinha/qemu/-/commit/5e8c629fdecc7cb650e4acaad8a8fcc2b248434e
> > >
> > > I ran the test on another box sitting in my office running centos7.9
> > > and it passed as well.
> >
> > For the records,
> > I ran make check-avocado on my Ubuntu 222.04 laptop:
> > https://pastebin.com/0ZKEEQds
> > On a separate centos 7.9 box (fairly new) : https://pastebin.com/QWLGDbp4
>
> As a question: Is it necessary to implement your own timeout here?
> What's wrong with relying on Avocado's timeout?

When I wrote the test I was not aware of the avocado timeout. Hence
implemented mine. However, I kind of think that timing out from the
test itself rather than from the framework provides an opportunity to
provide more meaningful information to the user when the timeout
happens. Of Course we can improve upon the current  "FAIL: True is not
false" assertion message.

> My hunch is that you'll get greater flexibility by leaning into the
> tool suite's configuration instead of hardcoding your own...
>
> For what it's worth, I am now trying to run this test manually by doing:
>
> > time ./qemu-system-x86_64 -cdrom 
> > /var/tmp/acpi-bits-b_br0ch8.tmp/bits-2020.iso -icount auto
>
> This is not a quick test:
>
> ________________________________________________________
> Executed in   86.50 secs    fish           external
>    usr time   86.57 secs    0.00 micros   86.57 secs
>    sys time    0.30 secs  903.00 micros    0.29 secs
>
>
> This isn't the *most* cutting edge machine, but it's a Intel(R)
> Core(TM) i7-9850H CPU @ 2.60GHz with 32GB of memory and NVME storage.
> Is your machine really passing this test in under 60 seconds flat?

Yes it is both on my 3 year old lenovo laptop with 16 Gib memory:

Vendor ID:               GenuineIntel
  Model name:            Intel(R) Core(TM) i5-8365U CPU @ 1.60GHz
    CPU family:          6
    Model:               142
    Thread(s) per core:  2
    Core(s) per socket:  4
    Socket(s):           1
    Stepping:            12
    CPU max MHz:         4100.0000
    CPU min MHz:         400.0000

as well as on a desktop machine with 32 Gib of memory:

Vendor ID:             GenuineIntel
CPU family:            6
Model:                 158
Model name:            Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz
Stepping:              13
CPU MHz:               799.987
CPU max MHz:           4700.0000
CPU min MHz:           800.0000
BogoMIPS:              6000.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              12288K
NUMA node0 CPU(s):     0-7