Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[David Gibson]

On Wed, Sep 04, 2019 at 03:57:17PM -0400, Cleber Rosa wrote:
> On Sat, Aug 31, 2019 at 11:48:34AM +1000, David Gibson wrote:
> > On Fri, Aug 30, 2019 at 02:56:48PM -0300, Eduardo Habkost wrote:
> > > On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
> > > > If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
> > > > machine when the openssl-devel package isn't installed, I get the
> > > > following very cryptic error:
> > > > 
> > > >   VENV    /home/dwg/qemu/build/rhel8/tests/venv
> > > >   PIP     /home/dwg/qemu/tests/requirements.txt
> > > > Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import 
> > > > setuptools, 
> > > > tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize,
> > > >  'open', open)(__file__);code=f.read().replace('\r\n', 
> > > > '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record 
> > > > /tmp/pip-1efs22iz-record/install-record.txt 
> > > > --single-version-externally-managed --compile --install-headers 
> > > > /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography"
> > > >  failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
> > > > 
> > > > Using V=1 doesn't give any more useful information, and it's not
> > > > (easily) possible to manually re-run the given command since it relies
> > > > on things in /tmp that are removed once the attempt finishes.
> > > 
> > > V=1 is supposed to show the actual pip command being run.  I see
> > > it here:
> > > 
> > >   $ make check-venv V=1
> > >   [...]
> > >   python3 -B -m venv --system-site-packages 
> > > /home/ehabkost/rh/proj/virt/qemu/tests/venv
> > > * /home/ehabkost/rh/proj/virt/qemu/tests/venv/bin/python -m pip -q 
> > > install -r /home/ehabkost/rh/proj/virt/qemu/tests/requirements.txt
> > >   You are using pip version 19.0.3, however version 19.2.3 is available.
> > >   You should consider upgrading via the 'pip install --upgrade pip' 
> > > command.
> > >   touch /home/ehabkost/rh/proj/virt/qemu/tests/venv
> > > 
> > > Can you reproduce the problem if you re-run the pip command?
> > 
> > Yes, I can, but it's basically just the same error without any extra
> > information.
> > 
> > > > I only figured out it was openssl-devel being missing that was the
> > > > problem by (mostly) guesswork.  It would be really great if we could
> > > > generate a more helpful error here.
> > > 
> > > I don't think there's much we can do about it, as the error is generated 
> > > by a
> > > package being built by pip.
> > > 
> > > There's one thing we can do to make it slightly better: not passing `-q` 
> > > to pip
> > > if running with V=1, so people can see which packages are being installed 
> > > and
> > > which one is generating the error.
> > 
> > That sounds like a good start to me.
> > 
> > > > In addition, if I rerun "make check-acceptance" it no longer even
> > > > attempts the PIP install, since tests/venv already exists in my build
> > > > environment.  It then sort of works, but I think it might be hitting
> > > > other errors because of the missing python packages.  Sorry that's a
> > > > bit vague - I also seem to be getting unrelated errors that I'm still
> > > > trying to figure out.
> > > 
> > > This sounds like a bug and we need to fix it.
> > 
> > I agree.
> 
> Hi David,
> 
> I've pushed a branch here (most of the commits have already been sent
> separately):
> 
>    https://github.com/clebergnu/qemu/tree/ppc64
> 
> I've tested on a RHEL 8 ppc64le Power 9, and it seems to work for me.
> My steps for building QEMU:
> 
>   - Configured with: '/home/cleber/src/qemu/configure' '--enable-slirp=git' 
> '--python=/usr/bin/python3' 
> '--target-list=x86_64-softmmu,ppc64-softmmu,arm-softmmu,aarch64-softmmu,mips-softmmu,mipsel-softmmu,mips64-softmmu,mips64el-softmmu,sh4-softmmu,s390x-softmmu,alpha-softmmu,m68k-softmmu,riscv64-softmmu'
>   - make
>   - make check-acceptance
> 
> Would you be able to test if that branch works smoothly for you?

So, with this tree I'm no longer getting problems if openssl-devel is
not installed, so that much looks good.

I am getting some different errors - I was seeing this before (with
openssl-devel installed) sometimes, but only sometimes.  I haven't yet
worked out a pattern for when they appeared.  They also don't appear
to be fatal, the rest of the tests seem to be running ok.  Any ideas?

 VENV    /home/dwg/qemu/build/rhel8/tests/venv
  PIP     /home/dwg/qemu/tests/requirements.txt
  AVOCADO tests/acceptance
Error running method "run" of plugin "html": 'Namespace' object has no 
attribute 'get'
Error running method "run" of plugin "varianter_cit": 'Namespace' object has no 
attribute 'get'
JOB ID     : 6808bb316632af2bf7652586b3fbb22ac497e090
JOB LOG    : 
/home/dwg/qemu/build/rhel8/tests/results/job-2019-09-04T22.06-6808bb3/job.log
Error running method "initialize" of plugin "varianter_cit": 'Namespace' object 
has no attribute 'get'
Error running method "initialize" of plugin "yaml_to_mux": 'Namespace' object 
has no attribute 'get'
 (01/34) 
/home/dwg/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:
 PASS (4.45 s)

...

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature