Re: [PATCH 4/4] scripts/qmp: Fix QEMU Python scripts path

[John Snow]

On 4/30/20 1:04 AM, Markus Armbruster wrote:
> John Snow <address@hidden> writes:
> 
>> On 4/21/20 5:42 AM, Philippe Mathieu-Daudé wrote:
>>> QEMU Python scripts have been moved in commit 8f8fd9edba4 ("Introduce
>>> Python module structure"). Use the same sys.path modification used
>>> in the referenced commit to be able to use these scripts again.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <address@hidden>
>>> ---
>>>  scripts/qmp/qmp      | 4 +++-
>>>  scripts/qmp/qom-fuse | 4 +++-
>>>  scripts/qmp/qom-get  | 4 +++-
>>>  scripts/qmp/qom-list | 4 +++-
>>>  scripts/qmp/qom-set  | 4 +++-
>>>  scripts/qmp/qom-tree | 4 +++-
>>>  6 files changed, 18 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/scripts/qmp/qmp b/scripts/qmp/qmp
>>> index 0625fc2aba..8e52e4a54d 100755
>>> --- a/scripts/qmp/qmp
>>> +++ b/scripts/qmp/qmp
>>> @@ -11,7 +11,9 @@
>>>  # See the COPYING file in the top-level directory.
>>>  
>>>  import sys, os
>>> -from qmp import QEMUMonitorProtocol
>>> +
>>> +sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 
>>> 'python'))
>>> +from qemu.qmp import QEMUMonitorProtocol
>>>  
>>
>> Try to avoid using sys.path hacks; they don't work in pylint or mypy and
>> it provides an active barrier to CQA work here.
>> (They also tend to be quite fragile.)
>>
>> We can discuss the right way to do this; one of those ways is to create
>> an installable package that we can install locally in a virtual environment.
>>
>> Another way is perhaps to set PYTHONPATH in the calling environment so
>> that standard "import" directives will work.
>>
>> Both ultimately involve changing the environment of the user to
>> accommodate the script.
> 
> For what it's worth, tests/Makefile.involve does the latter for
> tests/qapi-schema/test-qapi.py.  Simple enough, but makes manual
> invocation inconvenient.
> 
> Not necessary for scripts/qapi-gen.py, because its "import qmp.FOO"
> finds qmp right in scripts/qmp/.
> 

Yes, using "proper" package hierarchies often means the loss of being
able to invoke the scripts directly, unless you are careful to organize
the package such that the scripts can run both in an "unpackaged" and a
"packaged" mode.

It can be done, but it's tricky and can be prone to error. Let's take a
look at how to do it!

Let's imagine we have an honest-to-goodness QAPI parser module. In
isolation, the layout for such a package would probably look like this:

qapi.git/
  setup.py
  qapi-gen.py
  README.rst
  qapi/
    __init__.py
    parser.py
    schema.py
    ...etc


Now, anything inside of qapi/ is considered the "qapi module" and you
will be unable to directly execute anything inside of this folder,
unless it does not depend on anything else in the "qapi module".

i.e. "import qapi.x" will work, but only from the executing context of a
thread that understands how to find "qapi". If you are IN this
directory, you do not have that context, so those directives will not work.

Python imports are always handled relative to the importing file, not
the imported file.

qapi-gen in the parent directory, however, can use "from qapi import
parser" without any problem, because if you are executing it directly,
it will be able to see the "qapi module" as a folder.

When packaging this for installed environments, there are some changes
we need to make.

(1) Move qapi-gen.py into the module itself! Remove the shebang, the
chmod, and any __main__ scaffolding. Name it something like "script.py".
It should look like this:

```
from qapi import parser, schema

def main():
    print("running qapi-gen!")
```

(2) Create a new shim executable where qapi-gen.py used to be. Give it
chmod u+x and name it qapi-gen.py, and it should look like this:

```
#!/usr/bin/env python3

from qapi import script

if __name__ == '__main__':
    script.main()
```

Now, at this point, you should be able to execute "qapi-gen.py" no
matter what your CWD is, as long as the qapi-gen.py shim is in your PATH.

To finish packaging this for installation purposes, though, we'll amend
setup.py to contain something like this:

setup_kwargs = {
    'entry_points': {
        'console_scripts': [
            'qapi-gen = qapi.script:main',
        ],
    },
}

Next, when you *install* this package, let's say by doing this:

> cd ~/src/qapi.git/

# Make a new virtual environment and enter it as a shell
> pipenv shell

# (If you skip the above step or don't want a venv, do this:)
# pip install --user -e .

# If you are inside the venv, do this:
pip install -e .

(This will install the python package in an "editable" or "develop"
mode, which creates a fake package header that simply redirects to the
real, live files. This means as you edit the files, the python package
is updated in real time. It's a lifesaver trick if you don't know about
it already.)


>From here, the "qapi" package is "installed" to your environment. You
should be able to pop open a python3 terminal anywhere and "import qapi"
and have that work.

You should also now have the "qapi-gen" executable in your path (or in
~/.local/bin/ if you are not in a venv) that will invoke the entrypoint
directly.

At this point, both the shim executable and the "installed version" of
the script should work correctly; offering a dual-paradigm package where
you can run and edit it in-tree, or in an installed environment.

It takes a bit more elbow grease, but tools like mypy, pylint and flake8
will better be able to understand and follow the code. You will also not
lose the ability to run scripts quickly in the source tree, but I must
caution that I don't believe this to be a viable strategy long term.

As soon as we have two or more packages that each need "common imports",
it's no longer possible to structure all dependencies as subfolders of
the common scripts, and we lose the ability to perform this "trick".

The only way to accomplish that complex structure is by installing the
packages to the venv; or using PYTHONPATH to amend the search path
outside of the script directly.

--js


P.S.: Of course, I could be wrong. Python packaging changes a lot and
there are a lot of caveats and weird hacks and ways to do almost
anything you want, but it's a matter of what will work well Out Of The
Box with common tooling. For that purpose, I generally want things to
work with pycharm, flake8, mypy, and pylint.