Re: Note for Python packages: packages do not have "inputs" (most of the time)

[Hartmut Goebel]

Am 23.10.2016 um 21:52 schrieb ng0:
> I think this is wrong. My assumption is I work with the old system as
> long as the new system is not in place. I see no changes which fix this,
> so why should I do work in advance when this must be fixed afterwards?
>
> Will the current system report packages as broken, or are they entirely
> broken? From my perspective this reads weird. I have not read the new
> documentation section, but I assumed this is not yet in place?

The new documentation section is already in place, since it is totally
independent from the new or the old build system.

Please note that I'm only asking to stop introducing more broken
packages (this is using "inputs" for python packages). I'm taking care
of the existing packages on the wip-python-build-system branch.

The current system does not detect that these packages are broken
(neither does the new one), but using "inputs" is conceptually broken,
as I'll describe the following:

Depending on the way the python packages are installed (pip,
easy_install, setup.py install with distutils, setup.py install with
setuptools, setup.py easy_install) what ends up in …/site-packages/ is
quite different. When using setup.py install with setuptools (which is
the most common way), a .pth-file will be created containing a link to
the required packages. See [Example 1 below]

Now imagine this:

Package P has packet "R" as input. When "P" gets installed, version 0.2
of "R" is in guix. The .pth-file (of packages P) refers to
/gnu/store/abcd…-R-0.2/… Everything is fine. Some time later, you
install into the same profile packet "Q" which required "R", too, but to
function correctly it requires at least version 0.4 of "R". The
.pth-file of packages Q refers to /gnu/store/1234…-R-0.4/…. But "R"
itself will not show up in the site-packages directory.

So you have two .pth-files in your site-packages directory, pointing to
different versions of the same package "R". Now when running python, all
.pth-files in site-packages are processed in alphabetical order (as
documented in [1]). So P.pth is processed prior to Q.pth and R-0.2.egg
will end up in front of R-0.4.egg. Package Q will not function correctly
since the old version of R will be imported.

[Example 2] is a real-world example for this. As you can seen, there are
*two* versions of werkzeug installed and version 0.11.5 imported. (In
this very case additionally "Werkzeug" will be in site-packages since it
is a propagated input for flask, resulting in werkzeug being here
*three* time.)

But it is even worse: Now install some package "A", requiring "R-0.4".
You will get a A.pth file, which will be processed even prior to P.pth,
and not (magically) Q functions correctly again. Huh!

So you have massive conflicts, and guix is not able to warn you about
the conflict. Since from the guix point of view everything is right.

And this is why using "inputs" is wrong and must be replaces by
"propagated-inputs".

When using propagated inputs, in the example above when installing Q,
R-0.4 would have been installed into site-packages. This alone would not
solve the import issue described above, but the problem would not be
hidden inside these .pth files (which barely anybody understands).


Example 1:
$ # Using guix 0.11.0
$ guix package -i python-flask
$ # should install mprfzzs3zy2z9zmlidh93g63nf70ki6z-python-flask-0.10.1
$ ls /gnu/store/mprfzzs3zy*python-flask-0.10.1/lib/py*/site-packages/
Flask-0.10.1-py3.4.egg/  __pycache__/  python-flask-0.10.1.pth  site.py
$ cat
/gnu/store/mprfzzs3zy*python-flask-0.10.1/lib/py*/site-packages/*.pth  #
Output shortened
./Flask-0.10.1-py3.4.egg
/gnu/store/2h4fsdw…-python-itsdangerous-0.24/lib/python3.4/site-packages/itsdangerous-0.24-py3.4.egg
/gnu/store/pw81lwj…-python-jinja2-2.8/lib/python3.4/site-packages/Jinja2-2.8-py3.4.egg
/gnu/store/0r2wdiciw…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg
/gnu/store/brz263mir…-python-markupsafe-0.23/lib/python3.4/site-packages/MarkupSafe-0.23-py3.4-linux-x86_64.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:];
p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert =
p+len(new)


[Example 2]
$ # Using guix 0.11.0
$ guix package -i python python-flask
$ # Using guix master as of today
$ ./pre-inst-env guix package -i python-pytest-localserver
$ grep -h werkzeug ~/.guix-profile/lib/python3.4/site-packages/*.pth  #
output shortend
/gnu/store/0r2wdiciw1…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg
/gnu/store/r59535dq0…-python-werkzeug-0.11.11/lib/python3.4/site-packages/Werkzeug-0.11.11-py3.4.egg
$ ~/.guix-profile/bin/python3 -c "import sys
for p in sys.path: 'werkzeug' in p and print(p)"
/gnu/store/0r2wdiciw1…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg
/gnu/store/r59535dq0…-werkzeug-0.11.11/lib/python3.4/site-packages/Werkzeug-0.11.11-py3.4.egg
$ ~/.guix-profile/bin/python3 -c "import werkzeug ;
print(werkzeug.__file__)"
/gnu/store/0r2wdiciw1…-python-werkzeug-0.11.5/lib/python3.4/site-packages/Werkzeug-0.11.5-py3.4.egg/werkzeug/__init__.py

[1] https://docs.python.org/3/library/site.html

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | address@hidden               |
| www.crazy-compilers.com | compilers which you thought are impossible |