[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH] edk2 build scripts: work around TianoCore#1607 without forci
From: |
Laszlo Ersek |
Subject: |
Re: [PATCH] edk2 build scripts: work around TianoCore#1607 without forcing Python 2 |
Date: |
Thu, 19 Sep 2019 23:40:57 +0200 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 |
On 09/19/19 21:56, Philippe Mathieu-Daudé wrote:
> On 9/19/19 9:08 PM, Laszlo Ersek wrote:
>> On 09/19/19 18:39, Philippe Mathieu-Daudé wrote:
>>> On 9/18/19 7:11 PM, Laszlo Ersek wrote:
>>>> It turns out that forcing python2 for running the edk2 "build" utility is
>>>> neither necessary nor sufficient.
>>>>
>>>> Forcing python2 is not sufficient for two reasons:
>>>>
>>>> - QEMU is moving away from python2, with python2 nearing EOL,
>>>>
>>>> - according to my most recent testing, the lacking dependency information
>>>> in the makefiles that are generated by edk2's "build" utility can cause
>>>> parallel build failures even when "build" is executed by python2.
>>>>
>>>> And forcing python2 is not necessary because we can still return to the
>>>> original idea of filtering out jobserver-related options from MAKEFLAGS.
>>>> So do that.
>>>
>>> FYI I tried uninstalling python2 on Fedora 29,
>>>
>>> $ make -C roms efi -j8
>>> make: Entering directory '/home/phil/source/qemu/roms'
>>> make -C edk2/BaseTools \
>>> EXTRA_OPTFLAGS='' \
>>> EXTRA_LDFLAGS=''
>
> ^ this is the 'edk2-basetools' target from roms/Makefile.
>
>>> make[1]: Entering directory '/home/phil/source/qemu/roms/edk2/BaseTools'
>>> [...]
>>> make -C Tests
>>> make[2]: Entering directory
>>> '/home/phil/source/qemu/roms/edk2/BaseTools/Tests'
>>> /bin/sh: python: command not found
>>> make[2]: *** [GNUmakefile:11: test] Error 127
>>>
>>> 'python' seems to be provided by python-unversioned-command which is
>>> wired to Python2:
>>>
>>> $ dnf info python-unversioned-command
>>> Last metadata expiration check: 0:03:08 ago on Thu 19 Sep 2019 04:21:21
>>> PM UTC.
>>> Available Packages
>>> Name : python-unversioned-command
>>> Version : 2.7.16
>>> Release : 2.fc29
>>> Arch : noarch
>>> Size : 13 k
>>> Source : python2-2.7.16-2.fc29.src.rpm
>>> Repo : updates
>>> Summary : The "python" command that runs Python 2
>>> URL : https://www.python.org/
>>> License : Python
>>> Description : This package contains /usr/bin/python - the "python"
>>> command that runs Python 2.
>>>
>>> I had to manually run update-alternatives to continue:
>>>
>>> $ sudo update-alternatives --install /usr/bin/python python
>>> /usr/bin/python3 69
>>>
>>> Not sure this is the expected behavior, it is confusing.
>>>
>>
>> The python detection is not fool-proof in edk2. A description is given at:
>>
>> https://github.com/tianocore/tianocore.github.io/wiki/BaseTools-Support-Python2-Python3
>>
>> To summarize that, it works like this, on Linux:
>>
>> - if you set PYTHON_COMMAND, then the binary pointed to by
>> PYTHON_COMMAND will be used. The edk2 build infrastructure will
>> determine whether the pointed-to binary is python 2 or python 3, and
>> branch to the corresponding implementation of the build tools.
>>
>> - Otherwise, *minor* version auto-detection is attempted. With
>> PYTHON3_ENABLE unset, or set to "TRUE", this minor version autodetection
>> will aim at minor versions of python3.
>>
>> - Otherwise (= PYTHON3_ENABLE set to a string different from "TRUE"),
>> the minor version auto-detection will focus on python2.
>
> What you document regarding PYTHON3_ENABLE is valid once we sourced
> edksetup.sh which is done in Makefile.edk2, one step after the previous
> call:
>
> efi: edk2-basetools # call 1 (python failing)
> $(MAKE) -f Makefile.edk2 # call 2 sourcing edksetup.sh
>
>> With this patch applied, the middle case is active. Apparently it fails,
>> because the edk2 build tools developers could not foresee the situations
>> that you've exposed the auto-detection to, on Ubuntu and Fedora. Back
>> when I tested the python3 enablement in edk2, I checked the patches in
>> the following environments:
>>
>> - on RHEL-7 with the system python 2,
>> - on RHEL-7 with python3.4 from EPEL-7,
>> - on RHEL-8 with python3.6,
>> - on RHEL-8 with platform-python.
>>
>> Everything worked fine for me. I have no clue what's going on in Ubuntu
>> and in Fedora.
>>
>> Can we require all build host installations -- where we expect to run
>> "make efi" -- to provide a Python 3 binary on $PATH that is plainly
>> called "python3"?
>
> Kevin recently suggested a similar patch (in another area):
> https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg04377.html
>
>> Then I think this patch should work. (If necessary, I could set
>> "PYTHON_COMMAND=python3", too.)
>
> Yes, I confirm forcing "PYTHON_COMMAND=python3 make -C roms efi" works.
>
> Not sure what is the cleaner way to fix this although...
Thanks for the analysis!
I understand the issue now.
- "edk2/BaseTools/GNUmakefile" runs $(MAKE) in three subdirs:
- Source/C,
- Source/Python,
- Tests (which depends on the former two)
- "edk2/BaseTools/Source/C/GNUmakefile" builds fine
- "edk2/BaseTools/Source/Python/GNUmakefile" does nothing
- "edk2/BaseTools/Tests/GNUmakefile" depends on PYTHON_COMMAND -- which
should either come from the user, or from sourcing "edksetup.sh"
Therefore the issue is:
- the "edk2-basetools" target in "roms/Makefile" does not
run (more precisely, does not "source") edksetup.sh
- the "build-edk2-tools" target in "tests/uefi-test-tools/Makefile"
does not run (more precisely, does not "source") edksetup.sh
I don't think I will reorganize the dependencies in those makefiles. Nor
will I source edksetup.sh in the makefile recipes. Instead, I'll
directly set PYTHON_COMMAND=python3 in the "tools" recipes, and in the
build wrapper shell scripts.
I'll try to post v2 soon.
Thanks!
Laszlo