bug#65665: package-mapping with #:deep? #t doesn't get all the implicit inputs

[Ulf Herrman]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Ulf Herrman <striness@tilde.club> skribis:
>
>> That and a growing thirst for a nuclear option for package rewriting
>> brought about by trying to debug deep transformations while
>> simultaneously experimenting with rewriting a manifest of around 270
>> packages.
>
> On that topic of a catch-all option for rewriting: there’s an unused
> procedure called ‘map-derivation’, which performs rewriting at the level
> of derivations.  It’s harder to use, more expensive, but who knows,
> perhaps we’ll find a motivating use case…  (Allowing for graph rewriting
> has been a major goal for me since the beginning.)
>
>> There are really several distinct issues at play here:
>> 1. The case of #:qtbase and #:guile being invisible to package-mapping.
>>    This is what I first noticed, and cannot be fixed without modifying
>>    the build systems.  This is what prompted looking for packages in
>>    package and bag arguments, and recursing into lists therein (just in
>>    case an argument took a list of packages).
>
> How are #:qtbase and #:guile invisible to package mapping?
>
> They appear in the bag inputs and thus are definitely visible to
> ‘package-mapping’, as a I showed with the CMake and Python examples in
> this thread.

"Invisible to package-mapping" was perhaps not the clearest phrasing;
"not completely replaced by package-mapping" might be better.  qtbase
does indeed go in the bag inputs, but replacing the bag inputs has no
effect on the bag arguments, and even if we replaced it in the bag
arguments as well, it might not even be in there to begin with, in which
case the bag-builder would introduce a default that is only visible at
the level of derivations.

>
> For good measure :-) here’s an example with #:qtbase:
>
> $ ./pre-inst-env guix build qgit -n
> substitute: updating substitutes from 'http://192.168.1.48:8123'... 100.0%
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> 0.4 MB would be downloaded:
>   /gnu/store/7b20q17yg90b62404chgbnwgvd6ry1qf-qgit-2.10
> $ ./pre-inst-env guix build qgit -n --with-latest=qtbase
> following redirection to 
> `https://mirrors.ocf.berkeley.edu/qt/official_releases/qt/'...
> following redirection to 
> `https://mirrors.ocf.berkeley.edu/qt/official_releases/qt/6.6/'...
> guix build: warning: cannot authenticate source of 'qtbase', version 6.6.0
>
> Starting download of /tmp/guix-file.CTehnY
> From 
> https://mirrors.ocf.berkeley.edu/qt/official_releases/qt/6.6/6.6.0/submodules/qtbase-everywhere-src-6.6.0.tar.xz...
>  …-src-6.6.0.tar.xz  46.1MiB                                                  
>                                                                       
> 12.9MiB/s 00:04 ▕██████████████████▏ 100.0%
> substitute: updating substitutes from 'http://192.168.1.48:8123'... 100.0%
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 
> 100.0%
> substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 
> 100.0%
> The following derivations would be built:
>   /gnu/store/paixxkdaakv55bffggxx4l9hiknl8i5r-qgit-2.10.drv
>   /gnu/store/f9fdjk1g1s1aqmlmi4clla2kqns7283v-qtbase-6.6.0.drv
> 0.4 MB would be downloaded:
>   /gnu/store/nl9dadzfmjm9wg7v3r31jkx773dl683x-module-import-compiled
>   /gnu/store/6zryxmypw0wygayc9pvhyxkx47w0lyci-gperf-3.1
>   /gnu/store/a57n7wy8mdi7l52pr4zg07132blgj5xp-qgit-2.10-checkout

And if you do a quick 'cat
/gnu/store/paixxkdaakv55bffggxx4l9hiknl8i5r-qgit-2.10.drv'...

------------------------------
Derive
([("out","/gnu/store/z8qrhfmicylxy2mwvcvh9sizfhd3x4d3-qgit-2.10","","")]
 ,[("/gnu/store/0jk33gpzyicyppbnh458213007v0qjhs-mesa-23.1.4.drv",["out"])
   ,("/gnu/store/0njvvgjlb52abhqnmb4rx22sfkxm2h9c-gcc-11.3.0.drv",["out"])
   ,("/gnu/store/0p1sns81qbgr8ayiv02fv4rm5drcycd7-libxdamage-1.1.5.drv",["out"])
   ,("/gnu/store/14a2ban238fng3c8632lrfkmz54y7m2c-binutils-2.38.drv",["out"])
   ,("/gnu/store/1n39zcbr528b7rh9bf1pwfrm7mv8nr8m-bzip2-1.0.8.drv",["out"])
   
,("/gnu/store/2pv3mjjiwh37b0m3m1hijxifnchrw76i-libpciaccess-0.16.drv",["out"])
   ,("/gnu/store/3fy3bf7wysi1n1qz9jz8xzx11sgy8m6d-git-2.41.0.drv",["out"])
   ,("/gnu/store/3r0r8j76l0qvxasmb7rgn7lvpikjdyn1-libxdmcp-1.1.3.drv",["out"])
   ,("/gnu/store/4jfy1ca1d5772z15jcyk1v8wdwdcllbi-gzip-1.12.drv",["out"])
   ,("/gnu/store/5nvwagz2hphvlax2bnj93smr1rgrzr8l-libx11-1.8.1.drv",["out"])
   ,("/gnu/store/64vwaah2spd7q66hji6sm1j2fl6pd1rn-diffutils-3.8.drv",["out"])
   ,("/gnu/store/6k4xxkp725r09vkn7rz2gc50asjjhpkk-xorgproto-2022.2.drv",["out"])
   ,("/gnu/store/91b6yraa6qax7lq7riqg1ag6lql2gfzi-tar-1.34.drv",["out"])
   ,("/gnu/store/9kcv1x0lrf6fdck2j42zarxrvjzxxznv-coreutils-9.1.drv",["out"])
   ,("/gnu/store/9piaq0aaf202r1gq7crig1cr131kx8zn-file-5.44.drv",["out"])
   
,("/gnu/store/9y28bf3ywai2ybhr92c904s3cxsc8apx-libpthread-stubs-0.4.drv",["out"])
   ,("/gnu/store/ak17xsjb4zcw7sf0r0lxxiy4xmh57i2h-findutils-4.9.0.drv",["out"])
   
,("/gnu/store/bmc1hqlb207n1mnf9rs7wy8zh9hhf0br-qgit-2.10-checkout.drv",["out"])
   ,("/gnu/store/cmd13zzya808ca8siibj8ib3llpkrr3f-sed-4.8.drv",["out"])
   ,("/gnu/store/cr84vpnszpy3hkwsm8ijwpnkknxvx8k4-grep-3.8.drv",["out"])
   ,("/gnu/store/cx6ws98i67qf6ba9msz7n63x78ky812c-libxxf86vm-1.1.4.drv",["out"])
   ,("/gnu/store/f9fdjk1g1s1aqmlmi4clla2kqns7283v-qtbase-6.6.0.drv",["out"])
   ,("/gnu/store/fpprrvfiqw6g0mn5gb7ac87nbmrsq64l-make-4.3.drv",["out"])
   ,("/gnu/store/g2z0fvzkh5xl4f00aixchkwxrwbbljlr-libxext-1.3.4.drv",["out"])
   ,("/gnu/store/gb247cil5nlnx175dhqmgg67q7ng7n2h-which-2.21.drv",["out"])
   
,("/gnu/store/hb2y5axynnpwqpx1znjsn5azspc9a4lw-module-import-compiled.drv",["out"])
   ,("/gnu/store/hjhr64r5x3bhdw63zz3a3v09vfrlkhrh-qtbase-5.15.10.drv",["out"])
   ,("/gnu/store/hvqfl21wx397k58jpn5dpn2l5y5k1dl0-patch-2.7.6.drv",["out"])
   
,("/gnu/store/kn7kcpkkcvy6gmrc74mf4mq0290dsf94-glibc-2.35.drv",["out","static"])
   ,("/gnu/store/l3ns54f8vgqmycwi50p5cwbr6l466kj3-libdrm-2.4.114.drv",["out"])
   ,("/gnu/store/lfsv077ggq8w8cbw86wf1075zhb3bhrw-xz-5.2.8.drv",["out"])
   
,("/gnu/store/lssiz5ppxfybwmr638x3bcmn4b6ixk89-linux-libre-headers-5.15.49.drv",["out"])
   
,("/gnu/store/n4p8vzp14k1pkgxyfb5mak12rm1yiwnr-cmake-minimal-3.24.2.drv",["out"])
   
,("/gnu/store/nldpb7xl1slkiigvr9rax7qhcky5flnc-util-macros-1.19.3.drv",["out"])
   ,("/gnu/store/nvaxgm6jvjy6agmp89ivahz7z9vl3ldh-libxcb-1.15.drv",["out"])
   ,("/gnu/store/pdkcmkk5h3zq82ws9fps4bxnfmsmz0kx-libxshmfence-1.3.drv",["out"])
   ,("/gnu/store/s5nyfh0cd3z9wsk69c9blj5y850kvvcm-libvdpau-1.5.drv",["out"])
   ,("/gnu/store/swblndkq2c4rzyv3xfkmsa4cjf6abphf-ld-wrapper-0.drv",["out"])
   ,("/gnu/store/vkns1vij5hjamh7dpd74zs5203526747-libxfixes-6.0.0.drv",["out"])
   
,("/gnu/store/w2mnryfds1bd7wyyqmdi0kblz0dym1bx-glibc-utf8-locales-2.35.drv",["out"])
   ,("/gnu/store/wwmk0rrnb5q8f48fm7h6grzar5qmslgi-libxau-1.0.10.drv",["out"])
   
,("/gnu/store/xwckz4hbfydkfiiaa7bgslbh317gdkiq-bash-minimal-5.1.16.drv",["out"])
   ,("/gnu/store/z7fxw9jj7avcr1ng88pak3ds3kxbfy15-gawk-5.2.1.drv",["out"])
   ,("/gnu/store/zraigp7miin3vzr5dcbr4i9rvds0i07r-guile-3.0.9.drv",["out"])]
 
,["/gnu/store/2lg0ibv3vw01xc83advvrhvmc8yxbhh0-qgit-2.10-builder","/gnu/store/p173g6kxa69qaypf8dvnw1ismd8g4k8q-module-import"]
 
,"x86_64-linux","/gnu/store/g8p09w6r78hhkl2rv1747pcp9zbk6fxv-guile-3.0.9/bin/guile",["--no-auto-compile","-L","/gnu/store/p173g6kxa69qaypf8dvnw1ismd8g4k8q-module-import","-C","/gnu/store/nl9dadzfmjm9wg7v3r31jkx773dl683x-module-import-compiled","/gnu/store/2lg0ibv3vw01xc83advvrhvmc8yxbhh0-qgit-2.10-builder"]
 ,[("out","/gnu/store/z8qrhfmicylxy2mwvcvh9sizfhd3x4d3-qgit-2.10")])
---------------------------

you'll see we have both a
("/gnu/store/f9fdjk1g1s1aqmlmi4clla2kqns7283v-qtbase-6.6.0.drv",["out"])
and a
("/gnu/store/hjhr64r5x3bhdw63zz3a3v09vfrlkhrh-qtbase-5.15.10.drv",["out"])
in use.

(and if this were a transformation that applied to all packages, it
would be using two variants of the entire world beneath qtbase also)

>> 2. The (perceived) case of packages hiding inside arguments.  In
>>    hindsight, this was probably actually (3) causing this, though it's
>>    hard to tell because I discovered it last.  I attempted to resolve
>>    this by recursing through <gexp>s and <gexp-input>s.
>
> Yeah.  I think we have to understand that “hiding” is to some extent
> inevitable; <package> is more concise than <bag>, which is more concise
> than <derivation>.  There are extra “inputs” showing up when we go from
> one abstraction to the one below.

Ideally no new packages would be introduced when going from bags to
derivations - there is also information at the package level that is
missing at the derivation level, like properties and arguments, that
transformations should be able to access.
>
>> 3. `this-package' referring to the inherited package in thunked fields,
>>    rather than the package inheriting them.  This is what prompted the
>>    use of package-{inputs,arguments,etc}-with-package.
>
> Ah yes, I agree; I reported it here:
>
>   https://issues.guix.gnu.org/50335
>
> I think we can discuss it separately though, in that bug report
> probably.
 
Ah, in the meantime I created issue #66510.  I suppose I should merge
that.  I think the syntactic option presented there seems promising -
extending record constructors so that (inherit parent) can be replaced
with (inherit parent (fieldname variable-like-macro-name) ...) to cause
uses of variable-like-macro-name within the constructor to refer to the
parent's value of fieldname, with any field thunk of the parent
evaluated while `this-package' is bound to the package being defined.

- Ulf

signature.asc
Description: PGP signature