[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Pinning package inputs using inferiors?
From: |
Efraim Flashner |
Subject: |
Re: Pinning package inputs using inferiors? |
Date: |
Sat, 22 Oct 2022 22:04:46 +0300 |
On Fri, Oct 21, 2022 at 10:08:10PM +0100, Phil wrote:
> Thanks Simon - I've given an example below.
>
> zimoun writes:
>
> > For an example, see python-numpy and python-numpy-next in (gnu packages
> > python-xyz).
>
> This was my original way of handling this but in what is perhaps a niche
> use of Guix by my department - it ultimately doesn't scale well, for our
> use-case.
>
> Originally the department was small enough that there was only a handful
> of applications sharing one or two common in-house libraries.
>
> As we've scaled-up we now have the situation where 3 or 4 common
> libraries are being used by say 10 applications.
>
> We have rapid release schedules - and want to be able to release the
> common libraries on a weekly basis. But the time to sign-off on a
> common library takes a few days per application, so it's not practical for
> every project to bump version every week - they have other priorities.
>
> In an ideal world automated unit and regression testing would be
> comprehensive enough that we could move aggressively each week, but at
> least for now that's not practical given the complex nature of signing
> off the libraries and the applications which use the libraries.
>
> So, ideally, what we'd like to do is just have each common library
> churn-out releases every week, and have the releases available in Guix,
> but without having an obligation on dependent applications to adopt
> these changes if they don't want to.
>
> Note all libraries and applications share the same channel - one
> solution would be to have each library in their own channel, but this
> feels ugly to me.
>
> Our solution (somewhat tricky to explain without a whiteboard -
> apologies!) is to co-locate the package definition of the common library
> in the common library repo itself - we call it something like
> .requirements.scm and it is naturally kept in lockstep with the code in
> that repo that the definition will build. This is very different to
> traditional Guix where channels contain definitions separately in a
> different repo to the code those definitions describe how to build.
>
> We then have a job in our CI/CD system that allows us to give a tag on the
> common library repo, and the name of an application that uses the common
> library.
>
> The job will copy the .requirements.scm into our channel inside a
> private module specific to the application that uses the common library.
>
> The idea is that you can have many versions of .requirements.scm private
> to every application package definition that references it.
>
> You could even read .requirements.scm using a function that clones the
> application repo on-the-fly rather than statically storing it in the
> channel - we haven't gone this far yet, as it makes the system even more
> complex to reason about.
>
> This is basically the same idea as the python-numpy-next but allows for
> many versions of python-numpy to co-exist by keeping them all in private
> modules so they don't clash.
>
> It's a cool idea and works pretty well, but requires us to augment Guix
> with a set of extra tools to lift and shift these private definitions
> around, which complicates our setup considerably.
>
> It feels like wanting to make many versions of a library available at
> once isn't an unreasonable way to work at-scale. However, it also feels
> like a departure from the philosophy of Guix to decentralise package
> definitions and to allow for a potentially large range of versions to
> co-exist in the same channel commit.
>
> We could try to further integrate the idea into guix by writing new guix
> commands to support it - we're still working out the details ourselves,
> but if it works well we'd love to present it at a future Guix Days or
> similar!
>
> In the meantime I was wondering if anyone else had a similar use-case
> for Guix and if they had tried something similar or different to handle
> many versions in an automated way in the same channel commit?
>
> Apologies that's more than I was intending to write - but hopefully that
> makes some sense! If it doesn't I can try to flesh out specific example?
Apologies for not slotting in the reply inline, I'm not sure exactly
where to put it.
This might be a good use for package transformations. Imagine the
following:
;;; Python-dep-1
(define-publid python-dep1-week1
...)
(define-publid python-dep1-week2
...)
(define-publid python-dep1-week3
...)
;;; Python-dep-2
(define-publid python-dep2-week1
...)
(define-publid python-dep2-week2
...)
(define-publid python-dep2-week3
...)
;;; Python package
(define my-python-package-base
(name "my-python-package-base")
...
(inputs
(list python-dep1
python-dep2)))
(define-public my-python-package
(inherit my-python-package-base)
(name "my-python-package")
(inputs
(modify-inputs (package-inputs python-package-base)
(replace python-dep1 python-dep1-week3)
(replace python-dep2 python-dep2-week2))))
or if you wanted to do it recursively
(define package-inputs-for-my-python-package
(package-input-rewriting/spec
`(("python-dep1" . ,(const python-dep1-week3))
("python-dep2" . ,(const python-dep2-week2)))))
(define-public python-package-with-correct-inputs
(package
(inherit (package-inputs-for-ython-package my-python-package-base))
(name "my-python-package")))
Both ideas use 'placeholder packages' so that you can swap them out for
your actual dependency, the first one only for itself and the second one
working recursively. I use the second one myself at work to replace
tensorflow with a version built for the machine it's running on, and at
home to use some newer golang libraries.
--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
signature.asc
Description: PGP signature