Re: Progress on 'guix deploy'

[Christopher Lemmer Webber]

Jakob L. Kreuze writes:

> Hello, Guix!
>
> Apart from a few patches and my introductory email about a month ago,
> I've been pretty silent. I feel it's time to finally break that silence,
> let people know where progress has been made, and request some feedback
> on the code I've written so far.

Hi Jakob!  Thanks for sending this to the list. :)

> As a quick refresher, my GSoC project this summer is 'guix deploy', a
> deployment automation tool for GuixSD that's been discussed more
> thoroughly in [1] and [2]. Development has taken place on my personal
> branch of Guix, specifically the 'wip-deploy' branch [3], and is
> represented by three new Scheme source files:
>
> - 'gnu/machine.scm', which provides an abstraction for /something/ that
>   can be deployed to in a heterogeneous deployment. Currently, the only
>   concrete implementation of this is the simple case of in-place updates
>   to machines running SSH whose names and IP addresses we know.
> - 'gnu/tests/machine.scm', which implements some tests for
>   'gnu/machine.scm'. This is where I'm most interested in receiving
>   feedback. More on that later.
> - 'guix/scripts/deploy.scm', which implements the rudimentary
>   command-line interface for 'guix deploy'.
>
> The command-line interface hasn't really been fleshed out yet, but if
> you'd like to play around with it, it takes a "deployment" file as a
> parameter, which is a Scheme file looking something like the following:
>
> #+BEGIN_SRC scheme
> (use-modules ...)
>
> (define %system
>   (operating-system
>    (host-name "gnu-deployed")
>    (timezone "Etc/UTC")
>    (bootloader (bootloader-configuration
>                 (bootloader grub-bootloader)
>                 (target "/dev/sda")
>                 (terminal-outputs '(console))))
>    (file-systems (cons (file-system
>                         (mount-point "/")
>                         (device "/dev/vda1")
>                         (type "ext4"))
>                        %base-file-systems))
>    (services
>     (append (list (service dhcp-client-service-type)
>                   (service openssh-service-type
>                            (openssh-configuration
>                             (permit-root-login #t)
>                             (allow-empty-passwords? #t))))
>             %base-services))))
>
> (list (make <sshable-machine>
>         #:host-name "localhost"
>         #:ssh-port 5556
>         #:system %system))
> #+END_SRC
>
> Basically, you attach an 'operating-system' declaration to a 'machine'.
> In this case, 'sshable-machine' is the specific type of 'machine' that
> we're deploying to (one that's running an SSH daemon and has a known IP
> + port + hostname).
>
> I've found that the GuixSD QEMU images work well for playing around with
> this, provided that you add the SSH service to the system configuration
> and start it. In the case of this deployment file, I had a GuixSD guest
> with port 22 forwarded to my host's port 5556. You'll also need to set
> up some sort of public key auth in your SSH config. The current code
> isn't capable of handling other forms of SSH authentication.
>
> In terms of implementation, GOOPS feels like a bit of an unusual choice
> when juxtaposed with the rest of the Guix codebase, but I've come to
> really enjoy it. I'll roll with it for now, since I think it will make
> it easier to flesh out the vocabulary for specifying deployments.

I also want to clarify: I know that GOOPS code isn't quite contemporary
style for Guix, but David Thompson and I encouraged Jakob to spend the
minimal amount of time necessary focusing on doing polymorphism in a
clever way, as this is what Dave and I found was slowing us down in our
initial prototype code.

It's possible we could/should move to something else, but that's the
motivation there.

> The implementation of '<sshable-machine>' is doing what
> 'switch-to-system' and 'install-bootloader' in 'guix/scripts/system.scm'
> do, but in terms of data that can be sent with 'remote-eval'. I imagine
> the code will make more sense if you read both simultaneously.
>
> Okay, on to the test suite.
>
> My understanding of the system test suite (tests run with 'check-system'
> as opposed to those run with 'check') is that the meat of the test code
> exists in a G-Expression and should _not_ be interacting with the store
> of the machine running the test suite (i.e. that's the reason we're
> using marionettes in the first place). 'gnu/tests/install.scm' seems to
> be somewhat of an exception, and because the code in '(gnu machine)'
> depends heavily on having access to a store, I've tried to extend what's
> done in 'gnu/tests/install.scm' so that my tests have access to store
> while instrumenting the marionettes.
>
> To be specific, the chicken and egg scenario I'm working around is that
> the SSH daemon on the marionette needs to be running in order for
> 'deploy-os' to work, but I can't call 'deploy-os' from the test
> G-Expression because the store wouldn't be accessible then.
>
> My gut is telling me that this is absolutely the wrong way to go about
> it, though. 'call-with-marionette' is one of a couple red flags making
> me feel that way -- I don't think marionettes were meant to be started
> outside the context of a derivation...
>
> If anyone has suggestions on how I might go about properly testing this
> code, I would appreciate it very much.
>
> Another point about the test suite: the 'ssh-deploy-os' test fails, but
> it's a reproducible version of the issue outlined in [2]. I'd like to
> conscript some help from those more familiar with guile-ssh before
> breaking out the ol' RFCs myself.
>
> ...
>
> So, if anyone's in the mood to peek at what's been written so far and
> give feedback, that'd make my day. Doesn't have to be feedback related
> to the test code, either. Any sort of comment, be it regarding the code
> style or perhaps even some suggestions on improving the interface, would
> be appreciated.
>
> I'm going to continue to spend time on the "internals" of 'guix deploy'
> this coming week, incorporating any feedback I receive and ensuring that
> I have a framework to build upon when I extend this to more complicated
> scenarios like deploying to a VPS provider. After that, I'll tackle
> fleshing out the way deployments are specified (the file part. I'm
> holding off on the command-line tool part for right now).
>
> Signing off for now. Huge thanks to everyone for the warm welcome I
> received following my introduction email.
>
> Regards,
> Jakob
>
> [1]: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00145.html
> [2]: https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00114.html
> [3]: https://git.sr.ht/~jakob/guix/tree/wip-deploy

FWIW I could use the help as well; Jakob is hitting some areas of code
I'm not very familiar with and unfortunately I've been traveling for the
last two weeks for some family stuff.  I'm keeping in good contact with
Jakob and talking with him about problems but more community code review
will be helpful until at least next week when I'll be able to hit the
ground running again.

Jakob has been very communicative btw, and I'm really optimistic about
the state of the project!

 - Chris