Re: Building around Guix, some observations

[Ludovic Courtès]

Hi Christopher,

Woow, that’s a lot of great work and a lot of questions!  :-)

Christopher Baines <address@hidden> skribis:

> Specifically, the paragraphs below are about:
>  - Bundler and gem packages
>  - Using a specific revision of Guix
>  - Shared networking in guix system container
>  - Packaging Ruby on Rails applications
>  - Configuration of service startup
>  - Database connections and utilities

It’s great to see that you’ve gone pretty far on these questions.  I’ll
skip the Ruby part since I’m ignorant (I think Dave Thompson uses Ruby
at work, so he might have ideas) and comment on the others.  Perhaps we
should start one thread per topic though.

> As these modules are intended to be used with the specified version of
> guix, there is a script guix-pre-inst-env that I believe does a good job
> now of making it possible to do this regardless of the environment its
> run in. To do this, guix environment is used twice, once to make the
> specific version of guix available, and again to use that version of
> guix to setup the desired environment. Then there are some changes to
> environment variables. XDG_CONFIG_HOME is set to /dev/null, as this
> prevents guix from loading package definitions, as this would lead to
> nondeterministic behaviour. Also, rather than using the --pure argument
> to guix environment, I ended up unsetting variables in bash as this made
> it possible to pass through variable values by not unsetting them. Are
> there any other approaches to using a specific revision of guix?

In commit 8a9cffb202414b20081910115ba76402924bdcdd Leo documented a
simple way of telling ‘guix pull’ how to get a specific revision.

The other possibility is of course to build Guix from Git, and do that
using ‘guix environment guix’, which seems to be essentially what you’re
doing.

The ultimate goal is to make it easier through the new ‘guix pull’
(discussed notably at <https://bugs.gnu.org/22629>.)  People should be
able to do:

  guix channel pull master COMMIT
  guix channel describe

Things like that.  Roughly, it should provide the same flexibility as
building from a Git checkout, but just as easy as running ‘guix pull’.

> One of the earliest changes I made was to change the guix system
> container command to share the host network in the same way as guix
> environment does. I recently made an attempt of making this
> configurable, like the guix environment --network option [3]. When
> writing this, it felt quite brittle, but when I remember to add the
> right option, it is very useful. Annoyingly, just adding and removing
> the --network option is insufficient, as you can't have a
> static-networking-service when you are sharing the host network, but,
> something still needs to provide 'loopback, so I add a
> pretend-loopback-service to the system by default. The differences
> between building operating-systems to run in containers or not are
> handled in some places, so the differences in networking services could
> be handled similarly?

Probably!  In (gnu system vm), ‘virtualized-operating-system’ returns an
OS config adjusted so that it can be used in a VM:

  http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/system/vm.scm#n367

Perhaps we should provide a ‘container-operating-system’ that does
something similar, and have ‘guix system container’ automatically use
it?

> Some of the systems that I have written so far, e.g. the
> operating-system in (gds systems govuk publishing-e2e-tests) configure
> the startup of some services by using a specific record
> (service-startup-config). This is currently being used to hook in the
> creation of databases and database users, as well as running scripts to
> seed data before the service starts. Has anyone else had a similar need
> to do this, as this might be a pattern that could be beneficial to apply
> to services within Guix?

Usually services use “activation snippets” for pre-startup
initialization, like:

  http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/databases.scm#n90

Would it be an option here?

Alternately, I imagine you could write your own initialization steps as
Shepherd services and introduce them as dependencies of the Shepherd
services that need to be started after this initialization step.  Does
that make sense?

> Within the (gds services utils databases) module hierarchy there are
> modules for working with PostgreSQL, MongoDB, Redis and MYSQL/MariaDB
> databases. For each, I created a record type to describe the details of
> a connection to the database, and wrote a generic function
> database-connection-config->environment-variables to get the environment
> variables that usually correspond to that configuration. Also in these
> modules are functions to create G-expressions for performing common
> actions, e.g. ensure this user exists, load data from this file. I get
> the feeling that both of these tools, the connection records, and the
> functions for common operations might be generally applicable?

The env. vars in
<https://github.com/alphagov/govuk-guix/blob/spike-systems-for-local-development/gds/services/utils/databases.scm>
could be added to /etc/environment by extending
‘session-environment-service-type’.

Things in
<https://github.com/alphagov/govuk-guix/blob/spike-systems-for-local-development/gds/services/utils/databases/postgresql.scm>
to initialize the PG database could be handled by
‘postgresql-service-type’, though perhaps it should be optional?

Databases are stateful by definition though, so there are limits as to
what can be done in a purely declarative way.

> Looking forward, I'm interested in how I can utilise Guix and the store
> for managing data, specifically dumps from databases. I've got some
> crude packages working that can be used with the G-expressions for
> loading data in to a database, but what is missing is a way to transform
> the data in a modular way. For example, using Guix to orchestrate the
> process of loading the data in to a database, then dumping it back out
> in a different form. I'm not aware of any packages that use databases at
> build time, but I'd be very interested in doing this. Package builds
> using databases also has an additional benefit, as databases are often
> used for running tests at build time. There are some utilities to make
> this easier, I've used pg_virtualenv with Debian packages before, but as
> Guix has a way to reason about services, some way to integrate that with
> package build processes would be amazing.

I guess I don’t know enough about these issues to provide meaningful
input.

Thanks for your feedback!

Ludo’.