Building around Guix, some observations

[Christopher Baines]

Hello Guix,

Over the last few months, I've been experimenting with setting up
packages, services and systems around the software used by GOV.UK.

This is not really related to Guix development, but I'd be very
interested if anyone has any feedback about the approach I've taken so
far, and if anyone knows of anything similar.

I've had two objectives with this, to learn more about Guix, and to make
my day to day work (which is working on the applications) easier, and
it feels like I've been doing quite well with both so far.

You can find the code here [1], which I'm using against a specific
revision of Guix with some patches that can be found here [2].

Specifically, the paragraphs below are about:
 - Bundler and gem packages
 - Using a specific revision of Guix
 - Shared networking in guix system container
 - Packaging Ruby on Rails applications
 - Configuration of service startup
 - Database connections and utilities

The (gds packages utils bundler) module includes utilities to extend a
package that contains a Gemfile and Gemfile.lock in the source, and
handles the downloading of the gem packages, and the installation in the
store. This is definitely not a great long term approach, but it
provided a quick way to get these services working.  I'd be interested
in hearing other approaches that people have taken when dealing with gem
packages and bundler?

As these modules are intended to be used with the specified version of
guix, there is a script guix-pre-inst-env that I believe does a good job
now of making it possible to do this regardless of the environment its
run in. To do this, guix environment is used twice, once to make the
specific version of guix available, and again to use that version of
guix to setup the desired environment. Then there are some changes to
environment variables. XDG_CONFIG_HOME is set to /dev/null, as this
prevents guix from loading package definitions, as this would lead to
nondeterministic behaviour. Also, rather than using the --pure argument
to guix environment, I ended up unsetting variables in bash as this made
it possible to pass through variable values by not unsetting them. Are
there any other approaches to using a specific revision of guix?

One of the earliest changes I made was to change the guix system
container command to share the host network in the same way as guix
environment does. I recently made an attempt of making this
configurable, like the guix environment --network option [3]. When
writing this, it felt quite brittle, but when I remember to add the
right option, it is very useful. Annoyingly, just adding and removing
the --network option is insufficient, as you can't have a
static-networking-service when you are sharing the host network, but,
something still needs to provide 'loopback, so I add a
pretend-loopback-service to the system by default. The differences
between building operating-systems to run in containers or not are
handled in some places, so the differences in networking services could
be handled similarly?

Many of the services are using the Ruby on Rails application
framework. I've had some difficulty trying to get these applications
running with the code still in the store, as they use the location of
the code to determine where to try to write data. At the moment, copying
everything out of the store, and then changing the paths in the files to
ensure that the store isn't used is a reliable way of making the
services work, but its not very elegant. Does anyone have examples of
packaging rails apps in a more elegant way?

Some of the systems that I have written so far, e.g. the
operating-system in (gds systems govuk publishing-e2e-tests) configure
the startup of some services by using a specific record
(service-startup-config). This is currently being used to hook in the
creation of databases and database users, as well as running scripts to
seed data before the service starts. Has anyone else had a similar need
to do this, as this might be a pattern that could be beneficial to apply
to services within Guix?

Within the (gds services utils databases) module hierarchy there are
modules for working with PostgreSQL, MongoDB, Redis and MYSQL/MariaDB
databases. For each, I created a record type to describe the details of
a connection to the database, and wrote a generic function
database-connection-config->environment-variables to get the environment
variables that usually correspond to that configuration. Also in these
modules are functions to create G-expressions for performing common
actions, e.g. ensure this user exists, load data from this file. I get
the feeling that both of these tools, the connection records, and the
functions for common operations might be generally applicable?


Looking forward, I'm interested in how I can utilise Guix and the store
for managing data, specifically dumps from databases. I've got some
crude packages working that can be used with the G-expressions for
loading data in to a database, but what is missing is a way to transform
the data in a modular way. For example, using Guix to orchestrate the
process of loading the data in to a database, then dumping it back out
in a different form. I'm not aware of any packages that use databases at
build time, but I'd be very interested in doing this. Package builds
using databases also has an additional benefit, as databases are often
used for running tests at build time. There are some utilities to make
this easier, I've used pg_virtualenv with Debian packages before, but as
Guix has a way to reason about services, some way to integrate that with
package build processes would be amazing.

So, I'd be very interested if anyone has anything to add on the points
I've mentioned, any feedback about the approach I've taken so far, and
if anyone knows of anything similar.

Thanks,

Chris

1:
https://github.com/alphagov/govuk-guix/tree/spike-systems-for-local-development
2:
https://github.com/alphagov/gnu-guix/tree/spike-local-development-system-container
3:
https://github.com/alphagov/gnu-guix/commit/ba5d6220a00143189411dc3fbf0c3e7eebc794ab

signature.asc
Description: PGP signature