Re: Building Docker images of GuixSD

[Chris Marusich]

Hi Ludo and others following along,

I've run GuixSD in a Docker container and returned to tell the tale!
The attached patch requires a lot of cleaning up (e.g., proper ChangeLog
entry, update documentation, remove some unnecessary imports and debug
messages that are probably still in there), so I'm taking a moment to
share my results and ask for feedback before committing to spending more
time on this.

Run GuixSD in Docker
====================

The attached patch makes it possible to build a GuixSD Docker image from
an operating system configuration file.

You can build your own like this:

1) Apply this patch to 3b2fa4787938a408fab27ef7b3bc1302b6b6a805.

2) Build an image (I used the attached file "very-bare-bones.scm"):

    ./pre-inst-env guix system disk-image -t docker very-bare-bones.scm

3) Copy the resulting image onto a host that has Docker installed.

4) On the host with Docker, load the image and note the image ID:

    docker load < pw3d4r4m1x9yc3d1kg9x3y6abdzq9z7g-docker-image.tar.gz

5) Run a Docker container from the image, and note the container ID:

    docker run --privileged -d -e GUIX_NEW_SYSTEM=/var/guix/profiles/system 
--net host --entrypoint /var/guix/profiles/system/profile/bin/guile 
dcaa8fb677c7 /var/guix/profiles/system/boot

6) Run a shell in the container, install a package, and use it:

    docker exec -it -e USER=alice -u alice fb06fdcd3a0d 
/run/current-system/profile/bin/bash --login

7) Install a package and use it:

    address@hidden /$ guix package -i hello
    ...
    Creating manual page database for 1 packages... done in 0.110 s
    1 package in profile
    address@hidden /$ guix package --list-installed
    hello       2.10    out     
/gnu/store/wf65hjwqwpz4wllasn63zysi5irql2sx-hello-2.10
    address@hidden /$ hello
    Hello, world!

Pretty neat!

How Useful Is This?
===================

Using Guix, it was already possible to generate Docker images using
"guix pack".  For example, I could have just generated a Docker image
from the GNU Hello package, created a container from that, and then run
"hello" from that container.  What does running GuixSD in Docker give us
that we don't have already?  At a minimum, it gives us the following:

* The ability to define what service(s) should run in the resulting
  Docker container, including their configs and start/stop scripts.
  
* Since the Docker image is generated from a GuixSD operating system
  configuration file, the rules for defining and configuring services
  are the same as always.  You don't have to learn anything new.

* If you want to run Guix on a system to which Guix hasn't been ported
  (like macOS) but your system does run Docker, now you can run Guix on
  that system by running it from a GuixSD Docker container.

Is this helpful?  Is it worth polishing up and maintaining?  I'm not
entirely sure, and I'd like to know what you think.

For the first two bullet points, that's nice, but instead of using a
full-blown OS and relying on the Shepherd for process management in this
case, would it be simpler to just provide a way to easily bundle
start/stop scripts inside of the packs produced by "guix pack"?  An
enterprising user can probably do this today by simply defining a
package that builds start/stop scripts for a given service; the user
would then just need to include that package in the pack.  The downside,
I guess, is that you can't re-use the service-specific stuff that you
can normally use in a GuixSD operating system configuration file.

For the third bullet point, I don't know of any other reasonable way to
get Guix working in Docker (although one could certainly run Guix in a
VM using a technology other than Docker, such as QEMU).  To run Guix,
you need the Guix daemon running somewhere, right?  And the Guix daemon
requires that certain build users exist.  It might require other things
from its environment, too.  In any case, you can't just run "guix pack
-t docker guix" and expect the "guix" command to work in the container
(I tried, and it doesn't work).  You have to take additional measures,
like create build users, at which point it seems easier to just put all
of GuixSD into a Docker image.  That's what my patch lets you do.

What do you think?  Is this worth polishing up and maintaining?

Problems I Noticed
==================

Now I'll mention some specific problems I've noticed while running
GuixSD in a Docker container.  First, I saw this while the Docker image
was being generated:

    tar: Removing leading `/' from member names
    tar: Removing leading `/' from hard link targets
    tar: ./dev/log: socket ignored

It's fine that we remove the leading '/' from member names, since it
looks like the tarball will be extracted relative to '/'.  I think the
same is true for the hard link targets.  However, because tar ignored
'/dev/log', that socket is missing in the Docker image.  I don't know if
that will interfere with syslogd, but it sure doesn't sound good.

Second, I noticed the following error in the Guix daemon's logs.  It
might be benign, since package installation worked fine, but I'm not
sure what it means or how to debug it:

    error in finalization thread: Bad file descriptor

Third, I noticed that the shepherd failed to start syslogd and nscd (and
user-homes, although I wasn't as concerned about that because the home
directory for alice did in fact get created).  I understand that, due to
the way Docker works, some services are either not required (like
networking) or might require modifications to "behave well" in a Docker
container.  However, I didn't think syslogd and nscd would fall into
either of those categories, so I was surprised that they failed to
start.  The only relevant debug information appears to be the following
messages in the Shepherd logs (/var/log):

    2017-11-09 06:41:27 Service user-homes could not be started.
    2017-11-09 06:41:32 Service nscd could not be started.
    2017-11-09 06:41:37 Service syslogd could not be started.

I thought maybe syslogd wasn't working because /dev/log hadn't been
created in the Docker image, so I tried creating it manually.  However,
that didn't help; the Shepherd still couldn't start syslogd.

Fourth, I wasn't able to run GuixSD in a Docker container without
supplying the "--privileged" option.  GuixSD writes to sysfs during boot
(I don't know why, but the details are apparently in
guix/gnu/build/activation.scm), so the only way to get GuixSD to start
is to run the container in privileged mode.  This is unfortunate,
because privileged mode sounds quite dangerous for a lot of reasons.
For example, if both GuixSD in the Docker container and the host
operating system attempt to control the underlying hardware at the same
time, bad things might happen.

Thanks for reading this far.  I look forward to hearing your thoughts!

-- 
Chris

0001-Make-it-possible-to-build-GuixSD-docker-images.patch
Description: Text Data

very-bare-bones.scm
Description: Binary data

signature.asc
Description: PGP signature