certbot service experience

[Christopher Allan Webber]

I'm crossposting this to guix-devel, even though it's in reply to
guix-patches bug #26685 adding the cerbot service, because I think it's
more about my experiences with workflow and less about what might affect
that specific bug.  (If you reply on guix-devel, maybe remove the
debbugs address.)

Andy Wingo writes:

> Attached patch adds a certbot service extending nginx.  Only question
> is, how to ensure that nginx is runing when the certbot activation runs?
> In practice I bet this races so it's not a big issue but if there's a
> way to require nginx before activation, that would be nice.

So I got a server up and running using certbot and this.  Yay!  It went
pretty well, and I'm happy about it now that it's running, though I did
run into some bumps (which is okay, I ran into bumps doing this on
Debian too, I just think now's a good time to reflect on the experience
and see if we can do anything better... plus maybe it's useful to
explore for someone else too).  I figured I'd document what the process
was like for me.  Here's my observations:

 - I was surprised that I was prompted for an email while doing guix
   system reconfigure (and in being asked if I was willing to supply
   that info to the EFF so they can put me on their mailing lists, which
   I'm not against but was surprising to encounter in a reconfigure).
   IIRC this is the first time anything like that has happened.  I
   generally anticipate such a reconfigure to be "prompt-less".
   I wonder if there should be an email field we can provide, so that it
   doesn't do this prompt?  Granted, it's for the first time run only.
   Maybe it's okay to do things as they are, but we should document it?
   I dunno?

 - Not specifically something to do with this package, but nginx wasn't
   coming up at a few points (btw, default nginx configuration won't
   work, because it points at certificate filepaths that we don't
   "automatically" put in place).  When nginx doesn't come up, there's
   pretty much no information as to why; nothing in the shepherd logs,
   nothing useful from shepherd itself ("shepherd: Service nginx could
   not be started."), and nothing in the nginx logs either.

 - Getting the certbot stuff to work basically involved three stages:

   1) Enable nginx without cert certbot, and point the root somewhere
      which will be the same place you'll set certbot-configuration's
      webroot at in the next step.  This looks something like the
      following:

        (service nginx-service-type
          (nginx-configuration
           (server-blocks
            (list
             (nginx-server-configuration
              ;; Replace these with your own domain and web root
              (server-name '("test.activitypub.rocks"))
              (root "/srv/activitypub.rocks/site/")
              ;; Note that you have to disable https and any certificates
              ;; esp because our nginx config defaults to pointing at keys
              ;; that probably aren't there.  Without doing this nginx
              ;; will die mysteriously.
              (https-port #f)
              (ssl-certificate #f)
              (ssl-certificate-key #f))))))

      (You can maybe do this in the same phase as the next step but
      there's a risk of a race condition I think.)

      Anyway now do "guix system reconfigure" and nginx should come up
      and serve your stuff *without* https.

  2) Enable the certbot-service-type (and mcron-service-type if you
     haven't already):

       (service certbot-service-type
                (certbot-configuration
                 ;; Replace these with your own domain and web root
                 (hosts '("test.activitypub.rocks"))
                 (webroot "/srv/activitypub.rocks/site/")))
       ;; if you don't have an mcron service already
       (service mcron-service-type)

     Okay, with those added, run "guix system reconfigure" again.
     You'll be prompted for your email address and about whether or not
     you want to join the EFF mailing lists, so watch for that.

  3) Okay hopefully that went successfully!  It should say.  Assuming it
     did, *now* we can add the keys appropriately to the nginx config.

       (service nginx-service-type
                (nginx-configuration
                 (server-blocks
                  (list
                   (nginx-server-configuration
                    ;; Again, adjust to your site
                    (server-name '("test.activitypub.rocks"))
                    (root "/srv/activitypub.rocks/site/")
                    (ssl-certificate
                     
"/etc/letsencrypt/live/test.activitypub.rocks/fullchain.pem")
                    (ssl-certificate-key
                     
"/etc/letsencrypt/live/test.activitypub.rocks/privkey.pem"))))))

     Reconfigure and cross your fingers!

  4) At this point I was surprised that it seemed like nginx should have
     been working with https since everything was in place, but I
     couldn't access it from my browser over https.  Frustrated, I
     restarted the server.

     And then it worked! :)

So, this involved reconfiguring, reconfiguring, reconfiguring, and then
a restart, then it worked for me.  (Well, plus a few reconfigures where
nothing worked at all because I broke things of course. ;))  I wonder if
that can be improved?

That said, it's still really exciting to be able to describe these
things declaratively, and to have Guix take care of keeping things
renewed for me. :)  Excited to have this landing, and to be that much
closer to doing server deployment with GuixSD!