Re: Implementing guix system rollback / switch-generation

[Ludovic Courtès]

Hi Chris,

Chris Marusich <address@hidden> skribis:

> Basically, I think there should be a command like "guix system
> roll-back" which does the opposite of "guix system reconfigure
> config.scm".  The rollback command should not require an operating
> system configuration file.  I think this would be better than the
> current rollback methods.  What do you think?

It would definitely be a welcome addition!  The refactoring that was
done to add ‘guix system list-generations’ was in the same spirit.

> As I understand it, when I invoke "guix system reconfigure config.scm",
> the following things happen (in guix/scripts/system.scm):
>
> * A new system is built in the store (e.g.,
>   /gnu/store/1qkdd4glvqjqf7azqniis7abkf7v1lng-system).
>
> * A new symlink is created in /var/guix/profiles (e.g.,
>   /var/guix/profiles/system-7-link), which points to the system in the
>   store.
>
> * The /var/guix/profiles/system symlink is updated to point to the new
>   symlink.
>
> * The new system's activation script is run, and the Shepherd services
>   are upgraded.
>
> * A new grub.cfg is copied to /boot/grub/grub.cfg.  This new grub.cfg
>   updates the default menu entry to point to the new system, and it adds
>   the previous system to the list of previous generations.
>
> At this point, you are running a new system.  I am not sure if you need
> to reboot to truly upgrade, but judging by the implementation of the
> upgrade-shepherd-services procedure, it seems like you might sometimes
> need to reboot to "really" complete the upgrade process.

Indeed.  Service upgrade is conservative, so it will not stop running
services, because it cannot know if this is something the user wants and
if it is safe; only services that are not currently running are loaded
and started upon reconfigure.

  http://bugs.gnu.org/22039

To improve on this, I think it should be possible for Shepherd services
to provide an ‘upgrade’ method in addition to start/stop, for those
services that can be upgraded with no downtime (there aren’t so many of
them, though.)

> * Manually update the /var/guix/profiles/system symlink to point to a
>   previous generation, e.g., /var/guix/profiles/system-5-link.
>
> If you do this, it seems the running system will not actually be rolled
> back.  The system pointed to by /run/current-system will not change,
> since it points to a store path rather than to the
> /var/guix/profiles/system symlink.

‘guix system reconfigure’ could explicitly update /run/current-system to
point to the old system.

> In addition, because /boot/grub/grub.cfg has not been modified, when
> you reboot, GRUB will still automatically select and boot the system
> from which you wanted to roll back.  Also, the previous system's
> activation script will not be run, and the shepherd services will not
> be downgraded.  With that in mind, I'm not really sure what manually
> flipping this symlink actually accomplishes.

Not much, indeed.

The problem with system rollback, as you’ve seen, is that we lack
information about the old system, such as what its activation script is
and what its Shepherd services are.

We could add the activation script’s file name to the ‘parameters’ file
that you see in the result of ‘guix system build foo.scm’.  But it would
be hard to add a forward-compatible yet complete description of the
Shepherd services there.

> * Use the emacs interface to update that symlink.
>
> This is the same as above, except that emacs does the symlink flip for
> you.  It does not seem to actually roll back the system.  The elisp code
> seems to assume that flipping the symlink is sufficient.  Maybe that
> works for changing user profile generations, but it seems to be
> insufficient for system generation changes.
>
> * While booting, at the GRUB menu, manually select a previous
>   generation.
>
> This is the only method that seems to actually put the system back into
> a previous state, and it appears to do so correctly.  I've used this
> method to save my systems a few times from bad upgrades, which is great.
> The only downside to this method is that it doesn't modify the grub.cfg
> file: until you reconfigure your system again, every time you restart,
> GRUB will continue to automatically select and boot the newer,
> problematic system from which you wanted to roll back.

ISTR that GRUB has a mechanism to record the last selected menu entry
and to use that as the next default entry.

Now, it’s not always what one would want to do.

However, ‘guix system roll-back/switch-generation’ could generate a
grub.cfg where the default menu entry points to whatever old generation
has been selected.

> I'd like to implement a rollback mechanism that lets you run a single
> command which actually does the opposite of "guix system reconfigure".
> I've looked at the code in guix/scripts/system.scm, and at first blush
> this seems like a straightforward task.  However, I've noticed that the
> switch-to-system procedure requires an operating system configuration
> file to do things like get the system's activation script and generate
> the new grub.cfg file.  Surely a user should not have to specify the
> previous system's operating system configuration file on the command
> line in order to roll back the system.  Is it possible to obtain these
> things (e.g., the service activation script, the previous system's
> grub.cfg) without requiring the user to supply the previous system's
> operating system configuration file on the command line?

Currently it’s not possible to obtain the activation script of past
generations, but as I wrote above, it’d be doable.

It’s not possible to obtain past grub.cfg files, but that’s not a
problem: we can always regenerate a new grub.cfg.

What seems more difficult to me is Shepherd services.  Maybe we could
store in the system output (result of ‘guix system build’) an sexp
representation of (part of) our <shepherd-service> records:

  (shepherd-service
    (provisions (x y z))
    (requirements (a b c))
    (start-script "/gnu/store/…-start-foo.scm")
    (stop-script "/gnu/store/…-stop-foo.scm")
    …)

Then ‘upgrade-shepherd-services’ could start from this simplified
representation instead of using the full-blown <shepherd-service>
objects, and thus could work both when instantiating a new generation
and when rolling back.

> More generally, are people satisfied with the way system rollback is
> currently implemented in GuixSD?

Personally I’m not fully satisfied, but it’s true that it covers my main
use case, which is to recover from a broken update.

I had never thought about live “downgrade” of services when rolling
back, because the only times where I’ve wanted to roll back is right
after booting (or trying to boot ;-)) into a new system generation.

> Do you think that the mechanism I'm proposing is a bad idea?  I'd hate
> to try to implement something that nobody else thinks is needed.

I think having basic delete-generations, switch-generations, and
roll-back sub-commands would be definitely welcome.

As a first step, switch-generations/roll-back commands could simply
update the symlinks and regenerate grub.cfg.

Milestone #2 would be running the previous system’s activation script,
which installs /run/current-system and adjust the set of users and
groups.

Milestone #3 would be live service downgrade, as you describe.

Thoughts?

Ludo’.