Friendly discussion about (package-initialize)

[Radon Rosborough]

Hi all,

I'd like to have a friendly discussion about (package-initialize).
Specifically, about this comment that I'm sure we've all seen from
time to time:

    ;; Added by Package.el.  This must come before configurations of
    ;; installed packages.  Don't delete this line.  If you don't want it,
    ;; just comment it out by adding a semicolon to the start of the line.
    ;; You may delete these explanatory comments.
    (package-initialize)

In this email, I'm going to explain why I think the current behavior
of package.el is flawed, and propose some alternative behaviors that I
believe would be better. If the community thinks that these
alternatives are reasonable, I would be interested in contributing
relevant patches. I know that this argument has the potential to be
contentious, so I'm going to try to be as objective as possible, and
stick strictly to the technical points.

Since this is a rather long email, I've arranged it into sections.
Feel free to skip or skim the parts that you already know. I've tried
to be quite verbose so as to create a comprehensive and authoritative
summary of the issue for people who may not be as familiar with Emacs
packaging.

################################################################################
# WHAT DOES PACKAGE.EL DO?

Since version 24.1 [1], Emacs has shipped with a built-in package
manager called package.el [2]. Like other package managers such as
Quelpa [3], Cask [4], el-get [5], Borg [6], and straight.el [7], the
primary function of package.el is to acquire Elisp code, preprocess it
(e.g. byte-compilation, autoload generation), and make it available to
Emacs.

The acquiry and preprocessing of Elisp code is not relevant to this
discussion. What is important is how the code is made available to
Emacs. The general process (called "activating" a package) is
two-fold:

* add the package's files to the load-path and Info-directory-list
* evaluate the package's autoloads, so that running an autoloaded
  function loads the appropriate files from the package

Importantly, these steps must be run every time Emacs is initialized.
Installing Elisp code on the filesystem does nothing, unless it is
somehow added to one of Emacs' default load-path entries. Before a
package is activated, attempts to `require' its features will fail,
and its autoloads will not be available.

Different package managers have different opinions on how and when
packages ought to be activated. For example, in straight.el [7],
packages are activated one at a time as they are referenced in the
user's init-file. However, the most common approach is for all
packages to be activated at the same time. This is what package.el
does; it provides a `package-initialize' function which fulfills this
purpose.

Therefore, absent any special treatment, one might expect a user's
init-file to go something like this:

    ... code to configure package.el ...
    (package-initialize '(... some list of packages ...))
    ... code to configure the packages that were initialized ...

But users do not typically think in this way. They expect that
installing a package is a permanent operation, and that no further
action is required in order for that package to continue to be
available. For this reason, package.el keeps track of which packages
have been installed already, and it does this outside the init-file
(in the state of the ~/.emacs.d/elpa directory). When
`package-initialize' is called, all the installed packages are
activated. You can request package.el to activate only a subset of the
installed packages, but this is not the usual mode of operation. Thus,
the user's init-file actually looks like this:

    ... code to configure package.el ...
    (package-initialize)
    ... code to configure the packages that were initialized ...

Notice that the call to `package-initialize' must come *after* any
code that configures package.el, but *before* any code that configures
the packages that were initialized. This is important! If you try to
configure package.el after `package-initialize', your configurations
will not affect how package activation takes place. And if you try to
configure packages before `package-initialize', then variables,
functions, and so on are not going to be properly defined. (This might
not be a problem, depending on how you handle your customizations. But
it certainly does make things more complicated.)

################################################################################
#### WHAT IS THE PROBLEM?

The problem is when users forget to initialize the package management
system before configuring their packages. In other words, their
init-file just looks like this:

    ... code to configure packages ...

with no mention of `package-initialize' anywhere. Obviously this will
not work, because the packages are never activated. The "correct"
solution is to just call `package-initialize' in the init-file. But it
is hard to teach users to do this, and the resulting errors have been
a common problem for new Emacs users [8].

Thus, there was a discussion [8] on emacs-devel about how to arrange
for things to work "correctly" in the absence of a user who knows what
they are doing. The eventual outcome of that discussion is the subject
of the next section.

################################################################################
#### WHAT IS THE CURRENT SOLUTION TO THE PROBLEM?

The initial attempt was to have Emacs call `package-initialize'
automatically during initialization. However, this is not so easy as
it sounds. If Emacs calls `package-initialize' before loading the
init-file, then users cannot customize package.el in their init-file
anymore. And if Emacs calls `package-initialize' after loading the
init-file, then users cannot customize their installed packages in
their init-file anymore, unless they cause their customizations to be
loaded after the end of initialization using `after-init-hook'.

The only reasonable way for things to be set up is for
`package-initialize' to be called in the user's init-file, but before
package customizations are done. Thus, a patch [9] was suggested that
causes Emacs to write a call to `package-initialize' into the user's
init-file at startup, if it was not there to begin with. This was
shortly merged and released in Emacs 25.1. Note that the behavior is
inhibited in 'emacs -Q' mode.

################################################################################
#### JUSTIFICATIONS AND COUNTERARGUMENTS

> So what are you saying is wrong with the current behavior?

It modifies the user's init-file without asking.

> What's wrong with modifying the user's init-file? After all, Custom
> has been doing that for decades.

The difference is that Custom only modifies your init-file when you
ask it to save your customizations. In other words, it only modifies
your init-file when you ask it to. On the other hand, package.el does
it at Emacs startup without confirmation!

> OK, so it's different than Custom. What's the problem, though?

There are several issues:

* Many users want to write their init-files by hand. If they want to
  initialize the package management system, they want to do it
  themselves, and they want to put in the code by hand so that they
  know it will run in the right place.
* Many users have a modular Emacs configuration in which init.el loads
  their other customization files, one of which is usually deals with
  the package management system (including calling
  `package-initialize'). The result is that package.el sticks a
  superfluous, duplicate `package-initialize' into the root init-file.
* It's confusing. New users have no idea why Emacs is modifying their
  init-file, and a proper explanation is not given.
* It's extremely error-prone. Modifying the init-file is an
  irreversible operation, and it means that any bugs in this part of
  package.el are immediately 100x more frustrating.
* Not everyone uses package.el. It's annoying to have to explicitly
  put in configuration to disable the existing package management
  system, if you want to use an alternative package manager.

> Slow down there! Let's take this one point at a time. You said that
> some users want to write their init-files by hand. But if they do
> that, then can't they just put a call to `package-initialize' in
> their init-files, and that will prevent the issue?

No, because (1) they might be using a modular Emacs configuration that
calls `package-initialize' somewhere other than init.el, and (2) they
might be using a package manager other than package.el, and therefore
have no need of calling `package-initialize' at all.

> Well, what's wrong with them just putting a commented-out call to
> `package-initialize' in their init-file, like the inserted comments
> suggest?

This is ugly and looks ridiculous. Is requiring a magic comment in the
init-file to prevent unnecessary code from being inserted on Emacs
startup really the best possible user experience?

> Then what would be a better user experience?

Let's wait until the ALTERNATIVES section for that :)

> Wait, you said that it was a problem if `package-initialize' was put
> somewhere other than in the init-file. But if it's called during
> startup, then package.el doesn't run the logic to insert the a call
> into the init-file. So what's the issue?

The problem is that if there is an error while loading the init-file,
then the call to `package-initialize' still gets inserted, if init
hadn't gotten to the point where the actual call to
`package-initialize' was supposed to happen. This is actually
terrible, since it immediately makes debugging much more complicated:
your init-file is being changed without your knowing it, while you're
trying to debug initialization!

> OK, but that only happens when there's an error during init. We
> could make it so that the `package-initialize' happens only after a
> successful init.

That doesn't solve the problem, because there are all kinds of
circumstances where you have a "successful" init even when the user's
init-file was not loaded. For example, many modular Emacs
configurations will sustain errors in one or more modules using
`condition-case', and report the errors as warnings for an improved
debugging experience. We're back to the beginning.

Besides, often one wants to test something in a plain Emacs but not in
'emacs -Q'. In this case, the call to `package-initialize' will still
get inserted.

> How often does this really happen?

At least 20 times in my use of Emacs, and I get progressively more
annoyed every time. The existence of threads like [10] [11] indicates
that there are other people who don't like this kind of behavior.

Regardless of how we set up this system, it will never be possible to
cover all the edge cases. Once in a while, there will be some
sufficiently unexpected use case where an errant `package-initialize'
gets inserted and screws something up. Since there are better
solutions (see ALTERNATIVES), I think it's best to sidestep the whole
problem by not taking this approach in the first place.

> You said the system was confusing. Why?

The comments that come with the inserted `package-initialize' tell the
user "don't delete this line" without explaining the consequences if
you do. The user inevitably will try deleting it, since they didn't
ask for it to be added in the first place. But it will come back next
time, prompting annoyance and consternation.

The comments also advise "this must come before configurations of
installed packages", without explaining why or anything about how
package activation works. Also, the whole idea of "if you don't want
this, comment it out, but don't remove it" is bizarre; I've never seen
anything like it in any other program's configuration system.

> Can't that be fixed by clearer comments?

Yes, but the fact remains that a program inserting text into its own
configuration file (except in the case where the program has a
semantic understanding of the whole file, like for YAML) is highly
nonstandard. I don't know of any other program that does something
like this. As I said before, there are ALTERNATIVES that are more
standard, and these will not violate the principle of least surprise
as badly.

> Why are you so concerned about bugs in package.el? Is that really
> likely?

I trust that the folks here on emacs-devel will do a good job in
preventing bugs from creeping in. But nevertheless, I think putting in
this kind of logic is simply a bad idea from a software engineering
perspective. There will always be bugs, and we want to minimize the
impact of the ones that inevitably do pop up. Having code that
automatically modifies the init-file at startup is setting up the
potential for a bug with a huge impact that will be extremely annoying
and dangerous to work around. I think this is especially important to
keep in mind since Emacs releases are extremely infrequent, and so
we're going to have to deal with bugs in any release version for a
long, long time.

> What's this about alternative package managers? Shouldn't Emacs
> include special support for the built-in one? After all, that's why
> it's built in.

Sure, I think it's reasonable for Emacs to provide special support for
packages which are built in. But there's such a thing as going too
far. And I personally think that you've gone too far in providing
special support when that support actively makes it *more* difficult
to swap out an alternative implementation.

If you use an alternative package manager (and many people do), then
you really don't want package.el to be automatically activating things
when it's not asked to. You'll end up with duplicate load-path
entries, conflicting versions, needlessly slow init times, and more:
fun times (I've been there).

And you *really* don't want package.el to be modifying your init-file
that configures your alternative package manager so that it also
activates package.el! Having such code inserted when it's actively
harmful is a really frustrating experience.

################################################################################
#### ALTERNATIVES

There is lots we can do to improve the situation. Some changes will be
harmless and backwards compatible, while others would have to be
rolled out carefully if we decide to follow them.

* Don't have package.el insert `package-initialize' into the init-file
  under any circumstances. Instead, when Emacs is started up for the
  first time (when not in 'emacs -Q' mode), if no file exists at
  ~/.emacs.d/init.el, create that file with a basic template.

  This template would have a call to `package-initialize', and also
  some explanatory comments saying where you should put configuration
  of package.el versus configuration of packages, and why.

  This is my favorite option. It is a well-established (and,
  importantly, *unsurprising*) tradition for programs to come with
  skeleton configuration files that the user can customize as they
  desire. Besides, this would allow us to make a better experience for
  new users (remember all the complaints about "bad defaults"?)
  without making actual changes to the defaults and breaking backward
  compatibility.

* Don't have package.el activate packages at startup. I would actually
  prefer this, but I understand that it's not likely to happen. There
  are two advantages: (1) users of alternative package managers don't
  have to explicitly disable package.el all the time (when testing
  without their configuration file, for example), and (2) the behavior
  is much less "magic".

  This would break backward compatibility, but mostly of expectations:
  only people who put their entire package configuration code into
  `after-init-hook' would have their configurations broken (and the
  fix would of course be trivial). If we implemented the skeleton init
  file, then the experience for new users would not change.

* Improve the explanatory value of the comments that are inserted by
  package.el. I really, really think that the whole behavior of
  automatically inserting code into the init-file should be killed
  (with fire), but if the community really disagrees with me here, at
  least the comments should make an attempt at explaining what's going
  on and what the potential caveats are. This would be a trivial
  change with no downsides.

* Make it trivial to disable package.el. Right now, the only foolproof
  way to prevent the init-file from being modified by package.el is to
  place multiple advices on internal functions, and to do this as
  early as possible during init just in case there is an error, and to
  also be really careful any time you load Emacs without your
  init-file, and to double-check for the inevitable times when you
  can't prevent package.el from doing stuff.

################################################################################
#### CONCLUSION AND NEXT STEPS

I've argued strongly for removing the auto-insertion of
`package-initialize' code into the init-file by package.el, in favor
of providing a skeleton init-file. Although I've addressed a number of
counterarguments, I'm sure there are other factors I haven't
considered. Let me know what you think. Is this reasonable? If not,
why not? If so, how can I help move it forward?

Best regards,
Radon Rosborough

[1]: https://lists.gnu.org/archive/html/info-gnu-emacs/2012-06/msg00000.html
[2]: https://www.gnu.org/software/emacs/manual/html_node/emacs/Packages.html
[3]: https://github.com/quelpa/quelpa
[4]: https://github.com/cask/cask
[5]: https://github.com/dimitri/el-get
[6]: https://github.com/emacscollective/borg
[7]: https://github.com/raxod502/straight.el
[8]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01016.html
[9]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01055.html
[10]: 
https://www.reddit.com/r/emacs/comments/56fvgd/is_there_a_way_to_stop_emacs_from_adding_the/?ref=share&ref_source=link
[11]: 
https://www.reddit.com/r/emacs/comments/4x655n/packageselectedpackages_always_appear_after/
[12]: https://github.com/raxod502/straight.el/issues/73