bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

Package: Emacs
Version: 27.0.50


There are many ways to make Emacs's startup slow.
One of them is to have many packages installed.  The growing popularity
of ELPA means that it's now common for users to have hundreds of ELPA
packages installed, which will easily cause Emacs's startup to take at
least 1s even with an empty ~/.emacs.  For users like me who (re)start
their Emacs session only rarely, this is not an issue, but for others it
can be an annoyance that's significant enough to try and circumvent it
by not using package.el (e.g. installing all their packages by hand or
using other packaging like DOOM).

If there's no objection, I plan on installing the patch below which
below lets users cut down package.el startup time by skipping package's
initialization and instead loading a single precomputed file which is
the concatenation of all the installed <pkg>-autoloads.el.  In my
experience this speeds up activation of package.el by a factor 5.
We could speed it up even further by byte-compiling this file, but this
has bumped into some corner cases problems so I'm sticking to
a non-compiled file for now.



        Stefan


diff --git a/doc/emacs/custom.texi b/doc/emacs/custom.texi
index a69888cdbd..c3cfaabb8d 100644
--- a/doc/emacs/custom.texi
+++ b/doc/emacs/custom.texi
@@ -2602,17 +2602,16 @@ Early Init File
 @cindex early init file
 
   Most customizations for Emacs can be put in the normal init file,
-@file{.emacs} or @file{~/.emacs.d/init.el}.  However, it is sometimes
-desirable to have customizations that take effect during Emacs startup
-earlier than the normal init file is processed.  Such customizations
-can be put in the early init file, @file{~/.emacs.d/early-init.el}.
-This file is loaded before the package system is initialized, so in it
-you can customize variables that affect the package initialization
-process, such as @code{package-enable-at-startup},
-@code{package-load-list}, and @code{package-user-dir}.  Note that
-variables like @code{package-archives} which only affect the
-installation of new packages, and not the process of making
-already-installed packages available, may be customized in the regular
+@file{.emacs} or @file{~/.emacs.d/init.el}.  However, it is sometimes desirable
+to have customizations that take effect during Emacs startup earlier than the
+normal init file is processed.  Such customizations can be put in the early
+init file, @file{~/.emacs.d/early-init.el}.  This file is loaded before the
+package system and GUI is initialized, so in it you can customize variables
+that affect frame appearance as well as the package initialization process,
+such as @code{package-enable-at-startup}, @code{package-load-list}, and
+@code{package-user-dir}.  Note that variables like @code{package-archives}
+which only affect the installation of new packages, and not the process of
+making already-installed packages available, may be customized in the regular
 init file.  @xref{Package Installation}.
 
   For more information on the early init file, @pxref{Init File,,,
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 0e30ad519a..77ecb667f4 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -103,12 +103,12 @@ Startup Summary
 directory instead.
 
 @item
-It calls the function @code{package-initialize} to activate any
+It calls the function @code{package-activate-all} to activate any
 optional Emacs Lisp package that has been installed.  @xref{Packaging
-Basics}.  However, Emacs doesn't initialize packages when
+Basics}.  However, Emacs doesn't activate the packages when
 @code{package-enable-at-startup} is @code{nil} or when it's started
 with one of the options @samp{-q}, @samp{-Q}, or @samp{--batch}.  To
-initialize packages in the latter case, @code{package-initialize}
+activate the packages in the latter case, @code{package-activate-all}
 should be called explicitly (e.g., via the @samp{--funcall} option).
 
 @vindex initial-window-system@r{, and startup}
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 7e7a8cd9bc..37c1ee6697 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -105,16 +105,15 @@ Packaging Basics
 evaluates the autoload definitions in @file{@var{name}-autoloads.el}.
 
   Whenever Emacs starts up, it automatically calls the function
-@code{package-initialize} to make installed packages available to the
+@code{package-activate-all} to make installed packages available to the
 current session.  This is done after loading the early init file, but
 before loading the regular init file (@pxref{Startup Summary}).
 Packages are not automatically made available if the user option
 @code{package-enable-at-startup} is set to @code{nil} in the early
 init file.
 
-@deffn Command package-initialize &optional no-activate
-This function initializes Emacs' internal record of which packages are
-installed, and makes the packages available to the current session.
+@defun package-activate-all
+This function makes the packages available to the current session.
 The user option @code{package-load-list} specifies which packages to
 make available; by default, all installed packages are made available.
 If called during startup, this function also sets
@@ -122,15 +121,20 @@ Packaging Basics
 evaluating package autoloads more than once.  @xref{Package
 Installation,,, emacs, The GNU Emacs Manual}.
 
-The optional argument @var{no-activate}, if non-@code{nil}, causes
-Emacs to update its record of installed packages without actually
-making them available; it is for internal use only.
-
-In most cases, you should not need to call @code{package-initialize},
+In most cases, you should not need to call @code{package-activate-all},
 as this is done automatically during startup.  Simply make sure to put
-any code that should run before @code{package-initialize} in the early
+any code that should run before @code{package-activate-all} in the early
 init file, and any code that should run after it in the primary init
 file (@pxref{Init File,,, emacs, The GNU Emacs Manual}).
+@end defun
+
+@deffn Command package-initialize &optional no-activate
+This function initializes Emacs' internal record of which packages are
+installed, and then calls @code{package-activate-all}.
+
+The optional argument @var{no-activate}, if non-@code{nil}, causes
+Emacs to update its record of installed packages without actually
+making them available.
 @end deffn
 
 @node Simple Packages
diff --git a/etc/NEWS b/etc/NEWS
index 99f3f27486..a17791252f 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -59,14 +59,20 @@ package system is initialized given that initialization now 
happens
 before loading the regular init file (see below).
 
 +++
-** Emacs now calls 'package-initialize' before loading the init file.
+** Installed packages are now activated *before* loading the init file.
 This is part of a change intended to eliminate the behavior of
 package.el inserting a call to 'package-initialize' into the init
 file, which was previously done when Emacs was started.  As a result
 of this change, it is no longer necessary to call 'package-initialize'
-in your init file.  However, if your init file changes the values of
-'package-load-list' or 'package-user-dir', then that code needs to be
-moved to the early init file (see above).
+in your init file.
+
+However, if your init file changes the values of 'package-load-list' or
+'package-user-dir', or sets 'package-enable-at-startup' to nil then it won't
+work right without some adjustment:
+- you can move that code to the early init file (see above), so those settings
+  apply before Emacs tries to activate the packages.
+- you can use the new 'package-quickstart` so activation of packages does not
+  need to pay attention to 'package-load-list' or 'package-user-dir' any more.
 

 * Changes in Emacs 27.1
@@ -149,6 +155,15 @@ for abbrevs that have them.
 It now treats the optional 2nd argument to mean that the URL should be
 shown in the currently selected window.
 
+** Package
+*** New 'package-quickstart' feature
+When 'package-quickstart' is non-nil, package.el precomputes a big autoloads
+file so that activation of packages can be done much faster, which can speed up
+your startup significantly.
+It also causes variables like package-user-dir and package-load-list to be
+consulted when 'package-quickstart-refresh' is run rather than at startup so
+you don't need to set them in your early init file.
+
 ** Ecomplete
 *** The ecomplete sorting has changed to a decay-based algorithm.
 This can be controlled by the new `ecomplete-sort-predicate' variable.
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 1edc06d024..9faae54f7b 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -681,6 +681,9 @@ package--activate-autoloads-and-load-path
 (defvar Info-directory-list)
 (declare-function info-initialize "info" ())
 
+(defvar package--quickstart-pkgs t
+  "If set to a list, we're computing the set of pkgs to activate.")
+
 (defun package--load-files-for-activation (pkg-desc reload)
   "Load files for activating a package given by PKG-DESC.
 Load the autoloads file, and ensure `load-path' is setup.  If
@@ -723,7 +726,10 @@ package-activate-1
             (message "Unable to activate package `%s'.\nRequired package 
`%s-%s' is unavailable"
                      name (car req) (package-version-join (cadr req)))
             (throw 'exit nil))))
-      (package--load-files-for-activation pkg-desc reload)
+      (if (listp package--quickstart-pkgs)
+          ;; We're only collecting the set of packages to activate!
+          (push pkg-desc package--quickstart-pkgs)
+        (package--load-files-for-activation pkg-desc reload))
       ;; Add info node.
       (when (file-exists-p (expand-file-name "dir" pkg-dir))
         ;; FIXME: not the friendliest, but simple.
@@ -1463,18 +1469,34 @@ package-initialize
   (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
+  (setq package--initialized t)
   (unless no-activate
+    (package-activate-all))
+  ;; This uses `package--mapc' so it must be called after
+  ;; `package--initialized' is t.
+  (package--build-compatibility-table))
+
+(defvar package-quickstart-file)
+
+;;;###autoload
+(defun package-activate-all ()
+  "Activate all installed packages.
+The variable `package-load-list' controls which packages to load."
+  (setq package-enable-at-startup nil)
+  (if (file-readable-p package-quickstart-file)
+      ;; Skip load-source-file-function which would slow us down by a factor
+      ;; 2 (this assumes we were careful to save this file so it doesn't need
+      ;; any decoding).
+      (let ((load-source-file-function nil))
+        (load package-quickstart-file))
+    (unless package--initialized
+      (package-initialize t))
     (dolist (elt package-alist)
       (condition-case err
           (package-activate (car elt))
         ;; Don't let failure of activation of a package arbitrarily stop
         ;; activation of further packages.
-        (error (message "%s" (error-message-string err))))))
-  (setq package--initialized t)
-  ;; This uses `package--mapc' so it must be called after
-  ;; `package--initialized' is t.
-  (package--build-compatibility-table))
-
+        (error (message "%s" (error-message-string err)))))))

 ;;;; Populating `package-archive-contents' from archives
 ;; This subsection populates the variables listed above from the
@@ -1856,18 +1878,26 @@ package-installed-p
 should be a version list.
 
 If PACKAGE is a `package-desc' object, MIN-VERSION is ignored."
-  (unless package--initialized (error "package.el is not yet initialized!"))
-  (if (package-desc-p package)
-      (let ((dir (package-desc-dir package)))
+  (cond
+   ((package-desc-p package)
+    (let ((dir (package-desc-dir package)))
         (and (stringp dir)
-             (file-exists-p dir)))
+             (file-exists-p dir))))
+   ((and (not package--initialized)
+         (null min-version)
+         package-activated-list)
+    ;; We used the quickstart: make it possible to use package-installed-p
+    ;; even before package is fully initialized.
+    (memq package package-activated-list))
+   ((not package--initialized) (error "package.el is not yet initialized!"))
+   (t
     (or
      (let ((pkg-descs (cdr (assq package package-alist))))
        (and pkg-descs
             (version-list-<= min-version
                              (package-desc-version (car pkg-descs)))))
      ;; Also check built-in packages.
-     (package-built-in-p package min-version))))
+     (package-built-in-p package min-version)))))
 
 (defun package-download-transaction (packages)
   "Download and install all the packages in PACKAGES.
@@ -1918,7 +1948,9 @@ package-install
                      (package-compute-transaction (list pkg)
                                                   (package-desc-reqs pkg)))
                  (package-compute-transaction () (list (list pkg))))))
-        (package-download-transaction transaction)
+        (progn
+          (package-download-transaction transaction)
+          (package--quickstart-maybe-refresh))
       (message "`%s' is already installed" name))))
 
 (defun package-strip-rcs-id (str)
@@ -2090,7 +2122,9 @@ package-delete
              (delete pkg-desc pkgs)
              (unless (cdr pkgs)
                (setq package-alist (delq pkgs package-alist))))
-           (message "Package `%s' deleted." (package-desc-full-name 
pkg-desc))))))
+           (package--quickstart-maybe-refresh)
+           (message "Package `%s' deleted."
+                    (package-desc-full-name pkg-desc))))))
 
 ;;;###autoload
 (defun package-reinstall (pkg)
@@ -3415,6 +3449,95 @@ package-list-packages-no-fetch
   (interactive)
   (list-packages t))
 
+;;;; Quickstart: precompute activation actions for faster start up.
+
+;; Activating packages via `package-initialize' is costly: for N installed
+;; packages, it needs to read all N <pkg>-pkg.el files first to decide
+;; which packages to activate, and then again N <pkg>-autoloads.el files.
+;; To speed this up, we precompute a mega-autoloads file which is the
+;; concatenation of all those <pkg>-autoloads.el, so we can activate
+;; all packages by loading this one file (and hence without initializing
+;; package.el).
+
+;; Other than speeding things up, this also offers a bootstrap feature:
+;; it lets us activate packages according to package-load-list and
+;; package-user-dir even before those vars are set.
+
+(defcustom package-quickstart nil
+  "Precompute activation actions to speed up startup.
+This requires the use of `package-quickstart-refresh' every time the
+activations need to be changed, such as when `package-load-list' is modified."
+  :type 'boolean)
+
+(defcustom package-quickstart-file
+  (locate-user-emacs-file "package-quickstart.el")
+  "Location of the file used to speed up activation of packages at startup."
+  :type 'file)
+
+(defun package--quickstart-maybe-refresh ()
+  (if package-quickstart
+      ;; FIXME: Delay refresh in case we're installing/deleting
+      ;; several packages!
+      (package-quickstart-refresh)
+    (delete-file package-quickstart-file)))
+
+(defun package-quickstart-refresh ()
+  "(Re)Generate the `package-quickstart-file'."
+  (interactive)
+  (package-initialize 'no-activate)
+  (require 'info)
+  (let ((package--quickstart-pkgs ())
+        ;; Pretend we haven't activated anything yet!
+        (package-activated-list ())
+        ;; Make sure we can load this file without load-source-file-function.
+        (coding-system-for-write 'emacs-internal)
+        (Info-directory-list '("")))
+    (dolist (elt package-alist)
+      (condition-case err
+          (package-activate (car elt))
+        ;; Don't let failure of activation of a package arbitrarily stop
+        ;; activation of further packages.
+        (error (message "%s" (error-message-string err)))))
+    (setq package--quickstart-pkgs (nreverse package--quickstart-pkgs))
+    (with-temp-file package-quickstart-file
+      (emacs-lisp-mode)                 ;For `syntax-ppss'.
+      (insert ";;; Quickstart file to activate all packages at startup  -*- 
lexical-binding:t -*-\n")
+      (insert ";; ¡¡ This file is autogenerated by 
`package-quickstart-refresh', DO NOT EDIT !!\n\n")
+      (dolist (pkg package--quickstart-pkgs)
+        (let* ((file
+                ;; Prefer uncompiled files (and don't accept .so files).
+                (let ((load-suffixes '(".el" ".elc")))
+                  (locate-library (package--autoloads-file-name pkg))))
+               (pfile (prin1-to-string file)))
+          (insert "(let ((load-file-name " pfile "))\n")
+          (insert-file-contents file)
+          ;; Fixup the special #$ reader form and throw away comments.
+          (while (re-search-forward "#\\$\\|^;\\(.*\n\\)" nil 'move)
+            (unless (nth 8 (syntax-ppss))
+              (replace-match (if (match-end 1) "" pfile) t t)))
+          (unless (bolp) (insert "\n"))
+          (insert ")\n")))
+      (pp `(setq package-activated-list
+                 (append ',(mapcar #'package-desc-name 
package--quickstart-pkgs)
+                         package-activated-list))
+          (current-buffer))
+      (let ((info-dirs (butlast Info-directory-list)))
+        (when info-dirs
+          (pp `(progn (require 'info)
+                      (info-initialize)
+                      (setq Info-directory-list
+                            (append ',info-dirs Info-directory-list)))
+              (current-buffer))))
+      ;; Use `\s' instead of a space character, so this code chunk is not
+      ;; mistaken for an actual file-local section of package.el.
+      (insert "
+;; Local\sVariables:
+;; version-control: never
+;; no-byte-compile: t
+;; no-update-autoloads: t
+;; End:
+"))))
+
 (provide 'package)
 
 ;;; package.el ends here
diff --git a/lisp/files.el b/lisp/files.el
index 8ec2bde588..1ead4a78da 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -3627,7 +3627,8 @@ hack-local-variables
                                 (push (cons (if (eq var 'eval)
                                                 'eval
                                               (indirect-variable var))
-                                            val) result))))))
+                                            val)
+                                       result))))))
                    (forward-line 1))))))))
       ;; Now we've read all the local variables.
       ;; If HANDLE-MODE is t, return whether the mode was specified.
diff --git a/lisp/startup.el b/lisp/startup.el
index 2669342eda..1faeabf23b 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1185,7 +1185,7 @@ command-line
                                 (package--description-file subdir)
                                 subdir))))
                   (throw 'package-dir-found t)))))))
-       (package-initialize))
+       (package-activate-all))
 
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.