Re: Loading a package applies automatically to future sessions?

[George Plymale II]

> I think setting package--initialized to t is wrong, here (IIRC
> package--initialized means that we've filled the tables keeping track
> of which packages are available and such).  Not a problem when you're
> just trying to test the speed of my proof-of-concept patch, of course.

Ok, thank you for letting me know the implications of doing that. I had
to do it because I got some complaints from Emacs about my packages not
being initialized. I can post the exact errors if you'd like.

> Hmm... are you saying that my code ends up fetching code for
> slime-autoloads.elc instead of slime-autoloads.el?
> Or that slime-autoloads.el contains (pre)compiled code?

It looks like your code is fetching slime-autoloads.elc:

(let ((load-file-name 
"/Users/my_username/.emacs.d/elpa/slime-20180126.1033/slime-autoloads.elc"))
;; ... bunches of bytecode
)

> Also, I'm not 100% surprised that byte-compiling byte-compiled code would
> fail, but I can't see any immediate reason why it should fail, so it's
> quite possible to it'd be easy to make it work.

Not sure why it would fail either, but this is the error that I see for
that specific chunk of bytecode:

package-fastpath.el:899:1:Error: Wrong type argument: sequencep, 1004

I could attach the bytecode in another message if you'd like to see it,
but I'm not sure whether you do.

> Not quite sure what "my startup commented out" means.  I'll assume it
> means a more or less empty ~/.emacs, or maybe the use of -Q ?
> In any case I guess it means that 1.2s is the "speed of light", i.e. the
> holy grail of package.el on your machine.

I mean that I commented out my ~/.emacs file, which disables all
initialization code that I run, including in other files. It does _not_
mean using -Q, which brings my startup time to 0.5 seconds (FYI, I have
recently been using the Emacs Mac Port by Yamamoto Mitsuharu so my
startup is a bit slower than vanilla GNU Emacs). That therefore means
that 1.2s is how fast Emacs is with no customizations from myself in
effect. -Q and -q have other implications, which are detailed in the
Elisp manual under "38.1.1 Summary: Sequence of Actions at Startup"

I actually just realized those 1.2 seconds include running
`package-initialize', as per usual. In other words, it was not taking
advantage of package-fastpath.el. I just ran a modified version of my
~/.emacs file which comments out almost everything except:

(load-library "package") ;; I need this else Emacs says that
                         ;; `package-activated-list' is void
(setq package-enable-at-startup nil)
(load "~/.emacs.d/package-fastpath.el")
(setq package--initialized t)

And lo and behold, `emacs-init-time' yields 0.7 seconds. Which is just
0.2 seconds higher than what I get with -Q, which I believe is Emacs'
peak startup time. I also verified that I have access to all my packages
so it is using package-fastpath.el.

So that means your patch has actually increased startup time a lot more
than I initially thought and it's actually my own initialization code
which is bogging things down.

> If you subtract the 1.2s taken by "other things", it means that your
> config used to take 0.4-0.5s of startup time, and that the use of a big
> precomputed autoloads file reduced that to 0.1-0.2s, so sped it up by
> factor between 2.5 and 4, which I find a bit disappointing but still
> respectable (and it depends which proportion was taken by
> package-initialize vs which proportion was taken by your particular
> customizations).

Well, actually those figures are better than what you're saying as per
my information above. I guess that makes your patch even more
respectable for a first stab! :)

I guess since my startup time with my initialization code is 1.3
seconds, and running Emacs with package-fastpath.el is 0.7 seconds, my
own code is taking up about 0.5 to 0.6 seconds.

> As it stands, my patch only pre-sets package-activated-list, whereas
> package-installed-p requires other things set up by package-initialize.
> If you don't set package--initialized to t, you'll see that
> package-installed-p will rightfully complain that you haven't called
> package-initialize.

Ok, thank you for clarifying that issue. Although, as I mentioned above,
I had to set package--initialized to t, otherwise Emacs would halt
startup and complain about me not calling `package-initialize'.

> I guess my patch could also add a setting for `package-alist` to the
> package-fastpath.el, but I'm not sure I like the idea (it increases the
> consequences of having an out-of-date package-fastpath.el).

I'm not sure on that either. Although, we could make it so that
`package-fastpath-refresh' is run more often or in more places so as to
lessen that risk.

> Wouldn't it be better for your code to assume that if there's
> a package-fastpath.el, then those checks have already been performed
> (and to re-execute those checks in package-fastpath-refresh instead)?

Yes, it would likely be better for my code to do that instead.

> Or maybe package-installed-p should be changed such that when it's
> called with a symbol argument, it should first check if it's in
> package-activated-list, and if it's not, then it should call
> package-initialize before checking package-alist.

I can't give a strong opinion on that. Perhaps someone else who is
better-versed on package.el could shed some light here?

> Hmm... I thought this can't happen because the files are concatenated in
> the same order that they are normally loaded by `package-initialize`.
> I guess something doesn't work the way I thought it does.
> Can you investigate to see when or even why it happens?

It specifically happened with gh.el (https://github.com/sigma/gh.el),
which is dependent on marshal.el (https://github.com/sigma/marshal.el)

Apparently gh-autoloads.el uses a `gh-defclass' macro, which in turn
uses a `marshal-defclass' macro. There is definitely some strangeness
that could be going on here.

> The fact that it's a text editor shouldn't make any difference in this
> respect.  I'm not sure exactly what you mean by "the amount of packages
> that you have never hurts the startup of the main program", I guess you
> mean that program A is not slowed down when you install extra packages.

Yes, I mean that program A is not slowed down by extra packages.

> In Emacs the corresponding behavior happens if you install a package by
> hand and don't activate/use it.  `package.el` by default tries to
> auto-activate all your packages, so indeed it's slowed down by the mere
> presence of extra packages.

Right.

> I think a key element that lets Python and Ruby work that way is the
> namespacing and its tight binding to the filesystem.  Currently, while
> conventions are "similar" (e.g. function toto-titi is likely found in
> file toto*.el in package toto), they're just vague conventions.

Yes, Ruby and Python have a simpler "path" system for finding packages
than Emacs does currently.

> Making them more strict would indeed make it possible to change
> `require` so that a (require 'toto-tata) will automatically look for
> a file toto-tata.el in a package toto without having to "activate" that
> package beforehand and it would let us have a "missing function handler"
> which would automatically try to find the function toto-titi in one of
> the files of the titi package (e.g. by activating this package).

> This might make it possible to activate some packages lazily, but
> There'd still be aspects of auto-activation that would go missing, tho:
> e.g. registering jgraph-mode as a handler for *.jgr files.
> So some packages would still need some form of auto-activation.

Right, your last point is what I meant by "we have somewhat more
intricacies to worry about."

> The downside is that you have to explicitly `import` those gems when you
> need them.  In Emacs you could do the same: call (package-initialize t)
> in your ~/.emacs and then call `package-activate` when you need
> a particular package.

Yes, this is true. Still, I think we need some more simplicity in this
system of autoloads and so forth.

> IIUC there are 1.2s of your startup time which are spent without even
> running a single line of package.el code, so speeding up package.el
> might not be good enough (at least it seems to me that if 1.6s is too
> slow, then 1.2s is likely to also be too slow).

Well, those 1.2 seconds which I mentioned are just starting up Emacs
without any _initialization_ code. And that also means that
`package-initialize' is running and not taking advantage of
package-fastpath.el. Again, as I mentioned above, the startup time which
takes advantage of package-fastpath.el is ~0.7 seconds so we have
already sped up package.el by a lot and with noticeable effect.

> The usual answer in Emacs for those problem is "don't do that"
> (i.e. use emacsclient instead).  I understand it's not always an
> option.

Yes, I actually am an avid user of the Emacs client and server. I think
it's a great feature which helps me do a lot of useful
things. Unfortunately, it is not the panacea for startup time woes that
it is often touted as. I know it is that way for a lot of people, but I
unfortunately push Emacs too hard most of the time to be able to leave
it running for weeks or months. If that were the case, I wouldn't have
this itch to scratch.

- George Plymale II