bug#73681: Maybe partly undo the patch on Elisp comp-el-to-eln-filename

[Martin Edström]

Hi, thanks for info! This email will be a bit long because I'm quite confused 
and thinking aloud, so no need to reply to all of it.  But I appreciate any 
attempt to shed clarity!


On Tue, 08 Oct 2024 19:33:06 +0200, Liliana Marie Prikler 
<liliana.prikler@gmail.com> wrote:

> > It comes as part of the package. I don't want to assume that it has
> > been compiled, since it's fairly performance-sensitive. That's why
> > I'll either use a previously existing compiled object or make a new
> > one.
> Could you leave that decision to the user?

I'm considering it, but ran into essentially the same problem.  I need a way to 
tell which one was loaded, of the .elc, .eln or .el.

This takes the thread a bit off-topic but ... Do you know how?

Currently I'm leaning towards an algorithm like

#+begin_src elisp
(defun guess-loaded-lib (basename)
  (let* ((el (locate-library (concat basename ".el")))
         (elc (locate-library (concat basename ".elc")))
         (eln (comp-el-to-eln-filename el)))
    (if load-prefer-newer
        (car (sort (delq 'nil (list el elc eln)) #'file-newer-than-file-p))
      (if (and (file-newer-than-file-p elc el)
               (file-newer-than-file-p eln el))
          ;; If elc is newer than el, assume it was compiled from el, and
          ;; therefore is safe to replace with eln
          eln
        elc))))
#+end_src

I don't know how to "leave it to the user" any better than that.  

On Guix, even in the future when we re-enable JIT compilation, this algorithm 
could never return an .eln, since =file-newer-than-file-p= returns nil when 
both paths have identical timestamps from 1970.

I found a pretty neat built-in: 

     (symbol-file 'SOME-FUNCTION-FROM-THE-LIB nil t)

but its internals also use =file-newer-than-file-p= and 
=comp-el-to-eln-rel-filename=, so not sure it is any more reliable than what I 
have above.


> There is a separate load path for natively compiled files, called
> `native-comp-eln-load-path'.

Good to know! I don't know how Emacs consults it though.  I can't e.g. locate 
an .eln of a given library by calling something like

#+begin_src elisp
(locate-eln-file "org-node-parser")
#+end_src

so it seems I must simply do:

#+begin_src elisp
(let ((el (locate-library "org-node-parser.el")))
  (when el
    (locate-eln-file (comp-el-to-eln-rel-filename el))))
#+end_src

which is functionally equivalent to:

#+begin_src elisp
(let ((eln (comp-el-to-eln-filename (locate-library "org-node-parser.el"))))
  (when (file-exists-p eln)
    eln))
#+end_src


> > I infer that Emacs starts with finding a library in load-path, then
> > converts the path with `comp-el-to-eln-filename`, and checks if that
> > file exists, then loads it.
> >
> > And crucially, it is not just about the filepath, the function hashes
> > the file contents as well. That ensures that the output path is
> > always different if the source file changes.
> I think relying on such implementation details is perhaps permitted if
> it's inside of Emacs itself, but even then it clashes with our
> expectation that Emacs be graftable.

OK, so if passing "file.el" to =comp-el-to-eln-filename= produces a path with a 
hash of the content, like:

    .../eln-cache/29.4-HASH/module/file-HASH-BASED-ON-CONTENT.eln

this would make Emacs non-graftable.  Because the hash would change after 
file.el is upgraded.  I suppose that makes sense, thanks!

Maybe, as a hack, the graft process could symlink the pre-graft filename to the 
post-graft filename... so we don't have to alter the behavior of 
=comp-el-to-eln-filename=.

But yea... it's less dev effort to just not pre-compile any .eln files.

Out of curiosity: if Guix does no JIT compilation at all anyway, why does it 
not let Emacs do it the usual way into ~/.emacs.d/eln-cache, post-installation? 
 (Aside from the fact it would necessitate reverting the behavior of 
=comp-el-to-eln-filename=.)

Actually... (with reservation that I may be wasting your time) I'm starting to 
wonder why the filename needs to stay the same for Emacs to be graftable?  
Realistically, if all Emacs packages get upgraded, and if 
=comp-el-to-eln-filename= works like upstream, we have some paths like

    /gnu/store/emacs-29.4/bin/emacs
    /gnu/store/emacs-magit-OLD/lisp/magit.el
    /gnu/store/emacs-magit-NEW/lisp/magit.el
    .../wherever/magit-OLDHASH.eln
    .../wherever/magit-NEWHASH.eln

Then we start Emacs, and run its =comp-el-to-eln-filename= on the new magit.el, 
it would correctly return .../wherever/magit-NEWHASH.eln. No need for static 
filenames to swap out.

Is graftability just about the compiled objects inside of  
/gnu/store/emacs-29.4/share/emacs/site-lisp/ that might be symlinked in from 
other Guix packages?

If so, I guess it ties back to my confusion about how Emacs uses the 
native-comp-eln-load-path, because it seems that IF it just converted an .el to 
find an .eln as I thought, then grafting would work automatically because the 
new .el is also symlinked in along with the new .eln, which would result in 
=comp-el-to-eln-filename= returning the new .eln correctly.

No, grafting is about avoiding a world re-compile, right?  So like if package-A 
was built with an old dependency package-B, it does not get re-built when 
package-B is upgraded. But does grafting actually ever change anything in 
Emacs?  When you start emacs and it loads package-A which in turn loads 
package-B because there's a =require= statement for package-B, Emacs will 
actually load the fresh package-B from /gnu/store/emacs-package-B, won't it?  
So grafting would seem to be meaningless.

Not sure how to feel if it's like this instead: a compiled function from 
package A, containing a call to some function "package-B-frobnicate", will 
actually call some bytecode originating from 
/gnu/store/emacs-package-A/share/emacs/site-lisp/package-B.elc? That's... not 
what happens, right?  OK, probably yes if it was a defsubst or defmacro.

So we can have package A using a macro that was defined in an old version of 
package-B.  I like that not, but the reason I ask is I'm trying to find out 
where it is that the file path would influence the graft.


> The way our load paths are set up, it is actually the opposite (which
> still is a bug, just not the one reported).  While `guix upgrade` or a
> command to the similar effect will swap out the .eln under the hood,
> the `.el` and `.elc` files stay stable – remember what I wrote in the
> previous message about that having caused issues with byte compilation?

I confess I'm puzzled.  Does this just apply to a case like the following?

- Emacs itself is upgraded
- A package emacs-magit is NOT upgraded

so that

- magit.eln is rebuilt
- magit.el and magit.elc stay the same

That would make sense.  Do you mean it's a bug because .elc files are also not 
portable between different Emacsen?

Just checking: is it correct to expect that if you actually upgrade the 
emacs-magit package, then its .el, .elc and .eln are all upgraded?