Re: Stability of core packages

[Mohsen BANAN]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: emacs@mohsen.1.banan.byname.net
>> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>,  joaotavora@gmail.com,
>>   jporterbugs@gmail.com,  dmitry@gutov.dev,  emacs-devel@gnu.org
>> Date: Fri, 28 Apr 2023 22:21:46 -0700
>>
>>
>> I have been following this thread from a meta perspective.
>
> Thanks for chiming in.
>
>> Because we use git for everything and because it
>> is possible to take complete git snap shots of all
>> relevant packages, it is possible to easily
>> provide stability for the user based on snapshots.
>>
>> Consistent upgrades are then possible for new
>> consistent git snapshots. So, based on the snapshots
>> the packages are not stale and remain consistent
>> while evolving.
>>
>> This is what doomemacs does. For every package,
>> there is a git tag which doomemacs keeps with the
>> adoption of the package. The tags are guaranteed
>> to be consistent. With Blee (ByStar Libre Emacs
>> Environment) I do the same but instead of keeping
>> that tag with the packages, I keep central
>> manifests for emacs releases. So, emacs-28.1 has
>> its own packages manifest which can be upgraded
>> from time to time. All upgrades go to the
>> appropriate repo and get the appropriate tag based
>> on the manifest. I do this across all emacs
>> package archives -- which I see as collections of
>> git repositories.
>>
>> If you want to go with something like this, you
>> have to revisit the package retrieval model.
>>
>> For every release of emacs, you maintain a
>> separate manifest. And you keep updating that at
>> new releases and even in between releases. Since
>> all emacs archives are git based, it is possible
>> to apply this strategy to all of them -- where
>> each take care of its own consistency.
>
> I've read this several times, and I admit I still don't really
> understand the suggestion.  Maybe it's because you are using some
> terminology without explaining it in enough details, perhaps because
> you assume prior knowledge of what it means.

Sorry about that.

In fact, that is the case. I was assuming that you
and others would be familiar with the lessons
learned from the "straight.el" and doom emacs
efforts over the past many years. My apologies.

To better set the context, let me start with a
quote from:
https://github.com/doomemacs/doomemacs/blob/master/docs/faq.org

-----------
Why does Doom use straight.el and not package.el?

package.el simply doesn’t cut it. Its flaws become
apparent the more packages you manage, the more
complex your config becomes, and how often those
packages see updates:

- No Disaster recovery
...

- Rolling release or bust

package.el installs the latest version of every
package, or none at all. There’s no rollback, no
pinning to a stable ref, and no git history to
peek into. It offers little reproducibility;
wiping/losing/copying your config becomes a gamble
if you aren’t constantly backing up your packages
(then recompiling them if you’re up/downgrading
Emacs). melpa-stable was well intentioned, but it
offers no guarantees or standards on how/when
maintainers tag their projects. I’d rather the
user decide for themselves, because they’re the
ones in the trenches.

...

Package management needs to be easier, because
Emacs is hard enough already. Doom fills these
gaps with straight.el’s help, and beyond.
------------

I am not saying that everything that Henrik
Lissner is saying is correct. But I also think
that the model of package.el is inadequate and
that it should be replaced by something like
straight.el.

The readme of
https://github.com/radian-software/straight.el
has a Guiding principles section.

As João Távora observes, straight.el can be a
solution for the eglot use case:

João Távora> Who knows, maybe this will all be OK after all?? Half the
João Távora> users are using "straight.el" anyway (and I really doubt
João Távora> it is hampered by these minutiae about dependencies, seeing
João Távora> as MELPA-land is teeming with way more "dependency hell").

>
> Here are some questions for which I didn't find answers, and which
> precluded me from following your suggestions:
>
>   . what is a "git snapshot"? what does it include in the context of
>     this discussion?

By a "git snapshot" I mean a collection of Git
commit IDs which can be used as pins.

In the doom model, with streight.el and with the
eglot use case, consider:

(package! eglot :pin "e501275e06952889056268dabe08ccd0dbaf23e5")

https://github.com/doomemacs/doomemacs/blob/master/modules/tools/lsp/packages.el

That is an example of a git snapshot.


>   . what is meant by "upgrades for new consistent git snapshots"? how
>     is such an upgrade done, and how does it work in practice?

New collection of Git commit IDs can then become
basis for upgrades.

>   . what are those git tags which doomemacs keeps with the packages?
>     what does each such tag indicate, and how are these tags
>     maintained and updated?

In the context of evolution of say ELPA, I am
suggesting use of the model of streight.el.

And its use based on the manifest model -- not
doom's package adoption model.

For each release of emacs, emacs{28,29,30}
the archive will maintain a manifest-{28,29,30}.

Each of those files enumerates a list of Git
commit IDs for each of the packages in the
archive. An update to the Git commit ID becomes a
basis for a potential upgrade.

>   . what does it mean for a git tag to be "consistent"?

It means that it is known to work well with all
other Git commit IDs in that manifest.

>   . what are those "central manifests" you describe for Blee? what
>     does each manifest contain and how is it used?

See manifest-{28,29,30} above.

>   . wrt your proposal of keeping separate manifests for each Emacs
>     release, how will a manifest for a specific release be updated,
>     and where will it be kept and made available for users?

The manifests are an internal part of the archive.
Users can see them, but they need not.

Maintenance of a manifests is the work of the ELPA
maintainers. They need to be subjected to
regression tests/verification's that guarantee
consistent sets.

I think what I am saying above reflects current
ad-doc practices of many straight.el users and
reflects a natural evolution direction for
package.el.

Eli, if you have not used straight.el, I suggest
experimenting with it. And again sorry about
having been overly cryptic.

...Mohsen