Re: Adding the `prescient` packages to NonGNU ELPA?

[Jonas Bernoulli]

Philip Kaludercic <philipk@posteo.net> writes:

>> All of these packages live in the same repository in the link below.
>
> Would it be possible to change this?  E.g. by making each project live
> on it's own branch.  That would make the :ignored-files rules below a
> lot simpler.

When making such requests, we should think about the needs of all
involved parties.  We all have our own ideas of "best practices"
and sometimes these ideas are in conflict with each other and/or
convenience.

For the maintainers of a "main package" with "extension packages", it is
most convenient to keep these packages in a single repository; making
changes across more than one of the packages is easier that way.  Since
they work on these packages the most, their needs should come first, and
as far as I can tell, nobody tries to *force* them to do something else.

For most users it doesn't matter much.  [Non]GNU Elpa and Melpa take
care of distributing the extension packages separately, making sure that
users do not have to install any additional packages, which are only
needed by one of the extensions, unless they actually want to use those.

For Melpa it is trivial to split a repository into multiple packages.

[Non]GNU Elpa take a different approach when it comes to deciding which
files to include in package tarballs.  Where Melpa excludes everything
except certain files, [Non]GNU Elpa include everything except certain
files.  (I might be wrong about this, but I believe) [Non]GNU Elpa do
not have a default exclude list used for all packages.  As a
consequence, and even though they include more files, I believe their
file specifications tend to be more verbose.  [I don't intend to argue
here for one approach or the other, that isn't the question here.]

People involved in [Non]GNU Elpa pushing back harder against putting
extensions in the same repository as the main package than the Melpa
folks, is probably an indicator that dealing with such bundling is more
work for them.

But AFAIK it cannot be *that* much more work, that it would justify by
itself, to ask package maintainers to take on even more additional work
by splitting up their repositories, and to implying in those requests
that doing so is a best practice that will benefit everyone.

If package maintainers are willing to split up their repositories, even
though it means more work for them, that is fine by me.  (Here too, it
isn't *that* much additional work, and I agree that if we all had
infinite time, this clearly would be the best way of doing it.)  But I
don't think splitting things up to the extreme should be described as an
unquestionable best practice; after all emacs.git itself bundles a lot
of unrelated packages in the same repository.  [Off topic and maybe a
bit surprising: I would be in favor of splitting up emacs.git more, but
let's not get into that here.]

I should note that I am aware that the work is not done by once writing
the file include/exclude rules when the multiple packages that are
maintained in a single repository are first added.  The later addition
of yet another library/package can make it necessary to adjust those
existing rules.  But I don't think this is a huge burden and that it is
small compared to the additional work that you are asking package
maintainers to take on by splitting up their repositories.

Another drawback of main+extension repositories from the perspective of
[Non]GNU Elpa is that it makes it necessary for elpa-admin.el to
checkout the same repository multiple times.  I think this is a small
drawback and I would recommend that just ignore it.  Otherwise it is an
internal tooling issue that shouldn't concern package maintainers;
elpa-admin.el could, for example, be taught to clone the repository just
once and to use a separate worktree for each package.

Moving to another interested party, the Emacsmirror, which I maintain.
The Emacsmirror provides a superset of the packages in any one of the
elpas, or all elpas taken together for that matter.  A consequence of
that is that it has to deal with a lot more diverging practices, and
that the various elpas promote in some cases competing best practices.

Also, because the benefits of the Emacsmirror are less obvious I am in
a weaker position when requesting changes from packages maintainers,
compared to people maintaining an elpa.  But since we are nevertheless
in a similar boat, I appreciate it, when the maintainers of the elpas
take the needs of the Emacsmirror into account.

The Emacsmirror itself does not even attempt splitting up repositories
that contain multiple packages.  It mirrors repositories.  Ideally one
repository contains one package, but that isn't always the case and
trying to change it would be an uphill battle, which could never be won.
I have been fighting some other uphill battles for years, but ultimately
I often have no choice but to accept that package maintainers do things
that I think are not ideal or even wrong.

Nowadays, for the most part, the Emacsmirror learns about new packages
to mirror, by importing the lists of package recipes/specifications of
the various elpas.  Because the elpas do sometimes build multiple
packages from a single repository, duplicates appear in those lists
(from the mirror's perspective) and I have to make sure to not add the
same repository twice.

This is where this recommendation becomes problematic:

> E.g. by making each project live on it's own branch.

The current Emacsmirror code cannot deal with this special case.  It
wouldn't be hard to support it, but so far I had no reason to implement
that.  There is *a single* package that currently does this (as a result
of Stefan Monnier asking its maintainer to do so).

The Straight package manager apparently cannot deal with this either.  I
would assume that it wouldn't be hard to implement support, but here too
it does not make much sense to do so for the benefit of a single package
that wants to do things differently.

My own package manager, Borg, follows the lead of the Emacsmirror.  It
clones the main-package+extensions repository as-is.  Users have to
explicitly exclude the extensions that they are not interested in, or
they get compilation errors.  This might sound not very user-friendly to
those who don't use Borg, but in practice it works well; Borg is just
more picky about who its friends are (but that isn't the topic here).

As I have said before, I agree that ideally extensions to a main package
would be maintained in separate repositories, but also that this means
more work for the maintainers, and if they don't want to do it because
of that, then that should be accepted.

I have one additional request: When you encourage package maintainers to
split up their repositories, then do NOT recommend that they put the
separate packages into separate branches of the SAME repository.

Instead ask them to use separate repositories altogether.  Except for
the small additional initial need to create the separate repositories,
this should not be any more work than using merely separate branches.

Merely using separate branches is a half measure, that doesn't really
benefit anyone.  Various tools that deal with emacs packages currently
don't support it.  Setting up the branches requires knowing or learning
about orphan branches.  Checking out multiple packages requires either
cloning the same repository multiple times, or knowing or learning how
to use multiple worktrees.

     Jonas