Re: Interoperation between package managers

[Jonas Bernoulli]

>> As mentioned above, I haven't thought about what package.el could do
>> to make it easier to automate what I do manually, so if you have
>> suggestions for things that we could change in (or add to)
>> package.el to make it easier for Borg to interoperate with it,
>> please send them along,
>
> You should talk to Jonas (cc'd) about this as he's the author of Borg.
> I can't really provide much useful help beyond this point since I have
> no desire to use package.el for anything.

(This has gotten a bit long, you may
 skip to "What Borg needs from package.el".)

I would like to eventually allow users to use Borg alongside package.el.

The idea behind Borg is that for some people using a package is only the
first step, the second is to contribute to it.

Borg installs packages by cloning the respective git repository and then
byte-compiling the libraries, generating the autoload file, generating
the info files, and adding the appropriate directories to `load-path'
and `Info-directory-list'.

That works surprisingly well, because most packages follow some informal
common sense conventions.  But Borg also has features to deal with
packages that stray for the common path.  The main difference from other
package managers is that in the case of Borg, those packages are not
brought into line at the time a package is created (because there is
neither a "package" not a "package maintainer" just the upstream git
repository).  But like Melpa or El-Get, Bort does support "recipes"; the
only difference is when and where the recipe is "applied".  In the case
of Borg it is applied on the users machine at the time of installation
(and update).

So as I said, Borg is primarily intended for people who contribute to
many of the package that they use.  In that sense it is similar to
adding the local clone of elpa.git to `package-directory-list.  One big
difference though is that in the case of Borg each package continues to
live in its own repository.

It is already possible to use Borg and package at the same time, but I
intend to make it a fully supported use-case.  All you have to (and can)
do currently to use both package managers at the same time is to
preserve this order in the init file:

  (package-initialize)
  (borg-initialize)

This guarantees that Borg wins in case a package is installed using both
package.el and Borg.  This allows users to install most packages using
package.el and only install the packages that they contribute to using
Borg.  The main benefit of using Borg to *install* some packages instead
of just *using* the package.el-installed version and *working* in a
manually cloned git repository, is that after restarting Emacs, the
locally modified version is loaded without having to manually adjust
the `load-path'.

Of course there are other benefits, for example `borg-install' only
needs the name of a package because it knows the location of thousands
of upstream repositories.  `magit-clone' on the other hand expects an
url.

In addition to cloning package repositories, it adds the clones as git
submodules of the ~/.emacs.d repository.  This makes sense if one only
uses Borg, but when also using package.el then probably not so much.

This is the main change required on Borg's side to make it more useful
for people who also use package.el.  (It is already possible to clone a
package repository without adding it as a submodule, but currently this
also means that Borg forgoes actually installing the package by
compiling it and adding it to the `load-path'.

---

What Borg needs from package.el?

If you initialize package.el before Borg, then the Borg-installed
version of a package shadows the package.el-installed version.  It would
be nicer if package.el could be told "this is a list of packages that
are somehow installed, do not install those packages, just assume they
are available".

I don't think that adding the directory that contains the Borg-installed
packages to `package-directory-list' would do because:

* That would cause package.el to load those packages, resulting in them
  being loaded twice.  (At least they would be added to the `load-path'
  twice.)

* While some of these directories follow the format expected by
  package.el, others do not.  For example in many repositories libraries
  live in ./lisp/ instead of ./.

So really, all that Borg needs is option `package-assume-installed'.
Maybe that already exists, I haven't checked, because I haven't actually
gotten around to making Borg package.el compatible yet.

---

And since that is what started this whole conversation, I should note
that I don't really care that users who use Borg exclusively have to put

  (setq package-enable-at-startup nil)

in their init file.  While I haven't followed this conversation, I have
thought a bit about how package.el is initialized, and can appreciate
how the current solution has grown out of necessity.  It tries to
satisfy users with very different needs and especially varying levels of
understanding.

The only thing that annoys me a bit is that when I use `emacs -Q' to
debug some issue without my configuration, then I end up with
~/.emacs.d/elpa being created.  It would nice if `package-initialize'
did not do that and would only *update* the local metadata if it is
already there.  Later when the user calls `package-list-package' or
`package-install', then the initial local copy of the metadata could be
downloaded.

But package.el is not unique in this regard.  When I use `emacs -Q',
then some other build in packages also create files in ~/.emacs.d that
would be different places if my personal init.el had been loaded.  I
can live with that and just cleanup manually after an `emacs -Q'.