Re: `make' written in elisp

[Stephen J. Turnbull]

>>>>> "Stefan" == Stefan  <address@hidden> writes:

    Stefan> I tend to think that most of the autoconfiguration should
    Stefan> be done upon use rather than during installation, which
    Stefan> basically implies "in elisp rather than in autoconf".

A word to the wise: This is expensive at startup, and will probably
take some effort to shake out.  Mike Sperber put in many hours
removing unnecessary stats from our naive package location code,
despite the very regular layout of our package hierarchy.  It's still
pretty straightforward, but the efficiency is not an accident.

It would theoretically be possible to have a cache which collects all
the auto-autoloads, but we felt that was over-engineering; startup is
not something that should happen very often in Emacs.

    dak> a) a directory layout.  XEmacs packages have a standard way
    dak> of placing Elisp, info, data and other files in a layout
    dak> under the top package directory.

XEmacs's layout is inverted from what makes sense for third-party
packages: we have a standard GNU Emacs-style tree with lisp/$package,
info/$package.info*, etc as needed.  Really, however, third-party
packages want $package/{lisp,lib-src,etc,...}.  The Emacspeak and JDEE
maintainers absolutely insist on this, for two examples.

XEmacs is moving toward flattening out the core lisp, and eventually
inverting the tree for packages.  The only exceptions will be package
meta-data (manifests used to ensure that package upgrades and removals
get rid of everything installed by unwanted packages), and info files.

This is probably not an appropriate strategy for Emacs, but you should
be aware that XEmacs is not necessarily going to stick with the
current structure.

    dak> In addition there are some standard file names for
    dak> initializing of packages and installation.  In short, this
    dak> part of the package system is basically just file layout
    dak> conventions relative to a package-specific top directory.

We have auto-autoloads.el, custom-loads.el, and _pkg.el.  The first is
what it looks like, the second handles attaching Custom's menus and
groups and so on, and the third contains meta-data which actually is
inserted into auto-autoloads.el, and never actually loaded itself.

    Stefan> I can't find any documentation about the in-package layout
    Stefan> used by XEmacs, tho.  Anybody knows where it's described?

The hierarchy is simple, and not really explicitly documented.  There
is some information on internal structure, especially for the source
trees, in XEmacs Users' Guide (node Packages) and Lispref (node
Packaging), see http://www.xemacs.org/Documentation/21.5/html/.  There
may be some commentary in the Internals Manual in the "Future Work"
nodes, but they're quite disorganized, and mostly written by Ben who
is not authoritative for the packages.

The installed package hierarchy is as described above:
$top/{etc,info,lib-src,lisp,man,pkginfo,...} with packages constrained
to install into like-named subdirectories, except for $top/info, where
the name should be $package.info*, and $top/pkginfo, which contains
the MANIFEST.$package files used when uninstalling.

We're policy-free with respect to source directory organization.
Basically in source the required files are an XEmacs-style Makefile
and a metadata file, package-info.in.  The three files referred to
above are automatically generated.

    dak> b) version management.  Versions must basically be floats,
    dak> higher versions correspond to later versions, are usually
    dak> based on some CVS archive and not at all with actual package
    dak> versions.  Uh.

    Stefan> Right, it seems to be used only to discover whether a
    Stefan> newer version is available or not.

Yeah, we need to do something about that.  I don't know what Steve
Baur was thinking.

    dak> c) a package manager that will download and upgrade packages.
    dak> It might do so in user-specific and system-wide packages.

XEmacs's is currently in flux.  Its #1 design bug is absence of an
HTTP transport, both because it makes EFS a single point of failure,
and because of the firewall issue.

However, unless there's some easily findable index of URLs for
downloadable packages, this is not terribly useful, I think.

    dak> I am not convinced about the value of the centralized server
    dak> approach to package management from XEmacs: it does not seem
    dak> to lend itself

    Stefan> Since `install' specifically targets external packages, a
    Stefan> centralized server doesn't make much sense indeed.  For
    Stefan> XEmacs, it was probably a good simplification.

The centralized server is _not_ designed as part of the package
management system, but as a service provided to the community.  Due to
the convenience for maintainers who don't necessarily have commit
access to the upstream project (assuming it exists), it has become a
dependency.  Doing something about that would require some effort.

There is one point that needs to be mentioned, however, and that is
that for complex packages that depend on other packages, the "calling
macros from byte-compiled code" kind of bug has dropped from FAQ to
fossil status.

    dak> But I do think that we should try to offer some way of
    dak> installing external Lisp code if one has acquired it as a zip
    dak> or tar archive in some manner already.

    Stefan> Not just tarballs but also single elisp files.  Most elisp
    Stefan> packages are single files.

This we really are not set up to handle efficiently.  I think we
should, so I'll be interested to see how you handle it.

    Stefan> The DWIM-aspect of `install' (just look for *.el and
    Stefan> *.info files) only works up to a point.  I've bumped into
    Stefan> problems where some *.el files should be ignored, or other

We handle this by having an explicit list as a make variable.  Usually
the default setting of "all .el files" works fine, of course, and
that's easily implemented with `wildcard'.

    Stefan> problems where the *.info file doesn't have the `info'
    Stefan> extension, or where the info file simply doesn't exist and
    Stefan> it's not clear how to build it based on a bunch of *.texi
    Stefan> files, ...

Yep.  Info files and package-specific data (eg, pixmaps, the Yow!
database, SGML DTDs, JDEE's Java code, etc) are the reason for 1/2 of
the hair in the package build infrastructure.  Most of the rest has to
do with supporting building on Windows.

    Stefan> We could keep adding intelligence and heuristics, but it
    Stefan> makes more sense to devise some conventions.

I agree in theory, but in practice, you'll run into resistance (ie,
they just won't bother) from 3rd-party maintainers, all the more since
you're focusing on "external" packages.

    dak> All of those certainly sound reasonable.  In particular, I'd
    dak> second having separate commands for installing system-wide
    dak> and user-local packages.

    Stefan> Agreed, it makes more sense.

XEmacs experience with "guessing" the target tree has not been
positive; we probably should consider moving in that direction
ourselves.


-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;