Re: Modern conventions for structuring Emacs Lisp libraries

[Thorsten Jolitz]

Stefan Monnier <address@hidden> writes:

>> as a second take on this topic, I would like to make the attached
>> proposal for improving the conventions for structuring Emacs Lisp
>> source-code files:
>
> Thanks.  I think it'd be good to improve the current situation, but
> I don't think anything more than refinement is an option.  IOW changing
> the convention is not really an option for various reasons, such as the
> fact that several tools rely on it, and there are so many files using
> it, many of which we can't fix, that we'd be stuck with two different
> conventions for the next many years.

I followed your advice after the 'first take' on this topic and adapted
my tools (outshine.el, outorg.el and navi-mode.el) to the conventions.

I still wonder if saner more modern conventions cannot co-exist with the
older ones without the necessity to touch the existing libraries, just
by modifying the existing tools a bit.

>> 2.2.1 Last Line Pathology
>> -------------------------
>
> This is a very minor issue, I think.  It's trivil to fix it by simply
> ignoring any "trailing header".

Can't the existing tools be modified to accept both, a trailing header
or a trailing comment? It might seem like a very minor issue, but its a
symptom for a not so minor issue - incorrect headline tree-structure.

>> 2.2.2 Wasted First Level Weakness
>> ---------------------------------
>>   | 7 matches for "^;;; " in buffer: help.el.gz
>>   |       1:;;; help.el --- help commands for Emacs
>>   |      25:;;; Commentary:
>>   |      30:;;; Code:
>>   |     280:;;; `User' help functions
>>   |     967:;;; Automatic resizing of temporary buffers.
>>   |    1041:;;; Help windows.
>>   |    1193:;;; help.el ends here
>>   `----
>>   The above example, produced by typing '1' in the associated
>>   *Navi-buffer* of help.el (-> show headlines up-to first-level), shows
>>   this weakness quite clearly:
>>   1. in a file with some 1200 lines of code, there are 7 1st-level
>>      headlines
>>   2. at most 3 (if not only 2) of 7 headlines should really be 1st-level
>>      headlines:
>
> Again, this seems like a rather minor issue.  Furthermore, ";;; Code:"
> is a well-known header, so if needed, we can special case it in code
> such as navi.

The first thing when looking at a library is getting an overview about
its structure - with tools like e.g. navi-mode.

For me its a huge difference then if I look at a well balanced and
logical hierachical tree-structure (org-element.el and ox.el are perfect
examples for this) or at something that seems inherently wrong and
illogical at first sight. And the sad thing is that sticking to the
conventions forces library authors to do things like putting headers on
the same level that should be subheaders really, and define headers that
are not headers at all.

>>   all other headlines in the header-comment section are subheadlines of
>>   this one, i.e. level 2 or higher.
>
> You can get that without any modification to existing files by
> considering that any header that comes before ";;; Code:" is really one
> deeper than what its semi-colons say.

When leaving existing libraries untouched - wouldn't it be possible to
allow for saner structuring in new libraries?

>>   In oldschool Elisp files, headlines often end in colons. In Org-mode,
>>   this is not usual, and the Org-mode conventions are better in this
>>   case.
>
> Again, trivial to "fix" in any tool like navi: just change a trailing
> colon into a full stop.

This seems the least important point, but again I ask myself: why
continue recommend the colons, when it turns out they don't look so good
in exports?

>>   The otherwise fantastic library dired.el illustrates why:
>
> All those non-headers are bugs.

ok

>>   8) shows a wild mix of real headlines and comments. This is because it
>>   is only natural for people to get creative with comment characters
>
> Maybe it was a bad idea to choose this convention, but it's easier to
> fix the few cases where people's use of comments ends up accidentally
> colliding with your convention.
>
>>   |       1:;; * iorg-scrape.el --- elisp glue code for 
>> `picoLisp/lib/scrape.l'
>
> Note that ";; *" is already in common use for other purposes (basically
> the * is used as a bullet point there), so your convention would also
> bump into other problems.
>
>>   A big plus: this is /major-mode agnostic/. All 3 libraries (outshine,
>
> I'm pretty sure there are many other languages that define their own
> notion of comment structure, and I'd be surprise if any of them happens
> to choose your convention.  IOW your convention is "major-mode agnostic"
> at the cost of not following the "normal" conventions.  Just as is the
> case with allout.el.
>
> This is a trade-off with its benefit and its downsides.
>
> The other option is to make your tools's structuring convention
> sufficiently customizable that it can make adapt to the
> "normal" convention (this is the path taken by outline.el).  It's not
> perfect either, of course, but I think that given the obstacles it's
> either to adapt the tools than to change the conventions.
>
>>   Conversion between oldschool and outshine headlines would be
>>   exceedingly easy - as long as the oldschool files don't mix comments
>>   and headlines, i.e don't have "invalid" syntax (in a loose sense).
>
> Exactly: if they're correct, then the conversion is easy.  So easy that
> you can do it on the fly, without touching the file! ;-)

I agree, if the conventions would allow for sane file structuring,
conversion between headline styles would be so easy that I could use my
favorite style (Org-style) and convert back to oldschool whenever
possible.

>>   | ;; * iorg-scrape.el --- elisp glue code for `picoLisp/lib/scrape.l'
>>   | ;; ** MetaData
>>   | ;;   :PROPERTIES:
>>   | ;;   :copyright: Thorsten_Jolitz
>>   | ;;   :copyright-since: 2013
>
> IIUC the syntax of the "copyright" line is legally significant
> (e.g. whether years can be abbreviated, whether year ranges can be used,
> etc...), and I don't think I want to get into discussions about what
> other syntax we can use.  So I'm not interested in changing the
> copyright line.
>
>>   | ;;   :version:  0.9
>
> The ";; Version:" format is used in ELPA, so changing it would require
> changes in too many places.  Not gonna happen.
>
>>   | ;;   :licence:  GPL3+
>>   | ;;   :licence-url: http://www.gnu.org/licenses/
>
> Here again, the many-lines blurb about the license has legal
> significance (otherwise we'd have switched to ";; License: GPLv3+" by
> now).

ok, I'm not into this legal stuff, I just wanted to demonstrate a syntax
that seemed more suited for meta-data IMO.

>>   2. Since a [Property API] for Org-mode's properties exists, reading
>>      and writing them from an Emacs Lisp program becomes almost trivial.
>
> Reading them is already trivial (via the lisp-mnt.el library).
> Writing is usually very easy already; the only exception could the
> "Keywords:" header which is just a big problem in itself anyway (we
> don't really know which categories should be available).
>
>> 3 Summary
>> =========
>
> Damn!  You skipped the part I was looking for: the "Commentary:"
> section.  I do want this part to be refined.  More specifically, I'd
> like someone to come up with a description of a markup format to use
> there (99% compatible with what we already have), together with code
> that can turn such a Commentary section into nicely rendered text in an
> Emacs buffer.

If I understand you right, maybe there is good news. Install and load
outshine.el and outorg.el, move point on the "Commentary:" header and do
'M-x outorg-edit-as-org' (or M-# M-#).

Then you will be offered that headline converted to Org in a temporary
Org-mode edit buffer (*outorg-edit-buffer*). In that buffer, you can use
the power of Org-mode for writing really complex commentary text, even
using Org-Babel for executing code and inserts results, and then export
the commentary section nicely formatted to HTML, LaTeX, ODT, ASCII and
other backends.

The docstring of `outorg-edit-as-org' shows several ways to include
export options when transforming the whole source-code buffer to Org
(without ARG, this function only acts on the headline at point). With
narrowing the source-code buffer to the commentary headline before
calling this function, you can make use of these options too when
exporting only a single headline.

,---------------------------------------------------------------------------
| outorg-edit-as-org is an interactive Lisp function in `outorg.el'.
|
| It is bound to M-# #, M-# ', M-# M-#, <menu-bar> <Outshine> <Edit As Org>.
|
| (outorg-edit-as-org &optional ARG)
|
| Convert and copy to temporary Org buffer
|
| With ARG, act conditional on the raw value of ARG:
|
| | prefix | raw | action 1          | action 2                         |
| |--------+-----+-------------------+----------------------------------|
| | C-u    | (4) | edit-whole-buffer | ---                              |
| | C-1    |   1 | edit-whole-buffer | insert default export-template   |
| | C-2    |   2 | edit-whole-buffer | ask user for template-file       |
| | C-3    |   3 | edit-whole-buffer | insert and keep default template |
| | C-4    |   4 | edit-whole-buffer | insert and keep template-file    |
|
`---------------------------------------------------------------------------


--
cheers,
Thorsten