[openLilyLib] Fundamental reorganization started

[Urs Liska]

Hello everybody,

triggered by some comments on my last post http://lilypondblog.org/2015/01/introducing-scholarly/ I decided to pick up a few loose ends that have been lying around for some time now, so I finally started restructuring openLilyLib, aiming at a quite fundamental change.

So this message is directed (in descending order of relevance) to

• Those who have already taken part in openLilyLib reorganization discussions some half a year ago
• Everybody who has contributed to openLilyLib so afr
• All openLilyLib users
• All other LilyPond users/developers

Some motivations to do that now:

• (Valid) Questions were raised if it is a good idea to have many different "libraries" scattered around.
• Every now and then the idea arose to create some sort of consistent interface/infrastructure/specifications to create/distribute/use libraries to extend LilyPond's functionality without having to stuff everything into Lily herself.
• openLilyLib has become a somewhat bubbly, unconcrete collection of code.

I had ideas about the topic that were partially very clear, partially rather vague. But now I simply started the task with the initial goal of moving ScholarLY into the new structure before I had done too much that would have to be reverted or modified later.

I intended to finish a certain step and announce the project in a blog post, but I decided to start discussion somewhat earlier and here on the list for two reasons: I'll be away for a week (with only mobile phone internet access) and won't be able to reach that state before. And I think I'd prefer getting feedback before stepping too far. So what follows is partially an explanation of what I've done, of what I intend to do next, and it's about opening discussion what direction this should take. Please understand that it's not about discussing a finished and polished project.

### Basic structure/concept
Currently openLilyLib is a library of includable LilyPond files, organized in a directory structure which intends to represent its conceptional hierarchy. In the new structure openLilyLib will be a repository of libraries, each one living in a subdirectory of the top-level /ly directory. While these library are also simple directories within the repository they are conceptually different in that adding a library has to be a significantly more deliberate act than  just creating a new directory when one doesn't really know where to put a new function:

• A library must have a maintainer who is responsible for the library's target and also for code contributions
• A library must have a clearly visible purpose, and it must have been discussed that such an addition makes sense (i.e. that the intended code shouldn't rather be added to other, existing libraries)
• A library must have a certain "weight" that might justify setting up a subdomain like scholarly.openlilylib.org (doesn't exist, just an example)

The idea is that openLilyLib becomes a one-stop-shop solution for LilyPond extension, so it will be easy to share functionality and have other users easily get hold of it. This will also make it possible for tools (e.g. LilyPond editors) to depend on external LilyPond code by simply directing their users to get hold of that one repository as a dependency.
The hope is that such a set-up will encourage the community to share code and enable people to use LilyPond without having to reinvent the wheel each time (just as I wouldn't want to implement regular _expression_ searching in any programming language but rather use an existing library for that).

### Accessing openLilyLib and its libraries

"Installing" openLilyLib involves the following steps (also needed if you want to contribute to the current discussion):

• Downloading/cloning openLilyLib from https://github.com/openlilylib/openlilylib
  (if you haven't done so already)
• Adding its root path to LilyPond's include path
  (if you haven't done so already)
• Adding its top-level /ly directory to LilyPond's include path.
  Once everything has been migrated the root path include can be dropped but for now this doubled include is unfortunately necessary.
  

Step one for using is to activate openLilyLib with
\include "openlilylib"

This will already load some common infrastructure and functionality, mostly because openLilyLib itself needs it, but it is also available for users' files. Examples already available are:

• LilyPond version predicates which make it possible for libraries to support different LilyPond versions by conditionally executing code through things e.g.
  #(if (lilypond-greater-than? "2.19.4") ...
  
• Logging
  Commands like oll:log, oll:warn etc. that trigger messages (and log them to files) when a certain log-level is set
• alist access
  Taken from Jan-Peter's library it greatly simplifies the handling of (nested) association lists.
  This is an example of functionality included because openLilyLib needs it but that can also be very useful in general contexts.

One of the nicest things so far is global configuration handling. Libraries can register options with default values that can be set very simply by end users. One actual example:

% Library code:
\registerOption scholarly.colorize #t

This tells the ScholarLY library to colorize their objects by default. Users using ScholarLY can later (when the score is going to be published and coloring isn't wanted anymore) simply write

% User code:
\setOption scholarly.colorize #f

to switch off the coloring.
Another example is changing the color for a certain annotation type in ScholarLY:

% User code
\setOption scholarly.annotate.colors.critical-remark #blue

So libraries define their option hierarchy, and users can easily configure the behaviour in their projects.

The best thing is that this isn't restricted to libraries but it's available as soon as openLilyLib is loaded. So you could for example write

% house library
\registerOption my-house.print-status-footer ##t

% project configuration
\setOption my-house.print-status-footer ##f

A function that is actually responsible for the footer can access the option's value through

#(if #{ \getOption my-house.print-status-footer #}) ...

Already with annotate this made my library configuration code extremely shorter because I didn't have to maintain separate configuration variables and individual access functions for everything.
See https://github.com/openlilylib/openlilylib/blob/master/ly/scholarly/annotate/config.ily
versus the earlier
https://github.com/openlilylib/openlilylib/blob/dc9fe55adbcb4c065e3451e48b3e6c45b9e4d91e/ly/scholarly/annotate/config.ily

---
Accessing items from one of the libraries is done with the new command

\loadModule "library/path/to/module" or
\loadModule "library/path/to/file.ily"

This command will ensure that any file and library is included only once and that it is properly initialized. If a library has an init file this will be parsed once before the first item from the library is loaded. That makes it possible to have shared code or data in a library, e.g. common helper routines or a shared data structure, without them having to be parsed multiple times.

The function is great IMO, but I'm not 100% sure if the term "Module" is appropriate here. That's one of the points where I think some discussion would be good. What is a library, what a module, what term should be used for individual files. Currently a module would be e.g.
contemporary/note-heads, while there could also be individual items such as contemporary/note-heads/my-fancy-tremoly.ily.


### Documentation
One very important aspect is unfortunately the least thought-through yet: auto-generated documentation.
In the context of last year's discussion we improved the way openLilyLib files have to be documented within header fields. This is to become much more strict because there is /will be an infrastructure to create consistent usage examples using the data provided in these headers. To get an idea about that try out ly/scholarly/usage-examples/diplomatic-line-breaks.ly. The actual code file ly/scholarly/diplomatic-line-breaks.ily is also a good example of using the configuration infrastructure.
Discussion on this has to be continued (before too many files have been included), but basically I think this is a very nice way to create consistent code and documentation.

Documentation will have to get a second level, though, and this is in a even more preliminary state. What I've pondered over for quite some time is a system of API documentation for LilyPond files. What we need is a consistent syntax to insert documentation comments in (library) LilyPond files and then a tool/module that creates documentation from this information. This is intended to create (HTML and other) docs for libraries or local projects but also a data structure that can be used in separate tools, e.g. editing environments (to provide e.g. syntax highlighting or code completion for openLilyLib's libraries).
We had already made some promising experiments, but only using the header-style documentation and usage examples mentioned above.


###
Please excuse this lengthy post, but I think it is necessary to get the new openLilyLib up and running. And I think it is worth the effort, as I have a feeling that this will significantly change the way we can use LilyPond in the future.

Best
Urs