Re: Keep parsed document in memory

[Urs Liska]

Am 22.02.2016 um 12:27 schrieb David Kastrup:
> Urs Liska <address@hidden> writes:
>
>> So, let's assume we already had implemented a server daemon mode for
>> LilyPond.
>> Would it be possible to make that daemon keep a representation of the
>> *parsed* document in memory?
>>
>> OK, looking at it from traditional use-cases this might seem
>> nonsensical. Why would you want to recompile an unchanged document?
>> Well, if I'm not mistaken the engraving can still be influenced *after*
>> the parsing by applying tweaks and overrides through engravers. I'm
>> concretely talking about the edition-engraver but eventually a newly
>> developed tool based on the current edition-engraver. And if I'm able to
>> do that *and* keep the parsed document in memory I could then recompile
>> the score or excerpts from it (using skipTypesetting) without
>> re-parsing.
> I think you grossly overestimate the time LilyPond spends on parsing.

I don't think so, my estimate is based on experience and observation
(although the following numbers are very coarse and only from
recollection). The effect increases with the size and complexity of the
score, but it should be *noticeable* with any score. Maybe it will be
just half a second sometimes, but with user experience even this *does*
make a difference.

I worked on a large score with about 100 A3 pages with 13 point staff
size (~ 50 minutes of huge orchestra).

Compiling the full score using manual breaks took around 7 minutes, with
the console output seemingly indicating the switch from parsing to
engraving after around 1'30.
Then I set up a system to compile a single system using skipTypesetting,
which took a little more than 1'30.

So I think when I could recompile a score that is kept in memory I could
(more or less) skip these first 1'30.

>
>> As most tweaks in the finishing stage of a project will only affect
>> the current system this would mean the recompilation can
>>
>>   * skip the Guile set-up
>>   * skip the document parsing
>>   * engrave only a single system
> Page layout, page breaking and so on are global.  LilyPond does not
> engrave single systems independently from other systems.

Of course not, but with using skipTypesetting it can engrave a score
containing only a single system. Concretely I would set the skip points
to have a "padding" of one beat before and after that system, so I'll
actually get three systems as single-system output files and can discard
the first and third one.

I already have a tool in place (in openLilyLib) where you can pass a
list of barnumbers (a.k.a the original breaks from the source) and then
ask LilyPond to compile e.g. systems 4-7, page 3 or similar.

>
>> So, does this seem feasible at all (that is: *is* there actually
>> something like a representation that could be kept in memory)?
> LilyPond reads and collects the input (including top-level input) into
> books.  They are "something like a representation" kept in memory, but
> creating them is consuming the smallest part of LilyPond's processing
> time.

As said above I'm convinced that even when it *is* the smallest part
(which I don't question) a set-up as I see it (well, it will have a
number of steps to achieve until it'll show it's real face) can make a
huge difference from the user's perspective.

Urs