lilypond-user
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Paralellizing Lilypond [was: Re: Sibelius Software UK office shuts down]


From: Joseph Rushton Wakeling
Subject: Paralellizing Lilypond [was: Re: Sibelius Software UK office shuts down]
Date: Fri, 10 Aug 2012 11:46:49 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20120714 Thunderbird/14.0

On 07/08/12 00:01, Han-Wen Nienhuys wrote:
On Mon, Aug 6, 2012 at 6:53 PM, Joseph Rushton Wakeling
<address@hidden> wrote:
I think you're focusing on the wrong kind of architecture.

I'm talking about the architecture of computers that people can buy in
the shops today. While cute, a 192-way ARM server is useless in
realistic scenarios. See eg.
http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/pt-BR/us/pubs/archive/36448.pdf
- aka. "Let's use 9 pregnant women, we'd have a baby within the
month."

Yes, good point. Re the hardware, I agree that it's not the current off-the-shelf package, but it _is_ clearly a direction that hardware is taking into the future. It seems wise to anticipate that in planning the future architecture of Lilypond.

Regarding parallelism, actually my real concern here isn't so much doing things in parallel _per se_ so much as as Lilypond effectively breaking up its work into smaller units, which might be dealt with in parallel or sequentially. Any speed boost is nice, but not essential compared to scalability (more on that in a moment).

Unless you have a embarrassingly parallel problem to begin with (which
music typesetting is not), lots of parallelism only buys you
synchronization overhead, both lock contention at run-time, and the
overhead of having to write race-condition-free parallel code.

Note that lilypond is embarassingly parallel at the file level, so for
the regression test, we already distribute the files on as many CPUs
as we have available.

Yes, but the problem that you have there is that it requires the user to separate out the projects manually using some kind of build system like Make. I don't see why in principle Lilypond shouldn't be able to work out those independent jobs itself -- essentially any elements separated by a page break can be handled as a separate job, and there must be other further optimizations available.

For example, if you've got a 100-page full orchestral score, is it really appropriate to do a global optimization of the whole thing? How mutually dependent, really, are the first and last pages? Isn't it possible to break the work up into manageable smaller units even in the case that it's 100 pages of continuous music?

It's not just about how many cores you can use, in fact that's probably a minor issue compared to:

    -- Largest possible memory consumption and/or calculation size.  Is it
       capped or does it scale in an unlimited way with score size?

    -- Scale of rebuild process relative to size of change made.  Your
       worst-case scenario here is probably still going to be a full rebuild,
       but can you minimize the _expected_ rebuild size?

I remember well trying to build Valentin's opera, and building even one part of it took a huge amount of both time and memory. I don't want to think about the difficulty of _editing_ that score.



reply via email to

[Prev in Thread] Current Thread [Next in Thread]