Re: Scheme syntax vs. other languages [was: Re: Appreciation / Financial support]

[Joseph Rushton Wakeling]

On 07/06/12 05:24, David Kastrup wrote:
> You picked a _Scheme_ function, not a music function.  That does not, I
> repeat _not_ at all show how you embed this thing into your LilyPond
> code, and we were talking about using D as an _extension_ language of
> LilyPond, not about its usefulness as a general purpose programming
> function.

I don't see any point in simply embedding D as an alternative scripting language 
to Scheme, and I'm not proposing that.

Rather, in my original post I was introducing D as a language that could be used 
to code all levels of LilyPond, from low-level internals to high-level stuff all 
the way up to scripts embedded in LilyPond input files.  That seems a pretty 
reasonable input to a discussion where the technology and architectural choices 
of LilyPond were being debated, and one of LilyPond's original authors had written:

> If I would re-do it, I would do so in a language where you can write
> have the data be inside native classes, and automate generating
> methods (setters, getters) and hooks (property callbacks), such that
> the core program wouldn't need to be aware of the scripting language.
>
> Then you could make (multiple!) scripting language bindings against
> those data, without having to integrate the language throughout.
>
> I personally have become very enamored with Go (golang.org), btw.
> Sometimes I wonder how LilyPond would turn out if I started it from
> scratch today.

I have only a very basic familiarity with Go, but if the intention would be to 
do a rewrite from the ground up, using a language that provides the features 
described here, then D would be a strong contender.

I'm not asking anyone to do that rewrite.  I'm simply pointing out that a 
possibly useful language choice exists, should a bottom-up rewrite be 
considered, and I tried to illustrate the general features that I think make it 
useful.  If that information is useful to anyone -- great.  If not -- let's move on.

Then there's a somewhat different discussion, which is the accessibility of 
Scheme as a scripting language.  If you want to argue that the accessibility 
issue is outweighed by the power and flexibility of the language, that's fine. 
I am not sure I disagree with you.  But that doesn't alter the fact that its 
syntax and style is very different from the mainstream languages that users and 
potential new contributors are more likely to be familiar with.

> The principal source of contributors are musicians, not programmers.
> C++ is much more mainstream, and will you claim that we have more people
> working with the C++ parts of LilyPond than with the Scheme parts?

Working with the internals is going to be heavy-duty no matter what the 
programming language.  On the other hand it's a reasonable concern (raised by 
someone other than me) that if too much of the internals is rewritten in Scheme, 
it may be difficult to secure new contributors to the codebase.

> How many people extend LaTeX writing Pascal code rather than TeX code?
> Which if Pascal and TeX is the more mainstream language?  Which is
> saner?  That's so far from being even competitive that it isn't funny.

... and TeX work is confined to a fairly small hardcore group, with the 
mainstream of publishing moving on to other technologies.

Besides, TeX has the advantage that when it was introduced it provided something 
that wasn't really available before, so it attracted input (both amateur and 
commercial) by virtue of being the only real show in town, and so built up a 
weight of add-on functionality that kept it relevant.

You can say something similar about your other example, Emacs and its extension 
language.  It was offering new functionality; when it first appeared, LISP was a 
much more mainstream language than it is now; and both then and now, its target 
user base is hard-core programmers, for whom LISP and its variants are very 
likely to be something they are either already familiar with or readily able to 
learn.

LilyPond doesn't have those advantages.  In terms of what it provides -- 
computer-based musical engraving -- it's preceded by multiple alternatives, 
including SCORE, Finale and Sibelius.  In terms of the surrounding programming 
environment, LISP was no longer at the heart of mainstream programming when 
LilyPond was introduced, and is even less so now.  In terms of user base, it's 
intended for musicians (who if they have encountered programming at all will 
probably have done so through a current mainstream language like C/C++, Java, 
PHP, JavaScript etc.), and music publishers (who probably don't know how to 
program, and if they need to hire programmers they will probably only be able to 
get hold of people versed in mainstream languages).

That means LilyPond has much less stature to persuade others to adopt an 
unfamiliar or non-mainstream syntax for scripting.  So while for now Scheme is 
the scripting language, in the event of a major, bottom-up rewrite it might be 
worth reconsidering the language choices throughout the whole of the codebase.


> > //////////////////////////////////////////////////////////////////
> > struct LilyPitch
> > {
> >      int octave;
> >      int notename;
>
> What is an int?  Why do I need these lines?

I defined a placeholder "LilyPitch" simply to make the subsequent code easier to 
understand.  In practice someone trying to script with LilyPond would never have 
to see this, just as the person who wrote the Scheme code didn't have to worry 
about the precise implementation of the existing LilyPond pitch data structure.

But in any case, "int" is something that will be understood by just about anyone 
with programming experience.  Someone that _doesn't_ have programming experience 
will still have to grasp the concept of types sooner or later, even if they're 
not in plain view.  You need to check for types in LilyPond Scheme scripts too 
-- (procedure? ly:music?) -- even though for a casual script you might ignore this.

> Too bad that alterations are _rationals_, namely exact numbers.  double
> arithmetic is not associative.

... so let's use a rational data type, then.
http://rosettacode.org/wiki/Arithmetic/Rational#D
http://d.puremagic.com/issues/show_bug.cgi?id=7885

I agree, though, that Scheme's support for rationals is much more elegant.

> >      pure this(int o, int n, double a)
>
> What is "pure"?  Can I eat it?

The "pure" keyword indicates that a function is pure, and this is enforced by 
the compiler/interpreter, which will throw an error if a function marked this 
way violates purity (e.g. by calling an impure function, or trying to modify 
global state).

I included it simply to illustrate some of the useful features of D, not because 
it's in any way essential to this example.

> >      in
> >      {
> >          assert(0<= n&&  n<= 6);
> >          assert(-1<= a&&  a<= 1);
> >      }
>
> What's with "in" and the braces?

Again, an illustration of a useful feature rather than an essential part of the 
example.  in{} and out{} statements in D allow the programmer to specify 
respectively preconditions for executing a statement and post-conditions for 
validating the results of a statement.

In this case, it allows you to validate that a request to create a pitch 
specifies a valid note-name and a valid pitch alteration.

> How do I know that&&  has lower
> precedence than<=?

... then put extra brackets in to ensure the precedence you want, which is 
something easy and familiar to anyone with any experience of mainstream 
programming languages.

> > pure auto naturalizePitch(LilyPitch p)
>
> What is pure?  What is auto?

Pure I've already explained.  auto indicates that the type (in this case, the 
return type) is to be inferred automatically.

> Why do I need to declare the types of p.octave etc, but not of o, n, a?

Because they can be inferred from the type of p.octave etc.  You can also 
specify them if you wish.

You may wish to have a dynamically rather than statically typed language for 
scripting, I do understand if that's a deal-breaker.

> >      // First, we disallow anything greater than +1/4-tone on E and B,
> >      // and anything less than -1/4-tone on C and F.  In other words,
> >      // no E# or B#, no Cb or Fb.
> >      if(a>  0.25&&  (n == 6 || n == 2))
>
> What are the implications of using an inexact data type?  Shouldn't I be
> doing my comparisons with some epsilon added?

With a rational data type, this isn't an issue.  You're objecting to casual 
shortcuts I took in order to have a quickly-available example, rather than on 
the substance of the point I'm making -- which is that even allowing for issues 
such as unfamiliar keywords ("pure", "auto"), the example notated in D is much 
easier to understand and follow (and tweak) for anyone with experience of 
mainstream programming languages.

> >      {
> >          a -= 0.5;
> >          n += 1;
> >      }
>
> What's a "-="?  Why do I need that semicolon before "}"?

Again, notational features familiar to anyone with experience of mainstream 
programming languages.  But write a = a - 0.5; etc. if you want.

> >      else if(a<  -0.25&&  (n == 0 || n == 3))
>
> Who will teach me about the dangling else problem?

If there's an error in the code, spell it out.

> Is "if" a function call?

Is "if" or "if-else" something that is unfamiliar to anyone with programming 
experience?

> >      {
> >          a -= 1.0;
> >          n += 1;
> >      }
> >      else if(a<  -0.5)
>
> should I not rather be indenting the "else if"?

You're welcome to follow a different indentation style if you think that 
improves readability.

> > Now, I don't think the naturalize-pitch Scheme function is necessarily
> > the best advertisement for Scheme I've ever seen, but I defy anyone to
> > tell me that the D code is not clearer to follow, even without my
> > added comments.
>
> For a musician?  Apart from the problems with the inexact numeric type?

A musician without experience of programming is going to struggle anyway, but a 
syntax closer to mainstream languages carries with it a greater range of 
learning opportunities.  A musician with experience of programming is probably 
more likely to have encountered one of the mainstream languages (C/C++, Java, 
PHP, JavaScript, ...) than one of the LISP family.

> And now you just need to recompile and link your LilyPond executable?
> And how will you _use_ this function?  Where do you have to declare it?
> Do you have to load a dynamic library?

You asked for an example where the syntax of another programming language would 
have been more accessible than Scheme -- I gave you the exact example where 
Scheme had caused me trouble.  I don't claim that this can just be trivially 
plugged into LilyPond.

But, in the event of doing a bottom-up rewrite of LilyPond in D, I think you 
would have a couple of options.  One would rely on the completion of some 
currently ongoing work in D to built a scripting module in the standard library 
which would enable execution of D code read in at runtime.  The other would 
involve doing a first pass through the .ly input files for code scripts, which 
would then be embedded as string mixins into a wrapper file and compiled on the 
fly against a dynamic library.  D's compilation times are extremely fast, so 
this would be unlikely to be much of a hit.

Attack me if you like for assuming features which are still in development, but 
I was also assuming that if a rewrite would happen _at all_, it wouldn't be any 
time soon.

> Or how would you write something like
>
> applyMusic =
> #(define-music-function (parser location func music) (procedure? ly:music?)
>     (_i"Apply procedure @var{func} to @var{music}.")
>     (func music))

Something like this:

    auto applyMusicFunction(alias func)(Parser p, Location l, LilyMusic music)
    {
        return func(music);
    }

There's also a method involving function delegates, which is a bit less elegant:

    auto applyMusicFunction(Parser p, location l, LilyMusic delegate(LilyMusic) 
func, LilyMusic music)
    {
        return func(music);
    }

I don't understand how the parser and location are used in the Scheme example 
you've provided (it doesn't seem to be documented in the CG).

> It is a bit unfair since you have not even _tried_ creating a LilyPond
> document where this function is defined and used.  You have created a
> scrap of source code dangling in the air.  That is not the hard part
> about an extension language.

It's a bit unfair to expect a simple example of how one notation is more 
accessible than another, should come with a fully-fledged implementation!

> > OK.  Here's some of what I see in D:
> >
> >    * Basic syntax is close to C/C++/Java/C# making it easy to adapt to from
> >      any of these widely-used languages.
>
> And those widely-used language that every musician is comfortable with,
> including the concept of inexact arithmetic for rationals.  The kind of
> math you learn in the school.

Exact rational types can be defined, albeit less elegantly than in Scheme.

> Nice if I am writing a packet driver.  So if we use it in LilyPond where
> C++ is used, we can use almost the same programming style, just with a
> performance hit for the pains of porting.

What you gain is that, unlike C/C++ you have the possibility to write high-level 
code _in the same language_ rather than having to have a split between fast 
low-level language and high-level scripting language; the code syntax is much 
more elegant than C++, making it easier to read and maintain; and a bunch of 
other useful features designed to help maintain code quality (built-in unittests 
just for starters).

Plus the other features I've already mentioned.

> >    * Fast to compile (faster than Go IIRC, _much_ faster than C++).
>
> Too bad that compilation times are not a problem in LilyPond.

I can't say that I would object to a cut in the compile times.  Or do you mean 
that the bottleneck is not the C++ but the font-related stuff?

> Theproblem would be rather having to compile and link _at_ _all_ just for
> writing a document.

I wouldn't think of doing a rewrite in D without the scripting issue (mentioned 
above) having been resolved.

> >    * Automatic garbage collection, but this can be disabled if needed.
>
> Why would I disable something that works and makes life easier?  Is it
> badly implemented/inefficient?

No; but in the event that you want to have precise control over when memory is 
allocated and freed, you can do so.  This is probably most relevant however to 
low-level system coding or to games developers.

> >    * Inbuilt support for functional programming, enforcement of purity etc.
> >      but with a tolerant approach that allows mutation of internal functional
> >      variables as long as it has no side-effects (e.g. my naturalizePitch is
> >      a pure function, even though the way it's written it has mutation of
> >      internal variables).
>
> Can you please explain this in terms a musician would understand?

Will this explanation of purity do? :-)
http://lilypond.org/doc/v2.15/Documentation/contributor/understanding-pure-properties

> >    * Thread-safe globals by default, inbuilt support for immutable variables,
> >      inbuilt support for both message passing and mutex-based approaches to
> >      concurrency.
>
> Can you please explain this in terms a musician would understand?  Also
> explain where and why he should acquire the computer science for being
> able to write LilyPond files?

All of this is relevant almost entirely to the internals of LilyPond.  From the 
point of view of a musician: D would probably make it easier to write code that 
would process different parts of the engraving process in parallel, which would 
speed up the production of the score.

I expect, though, that parallelizing the LilyPond engraving process is 
non-trivial.

> >    * Currently a small but very enthusiastic developer and user
> >    community, but picking up a growing interest from games developers,
> >    scientific programmers, and others who have strong joint
> >    requirements for speed, safety and ease of writing/debugging.
>
> Yup.  Sounds like a perfect match to the requirements of musicians.

Musicians presumably also need speed, safety and ease of writing/debugging? :-)

The community is a friendly and helpful one, and covers a whole range of 
different interests.

> >    It's also clear there is interest from Facebook and other
> >    organizations that have to write large-scale web applications with
> >    low per-user energy cost.
>
> A partitura is quite like a large-scale web application, I guess.

Merely pointing out that there are sufficient business interests and 
applications to give the language a sustainable future.

> > ... and other stuff; but that's enough for now, I think.
>
> Sure, enough.  Just not of the right kind.  The idea was to _solve_
> problems, not create them.

If the discussion is not useful, all you have to do is say so.  I've no wish to 
inflict unhelpful remarks on the community.

It's just that you keep challenging me to explain or respond to various points, 
so I do feel obliged to give an answer.