Fwd: Ekiga chat

[David Garfinkle]

Hey Lilypond devs!

GSOC student here. I have some questions regarding musicxml export, since
David is out of town without internet. Here is his outline for the first
half of the project (at the bottom) and my subsequent reply + questions (at
the top). In particular, I just want to make sure I fully understand what
needs to be done with a simple example. Thanks for any help!

-David



---------- Forwarded message ----------
From: David Garfinkle <address@hidden>
Date: Wed, Jun 3, 2015 at 7:34 PM
Subject: Re: Ekiga chat
To: David Kastrup <address@hidden>


Hi David,

I think I get what's going on and what I need to do. But I would like to
make sure with an example.

Given this input file "test1.ly"
\version "2.18.2"
\score{
 \relative{a4}
}

1. a file similar to dumpit.ly would find&separate the {} blocks? and for
our simplest case we're just looking at a score block with one note

2. then for this score block we have a scheme representation that looks
like this:

(make-music
  'SequentialMusic
  'elements
  (list (make-music
    'NoteEvent
    'pitch (ly:make-pitch -1 5 0)
    'duration (ly:make-duration 2 0 1))))

which further defines a C++ representation, that I don't need to work
with..?

3. And this is where I want to make really sure I know what's going on. We
want to turn the scheme function representation, using the display-lily
machinery, into an SXML representation which looks like this: (i think?)

(list (make-music
  (list 'SequentialMusic
    (list 'elements
      (list (make-music
        'NoteEvent
        (list 'pitch (ly:make-pitch -1 5 0)
        (list 'duration (ly:make-duration 2 0 1)))))

or equivalently,

'(make-music
  (SequentialMusic
    (elements
      (make-music
      NoteEvent
      (pitch (ly:make-pitch -1 5 0))
      (duration (ly:make-duration 2 0 1))))))

... so that each scheme function is now the first element in a series of
nested lists
and this will be done in a guile-1/lilypond script

4. then a guile-2 script uses (sxml simple) module and calls sxml->xml, and
with the above example, turns our music into

<make-music>
  <SequentialMusic>
    <elements>
      <make-music>
        NoteEvent
        <pitch>
          <ly:make-pitch>-150</ly:make-pitch></pitch>
        <duration>
          <ly:make-duration>201</ly:make-duration></duration>
      </make-music>
    </elements>
  </SequentialMusic>
</make-music>

And then of course before we make it xml, we need to do an sxml->sxml
transformation which maps lilypond to the musicXML specifications.
Does this look at all right, or is there a big gap in my understanding? I
am rather confused as to what (make-music) is doing in the xml, and what it
would mean in musicXML, so I feel like I have missed something.

Thanks for any help!

Best,
David
On Tue, Jun 2, 2015 at 4:00 PM, David Kastrup <address@hidden> wrote:

>
> Ok, sketching what I consider useful next.
>
> If you run
> lilypond -d include-settings=dumpit.ly xxx.ly yyy.ly
>
> with some LilyPond input files xxx.ly and/or yyy.ly and the following
> file "dumpit.ly"
>
>
> then you'll see that each contained score will be output to the
> terminal, preceded by a statement which file this is intended for.  Now
> instead of announcing writing to a file one will actually write to a
> file, and instead of using display-lily-music one will use something
> else.  Also one will not just extract the music from the score but also
> various headers, variables and other stuff.
>
> But the music in the score is the start.
>
> The tasks we have is to convert music data structures to XML.  And we
> may have different forms of XML, and they may not correspond
> structurally with the music we start with so we might need to recognize
> patterns/structures and convert to other patterns.
>
> Now in the long term we want to convert also from XML back to LilyPond.
> And for producing syntactically valid and nicely formatted XML, it makes
> sense to use a preexisting library producing XML output from Scheme data
> structures.
>
> music -> scheme representation of XML -> XML
>
> There are currently two XML libraries I see: namely SLIB and SXML, and
> the latter is only available for GUILE-2 if I am not mistaken.  The
> representation of SLIB and SXML is quite similar: changing from SLIB to
> SXML should not be much of a problem once LilyPond has been switched to
> GUILE-2.  The time frame for that is dubious.
>
> Now the GUILE-1 SLIB however is not useful for our task since it appears
> limited to reading and interpreting XML.  It produces an SXML rendition
> of an XML file but is not capable of writing one.
>
> That is, uh, unfortunate.  Darn.
>
> The GUILE-2 SXML library has a pattern matching and transform library
> that has a few perks (namely that stuff like attribute order do not
> matter when matching XML) but those are not really important for
> converting from music->XML since om that direction the order of any
> attributes will be the same every time since it is under our control,
> and MusicXML does not use attributes much anyway I think.
>
> At any rate, the basic music->scheme/XML transformation should be doable
> by copying and changing the display-lily-music machinery in
> scm/define-music-display-methods.scm and scm/display-lily.scm (currently
> producing strings from Music) to a generate-sxml machinery (producing
> lists from Music).  In a first approach, it would seem reasonable enough
> to use the same programmatic structure and just replace its respective
> outputs, formatted strings, by the respective list representation in
> SXML.
>
> So that should give us a basic path music->SXML which then may need
> another transformation layer at the SXML level in order to get the
> structure of MusicXML.  This structural transformation could be done at
> other layers in the conversion, but since a future XML input option to
> LilyPond will require the reverse direction, it makes sense to have this
> transformation not combined with a conversion layer since then the
> patterns for recognizing the structures would be for different formats
> when converting music->XML and XML->music.  And for maintenance reason
> we'd want both of those to be using the same kind of transform
> mechanism.
>
> So since we will ultimately do the "structural transformation" as a
> stage from SXML to SXML, we do not need to design it into the first
> prototype producing XML.
>
> What's really awful is that right now we cannot use GUILE-2 in LilyPond,
> and I was mistaken about the SXML capabilities of GUILE-1.  So we'll
> need to produce the SXML representation (which is just a list) in
> LilyPond/GUILE-1, write it out to a file or a pipe (using "write" or
> similar functions writing out Scheme data), read it back into a GUILE-2
> script that does the further processing/transformation and write out the
> resulting XML.
>
> Ugh.  That's ugly.  But I can't promise a time frame for a working
> GUILE-2 inside of LilyPond: that's an urgent project but it has been
> urgent for years.
>
> So we'll need to split this into a GUILE-1 and a GUILE-2 part until the
> GUILE-2 migration of LilyPond completes independently, or we need to
> skip the intermediate SXML stage/usage altogether and produce XML
> strings right away.  But I think that that approach would prove to be a
> bad choice for longterm maintenance, so I'd rather go with split parts.
>
> So it probably makes sense to first write a GUILE-2 standalone
> application first that reads in a file in SXML (which should just be one
> "read" statement reading in a Scheme list) and uses the SXML writer
> (described in the GUILE-2 manual in the library section) for producing a
> corresponding XML file.
>
> Then write a function in GUILE-1 that writes out a Scheme list and calls
> the GUILE-2 script.  If we have that, we basically have a function in
> GUILE-1 that can be called with SXML and write it out to XML (by writing
> out the SXML to a temporary file or a pipe and calling the GUILE-2
> script on it).
>
> Ok, try digesting that, see whether you get the points of the, uh,
> somewhat changed plan and whether you see yourself approaching it in
> that manner or whether you need more pointers or think doing it
> differently may make sense.
>
> I sure wish my original plans would have corresponded better with
> reality, but then this is also supposed to be relevant experience for a
> later job and, uh, "drop the initial plan and come up with something
> that has a chance of working" is quite a standard procedure.
>
> All the best
>
> --
> David Kastrup
>
>