More vertical spacing confusion

[Adam Spiers]

Hi all,

I have a piece of music which should easily fit on one page, and
indeed does until I change the paper size from A4 to US Letter, at
which point the final system spills over onto a second page.  The
source and resulting PDF are here:

  http://adamspiers.org/music/lilypond/

I am using a fresh build of the main git development branch.
(Apologies for not being to include a concise test case, but obviously
the spacing issue is a symptom of stacking chords and a RhythmicStaff
on top of a normal Staff.)

After enabling spacing annotation, I trawled the documentation e.g.

  http://lilypond.org/doc/v2.15/Documentation/notation/displaying-spacing

and list archives for an explanation of the various annotated lines,
but came up short.  I see I am not the first to be a bit confused
about how all the vertical spacing variables and algorithms work, e.g.

  http://article.gmane.org/gmane.comp.gnu.lilypond.devel/31049/

So I have several questions:

  - Is the behaviour I am experiencing (a system spilling over onto a
    second page) correct or a bug?  If correct, what's the best way to
    avoid the spillage?

  - Why does each system in my PDF (URL above) have a double-ended
    grey arrow line which stretches well above the highest part of the
    system and sometimes below it too?  What does this line mean?

  - Why is there a gap in between the bottom end of one of these
    per-system arrows and the top end of the next?

  - What does the extent-estimate line refer to?

  - How do the skylines relate to / impact vertical spacing?

  - Are skylines, debugging thereof, and springs documented anywhere?
    I could only find mentions in the mailing list archives.  I see
    there was an effort to improve documentation in this area, e.g.

      http://thread.gmane.org/gmane.comp.gnu.lilypond.devel/31117/
      http://codereview.appspot.com/2316042/

    but I'm still struggling to find anything which explains them in a
    way I can understand without having to read source code.

Many thanks!
Adam