Re: multiple issues with building docs (patch included)

[Adam Spiers]

On Wed, Nov 2, 2011 at 5:25 AM, Graham Percival
<address@hidden> wrote:
> On Wed, Nov 02, 2011 at 02:14:13AM +0000, Carl Sorensen wrote:
>> On 11/1/11 5:44 PM, "Adam Spiers" <address@hidden> wrote:
>> >`make doc' works, but takes too long to be of practical use except as
>> >a final sanity check before submitting a patch upstream.
>
> You can use multiple threads.

I already am.  Even with make -j3 CPU_COUNT=3 it still takes ages on
my rather modest AMD Athlon 64 X2.

>> >http://lilypond.org/doc/v2.15/Documentation/contributor/generating-documen
>> >tation#building-a-single-document
>> >
>> >explains how to build a single document, but only gives a PDF as an
>> >example.  I wanted to rebuild the split-page HTML version of the
>> >Notation Reference, but couldn't figure out how.  Instead I managed to
>> >build the big page version via:
>> >
>> >  make out=www out-www/notation-big-page.html
>>
>> AFAICS, the single manual doc build is not really functional.
>
> Nonsense, it should be perfectly fine.

It *seemed* to work for me, but like I said, the images are missing,
rendering it useless for my current work.

> I'm not certain about the
> big-page, but
>  make out=www out-www/notation/index.html
> should be no problem.

IIRC I tried that, and it seemed to behave as a NOOP - do the
makefiles register dependencies on index.html for all the subsections?

>> >    This script will require customization for your site if your
>> >    LilyPond git repository is anyplace but $HOME/lilypond.
>>
>> Lilypond-git is the canonical location now, instead of lilypond.  If
>> scripts/auxiliar/doc-section.sh does not refer to liypond-git, it should
>> be fixed.
>
> http://code.google.com/p/lilypond/issues/detail?id=1236
>
> been on the books for over a year.

Thanks, I've added my thoughts to that issue.

>> >When building in a build/ subdirectory as recommended by the Guide,
>> >this path is wrong; it should be:
>> >
>> >    cp $LILYPOND/build/Documentation/out/version.itexi
>> >$HOME/lilypond/tempdocs
>>
>> Actually, the script should be changed to refer to
>> $LILYPOND/build/tempdocs.
>
> yep.  Well, $LILYPOND_GIT/build/

Please see my comment on the above issue - any reason to prefer use of
this environment variable over auto-detection even where it is easily
achievable?

>> >Couldn't this simply be included in the script rather than as an extra
>> >step in the instructions?
>>
>> It certainly could.
>
> yep.

I did it in my patch, in fact :-)  See below.

>> >"/home/adam/lilypond-git/tempdocs/chords/out/snippet-names--514510028.ly"
>> >Child returned 1
>> >Lilypond-book returned code 1
>>
>> Now you are running into the key challenge for building docs.
>>
>> If you run
>>
>> Lilypond tempdocs/out/e5/lily-0f1202dd.ly, you'll get to see the error
>> message.  In fact, there's probably a logfile in tempdocs/out/e5 that has
>> the error message.

I already looked for a logfile there but couldn't see one.  Next time it happens
I'll try running lilypond on that file manually.

> or you could just look at tempdocs/out/e5/lily-0f1202dd.ly ,
> delete a few lines from it so you can compile it on the
> command-line, then debug that.
>
>> >I'm not sure how to debug this because there doesn't seem to be a
>> >useful error either on STDOUT or in a log file anywhere.
>
> yep.
> after about 2 months of debate, we've adopted
> http://lilypond.org/~graham/gop/gop_9.html
> but not much work has gone into implementing it yet.

Ahah!  Looks very promising.

> Incidently, you may want to browse the proposals:
> http://lilypond.org/~graham/gop/index.html
> to see what kind of mess we're in.

Wow.  There sure is a lot to do, but I certainly wouldn't call it a
mess; in fact I think this is probably the most thoughtfully organized
F/OSS project I've ever seen, and way better than many commercial
ventures.

>> >Finally, tempdocs is not in .gitignore.
>
> doc-section.sh isn't really supported, and in any case I would
> personally set tempdocs to be /tmp/ramdisk/lily-quick-docs or
> something like that... so I'm not keen on that change.

OK, I've changed it to honour three environment variables:

   LILYPOND_GIT         (auto-detects if not set)
   LILYPOND_BUILD_DIR   (defaults to $LILYPOND_GIT/build)
   LILYPOND_TEMPDOCS    (defaults to $LILYPOND_BUILD_DIR/tempdocs)

Is that better?  I've also adapted cg-section.sh in exactly the same
way.  Patch below ...

> Thanks so much for the patch to our development stuff, though!  I
> can't honestly recommend that anybody get involved in lilypond
> development at the moment, but it's great to see people still
> trying to do it and making the process easier for the "next
> generation".

Hah :-) As with your other mails, I really appreciate your honesty.
It would be far more disheartening to feel I was being tricked into
spending unexpected amounts of time on climbing a steep learning curve
without adequate support.

>From 6ab814b6456bec44dd816a3ab24a5e43731b1686 Mon Sep 17 00:00:00 2001
From: Adam Spiers <address@hidden>
Date: Wed, 2 Nov 2011 15:50:47 +0000
Subject: [PATCH] Usability improvements to doc-section.sh and corresponding
 section of CG.

- Now honors LILYPOND_GIT, LILYPOND_BUILD_DIR, and LILYPOND_TEMPDOCS
  environment variables (all optional).
- Provides usage help text if '-h' or '--help' or incorrect arguments
  are passed.
- Outputs more helpful message upon completion.
---
 Documentation/contributor/doc-work.itexi |   12 ++
 scripts/auxiliar/doc-section.sh          |  180 ++++++++++++++++++++---------
 2 files changed, 136 insertions(+), 56 deletions(-)

diff --git a/Documentation/contributor/doc-work.itexi
b/Documentation/contributor/doc-work.itexi
index 0aabc71..41bc371 100644
--- a/Documentation/contributor/doc-work.itexi
+++ b/Documentation/contributor/doc-work.itexi
@@ -1438,6 +1438,18 @@ You can then see the generated document for the
section at
 tempdocs/pitches/out/pitches.html
 @end example

+According to 
@uref{http://code.google.com/p/lilypond/issues/detail?id=1236,Lilypond
issue 1236}
+the location of the lilypond git tree is taken from @code{$LILYPOND_GIT}
+if specified, otherwise it is auto-detected.
+
+It is assumed that compilation takes place in the @file{build/}
+subdirectory, but this can be overridden by setting the environment
+variable @code{LILYPOND_BUILD_DIR}.
+
+Similarly, output defaults to @file{build/tempdocs/} but this can be
+overridden by setting the environment variable
address@hidden
+
 This script will not work for building sections of the
 Contributors' guide.  For building sections of the Contributors'
 Guide, use:
diff --git a/scripts/auxiliar/doc-section.sh b/scripts/auxiliar/doc-section.sh
index e9f94d1..cd7c23f 100755
--- a/scripts/auxiliar/doc-section.sh
+++ b/scripts/auxiliar/doc-section.sh
@@ -16,6 +16,18 @@
 #  At the end of the run, the user is prompted whether or not to
 #  remove the generated files.
 #
+#  According to http://code.google.com/p/lilypond/issues/detail?id=1236
+#  the location of the lilypond git tree is taken from $LILYPOND_GIT
+#  if specified, otherwise it is auto-detected.
+#
+#  It is assumed that compilation takes place in the build/
+#  subdirectory, but this can be overridden by setting the environment
+#  variable LILYPOND_BUILD_DIR.
+#
+#  Similarly, output defaults to build/tempdocs/ but this can be
+#  overridden by setting the environment variable LILYPOND_TEMPDOCS.
+#
+#
 #  Known limitations:
 #
 #     * Doesn't use website css files
@@ -23,84 +35,140 @@
 #     * Won't build Contributors' Guide; see scripts/auxiliar/cg-section.sh
 #

-cd "`dirname $0`"
-cd ../..
-LILY_SRC_DIR="`pwd`"
+usage () {
+    # Call as: usage [EXITCODE] [USAGE MESSAGE]
+    exit_code=1
+    if [[ "$1" == [0-9] ]]; then
+        exit_code="$1"
+        shift
+    fi
+    if [ -n "$1" ]; then
+        echo "$*" >&2
+        echo
+    fi
+
+    cat <<EOF >&2
+Usage: $0 MANUAL SECTION
+
+e.g. $0 notation rhythms
+EOF
+    exit "$exit_code"
+}
+
+if [ "$1" == '-h' ] || [ "$1" == '--help' ] || [ $# != 2 ]; then
+    usage 0
+fi
+
+
+if [ -n "$LILYPOND_GIT" ]; then
+    echo "Using source tree from value of \$LILYPOND_GIT: $LILYPOND_GIT"
+else
+    cd "`dirname $0`"
+    cd ../..
+    LILYPOND_GIT="`pwd`"
+    echo "\$LILYPOND_GIT was not set; auto-detected source tree at
$LILYPOND_GIT"
+fi
+
+if test ! -e "$LILYPOND_GIT/DEDICATION"; then
+    echo "Error: $LILYPOND_GIT did not look like a LilyPond source
tree; aborting." >&2
+    exit 1
+fi

-BUILD_DIR="$LILY_SRC_DIR/build"
-DOC_DIR="$BUILD_DIR/tempdocs"
-LILYPONDBOOK="$BUILD_DIR/out/bin/lilypond-book"
+: "${LILYPOND_BUILD_DIR:=$LILYPOND_GIT/build}"
+DOC_DIR="${LILYPOND_TEMPDOCS:-$LILYPOND_BUILD_DIR/tempdocs}"
+LILYPOND_BOOK="$LILYPOND_BUILD_DIR/out/bin/lilypond-book"
 TEXI2HTML="texi2html"
-REFCHECK="$LILY_SRC_DIR/scripts/auxiliar/ref_check.py"
+REFCHECK="$LILYPOND_GIT/scripts/auxiliar/ref_check.py"

-DIRECTORY=$1
-NAME=$2
-OUTPUT_DIR=$DOC_DIR/$NAME
+MANUAL="$1"
+SECTION="$2"
+OUTPUT_DIR="$DOC_DIR/$SECTION"
+MANUAL_PATH="$LILYPOND_GIT/Documentation/$MANUAL"
+SECTION_PATH="$MANUAL_PATH/$SECTION.itely"

-if test ! -d $BUILD_DIR; then
-  cat <<EOF >&2
-This script assumes that you compile within a build/ subdirectory of your
-source tree, as recommended by the Contributor's Guide.
-EOF
-  exit 1
+if test ! -d "$LILYPOND_BUILD_DIR"; then
+    echo "$LILYPOND_BUILD_DIR did not exist; check your setting of
LILYPOND_BUILD_DIR. Aborting." >&2
+    exit 1
 fi

-if test ! -d $DOC_DIR; then
-  mkdir $DOC_DIR
-  cp $BUILD_DIR/Documentation/out/version.itexi $DOC_DIR
+if test ! -d "$DOC_DIR"; then
+    mkdir "$DOC_DIR"
+    cp "$LILYPOND_BUILD_DIR/Documentation/out/version.itexi" "$DOC_DIR"
 fi
-if test ! -d $OUTPUT_DIR; then
-  mkdir $OUTPUT_DIR
+if test ! -d "$OUTPUT_DIR"; then
+    mkdir "$OUTPUT_DIR"
 fi
-if test ! -d $OUTPUT_DIR/out; then
-  mkdir $OUTPUT_DIR/out
+if test ! -d "$OUTPUT_DIR/out"; then
+    mkdir "$OUTPUT_DIR/out"
 fi

-cp $LILY_SRC_DIR/Documentation/common-macros.itexi
$OUTPUT_DIR/common-macros.itexi
-cp $LILY_SRC_DIR/Documentation/macros.itexi $DOC_DIR/macros.itexi
-cp $DOC_DIR/version.itexi $OUTPUT_DIR/version.itexi
+cp "$LILYPOND_GIT/Documentation/common-macros.itexi"
"$OUTPUT_DIR/common-macros.itexi"
+cp "$LILYPOND_GIT/Documentation/macros.itexi" "$DOC_DIR/macros.itexi"
+cp "$DOC_DIR/version.itexi" "$OUTPUT_DIR/version.itexi"
+
+if test -e "$OUTPUT_DIR/$SECTION.html"; then
+    rm "$OUTPUT_DIR/$SECTION.html"
+fi

-if test -e $OUTPUT_DIR/$NAME.html; then
-  rm $OUTPUT_DIR/$NAME.html
+if test -e "$OUTPUT_DIR/out/$SECTION.texi"; then
+    rm "$OUTPUT_DIR/out/$SECTION.texi"
 fi

-if test -e $OUTPUT_DIR/out/$NAME.texi; then
-  rm $OUTPUT_DIR/out/$NAME.texi
+if test ! -d "$MANUAL_PATH"; then
+    echo "$MANUAL_PATH was not a valid directory; is $MANUAL a valid
manual?" >&2
+    exit 1
 fi

-echo "Running $LILYPONDBOOK"
-$LILYPONDBOOK \
+if test ! -e "$SECTION_PATH"; then
+    echo "$SECTION_PATH did not exist; is $SECTION a valid section in
the $MANUAL manual?" >&2
+    exit 1
+fi
+
+echo "Running $LILYPOND_BOOK"
+"$LILYPOND_BOOK" \
         -f texi-html \
-        -I $LILY_SRC_DIR/Documentation/snippets \
-        -I $LILY_SRC_DIR/Documentation/snippets/new \
-        -I $LILY_SRC_DIR/input/manual \
-        -I $LILY_SRC_DIR/Documentation \
-        -I $LILY_SRC_DIR/Documentation/included  \
-        -I $LILY_SRC_DIR/Documentation/pictures \
-        -o $OUTPUT_DIR/out \
-        $LILY_SRC_DIR/Documentation/$DIRECTORY/$NAME.itely
+        -I "$LILYPOND_GIT/Documentation/snippets" \
+        -I "$LILYPOND_GIT/Documentation/snippets/new" \
+        -I "$LILYPOND_GIT/input/manual" \
+        -I "$LILYPOND_GIT/Documentation" \
+        -I "$LILYPOND_GIT/Documentation/included"  \
+        -I "$LILYPOND_GIT/Documentation/pictures" \
+        -o "$OUTPUT_DIR/out" \
+        "$SECTION_PATH"
 BOOKRC=$?
 if [ "$BOOKRC" != 0 ]; then
-  echo "Lilypond-book returned code $BOOKRC"
-  exit $BOOKRC
+    echo "Lilypond-book returned code $BOOKRC"
+    exit $BOOKRC
 fi

-echo Running RefCheck
-python $REFCHECK
-
-cd $DOC_DIR
-if test -f $OUTPUT_DIR/out/$NAME.texi; then
-  echo Running texi2html
-  cat $DOC_DIR/macros.itexi $OUTPUT_DIR/out/$NAME.texi > $OUTPUT_DIR/$NAME.texi
-  $TEXI2HTML \
-    --no-validate \
-    --output=$OUTPUT_DIR/out/$NAME.html \
-    --I=$OUTPUT_DIR/out \
-    $OUTPUT_DIR/$NAME.texi
+echo "Running RefCheck"
+python "$REFCHECK"
+
+cd "$DOC_DIR"
+if test -f "$OUTPUT_DIR/out/$SECTION.texi"; then
+    echo "Running $TEXI2HTML"
+    cat "$DOC_DIR/macros.itexi" "$OUTPUT_DIR/out/$SECTION.texi" >
"$OUTPUT_DIR/$SECTION.texi"
+    "$TEXI2HTML" \
+        --no-validate \
+        --output="$OUTPUT_DIR/out/$SECTION.html" \
+        --I="$OUTPUT_DIR/out" \
+        "$OUTPUT_DIR/$SECTION.texi"
 fi

+cat <<EOF
+
+The $SECTION section of the $MANUAL manual should now be viewable at
+
+  $OUTPUT_DIR/out/$SECTION.html
+
+If you want to keep the generated docs around for a while, answer
+'n' to the next question.  If you only needed them to quickly check
+something, view them now and then answer 'y' when you're done.
+
+EOF
+
 read -p "rm -rf $OUTPUT_DIR ? (y/n): "
 if [ "$REPLY" = "y" ]; then
-  echo "deleting files"
-  rm -rf $OUTPUT_DIR
+    echo "deleting files"
+    rm -rf "$OUTPUT_DIR"
 fi
-- 
1.7.6.4