Re: GOP-PROP 9: behavior of make doc

[Graham Percival]

On Wed, Aug 10, 2011 at 05:31:56AM +0000, Keith OHara wrote:
> Graham Percival <graham <at> percival-music.ca> writes:
> 
> > That proposal failed to achieve clarity and consensus
> > within a month, mostly due to the proposal being too broad.
> 
> By "to broad" I guess you mean that the old proposal affected `make bin`,
> which doesn't suffer the sea-of-output problem.
> The proposal was pretty clear, but required careful reading.

Yes and no -- the worst problem is that IIRC we reached agreement
on using some kind of log files for "make doc", but then got hung
up on arguing about "make".  However, the initial spark for this
work was the difficulty of "make doc", and the end result is that
serious work on "make doc" will be delayed by at least 6 weeks.

If the initial proposal had only dealt with "make doc", it might
have gotten approved with no trouble in 2 weeks, and we could have
had a full month of work by now.


That said, we could have still been working on the
--redirect-lilypond-output option for lilypond-book.  We'd be
thinking about our build process in the backs of our minds, but
explictly we'd be "trying to making it easier to write a thesis in
lilypond-book".  I mean, that's exactly the same use-case as our
build system, and this way we might even involve users in
debugging and/or ideas!

> However, the solidly-defined details pretty much redirect the sea
> of output to log files, similar to 
> 
>   make 2>err.log || echo "failed; see err.log"

That's not clear enough.  I'd want a separate log file for each
process (because we often have 2/4/8 processes going at once), and
to automatically echo the relevant part of the log file to the
screen.

It's not hard.  At some level, lilypond-book knows which
lily-1234.ly file it's trying to compile.  Each lily-1234.ly
contains a
  \sourcefilename "foo.ly"
that says where it came from.

Automatically extracting that information and informing the user
exactly where the problem is, could be done by me in about 2 hours
for an initial hack (30 minutes if I'm lucky), and about 4 hours
for a good job including reviews.  You could probably do it in the
same time frame.

Question is, does anybody want to spend that time if the final
answer might be "nice idea, but don't you dare touch our build
system" ?  ok, in the case of lilypond-book itself we might be
able to sneak it through (with regards to the "debugging a
lilypond-book thesis idea)... but if I was working on this with a
view to improving things for our doc writers, I'd be pissed if I
got stuff working and was told "that's fine for people writing a
thesis, but don't touch the build system".

So I wanted to get agreement in principle about this type of
system before encouraging Phil to move ahead with producing a
patch.


> >       The user may optionally request additional output to be
> > printed; this is controlled with the VERBOSE=x flag. In such
> > cases, all output will still be written to log files; the console
> > output is strictly additional to the log files.
> 
> If "all output" means the current verbose output, it might be hard
> to both get all output in log files, and desired output on screen.

Worst case: wrap the entire command in a python script.

> The easier way would be to tell the sub-processes how verbose to be,
> and tee all of stderr to file and screen.

Sure.  That's an IMO minor implementation detail, though.

> >     * Both stderr and stdout will be saved in *.log. 
> 
> Probably like you mean "in /the same/ *.log file".

Yes.

> Have I said recently how much I really really like `make bin` ? 

Sure -- but that highlights one of the huge frustrations I have
with the build system.

There's a lot of awesome stuff there.  "make bin" only needed two
lines of "syntactic sugar".  Compiling an individual manual is
something that doc people have been requesting for *years*.  I've
known that it was possible for years, and always gave a vague
guideline on how to do it.  Phil recently tried it and said that
it didn't work, and I was sufficiently annoyed that I went and
figured it out.
(but the instructions aren't in the CG, so we'll probably go
through the same thing in another 6 months)

My general impression is that 95% of whatever we want to do is
already there.  It's just that we either don't know how to do it,
or nobody takes the time to explain how to do it to other people.

The whole situation leaves a really bad taste in my mouth.  :(
- Graham