Re: 'make web' fails (and takes forever)

[David Bobroff]

On Fri, 2005-06-24 at 01:56 +0200, Han-Wen Nienhuys wrote:
> David Bobroff wrote:
> > On Thu, 2005-06-23 at 22:59 +0200, Han-Wen Nienhuys wrote:
> > 
> >>David Bobroff wrote:
> >>
> >>>I've tried a few times recently to build the web-docs from CVS.  I've
> >>>noticed that the build time for LilyPond itself seems to be much shorter
> >>>than it used to be.  On the other hand, the build time for the web-docs
> >>>seems extremely long.  I tried building the web-docs (after successfully
> >>>building LilyPond) from CVS ChangeLog 1.3812  ('release commit') and it
> >>>happily chugged away apparently building the website.  After it had been
> >>>doing this for over 7 hours I went to bed.  When I got up this morning I
> >>>saw this in the console:
> >>
> >>Can you give me a tiny bit more of the output? It will probably say 
> >>"Failed files: <foo.ly>" somewhere.
> > 
> > 
> > Ok, but I'll have to run 'make web' again.  I suspect it will take many
> > hours to get there again but I'll likely be able to send something in
> > the morning.

Ok, it's morning.  Here's a longer end of the output. I started from a
point where it looked to be working:

Parsing...
Renaming input to: `metronome-marking.ly'
Interpreting music... [3]
Preprocessing graphical objects...
Calculating line breaks... [3][4]
Writing header field `texidoc' to `lily-1740870191.texidoc'...
Writing header field `texidoc' to `lily-1740870191.texidoc'...
Writing lily-1740870191-systems.tex...
Writing lily-1740870191-systems.texi...
Layout output to `lily-1740870191-1.eps'...
Layout output to `lily-1740870191.eps'...
Converting to PNG...
Processing `out-www/collated-files.tely:1638 (lily-1652220718.ly)'
Parsing...
Renaming input to: `mm-rests2.ly'
Interpreting music... [8][12]
Preprocessing graphical objects...
Calculating line breaks... [3][6][9][12][13]
Writing header field `texidoc' to `lily-1652220718.texidoc'...
Writing header field `texidoc' to `lily-1652220718.texidoc'...
Writing lily-1652220718-systems.tex...
Writing lily-1652220718-systems.texi...
Layout output to `lily-1652220718-1.eps'...
Layout output to `lily-1652220718.eps'...
Converting to PNG...
Processing `out-www/collated-files.tely:1647 (lily-834776204.ly)'
Parsing...
Renaming input to: `multi-measure-rest-center.ly'
Interpreting music... [3]
Preprocessing graphical objects...
Calculating line breaks... [3][4]
Writing header field `texidoc' to `lily-834776204.texidoc'...
Writing header field `texidoc' to `lily-834776204.texidoc'...
Writing lily-834776204-systems.tex...
Writing lily-834776204-systems.texi...
Layout output to `lily-834776204-1.eps'...
Layout output to `lily-834776204.eps'...
Converting to PNG.../usr/src/lilypond/share/lilypond/2.5.32/scm/ps-to-
png.scm:91:16: In procedure system in expression (system cmd):
/usr/src/lilypond/share/lilypond/2.5.32/scm/ps-to-png.scm:91:16: Cannot
allocate memory
lilypond-book.py: warning: `lilypond' failed (status 2) (ignored)

lilypond-book.py: error: Process /usr/src/lilypond/lily/out/lilypond --
backend=eps --formats=ps,png --header=texidoc -I ./input/test -
dinternal-type-checking -danti-alias-factor=2  -
I /usr/src/lilypond/input/regression  -
I /usr/src/lilypond/input/regression  -
I /usr/src/lilypond/input/regression/out-www  -I /usr/src/lilypond/input
-I /usr/src/lilypond/input/regression  -I /usr/src/lilypond/input/test
-I /usr/src/lilypond/input/tutorial  -I /usr/src/lilypond/mf/out  -
I /usr/src/lilypond/mf/out snippet-map.ly lily-1956597462 lily-311413851
lily-1627534793 lily-1097443063 lily-210382186 lily-2036652784
lily-885928158 lily-1727067039 lily-437356607 lily-2140417309
lily-480868577 lily-1759750501 lily-1189324376 lily-931321402
lily-1776861541 lily-50750963 lily-562104431 lily-980184230
lily-2083426454 lily-1087557510 lily-1543451470 lily-2005545575
lily-675728828 lily-2114415505 lily-443829910 lily-576354807
lily-1186204425 lily-238271921 lily-581475682 lily-2092810622
lily-1737126160 lily-1925784050 lily-1917879539 lily-1172160865
lily-1920652884 lily-1379504195 lily-748493478 lily-1982992675
lily-544946704 lily-2096204882 lily-1054105038 lily-758797365
lily-537168227 lily-675640774 lily-1549631703 lily-1093120878
lily-1492170866 lily-230214834 lily-236959324 lily-862451569
lily-955178042 lily-403892608 lily-826831307 lily-2112294246
lily-1569836352 lily-931500941 lily-371000131 lily-1033383822
lily-1786326899 lily-470405488 lily-1386948243 lily-1712884898
lily-61297531 lily-1446922892 lily-836947082 lily-18381457
lily-941713124 lily-1625534235 lily-2018026026 lily-1019464378
lily-1567401639 lily-684037413 lily-2064877304 lily-359170197
lily-2147305447 lily-581374284 lily-1384039266 lily-783356452
lily-1626160672 lily-127317372 lily-1040857085 lily-1435001409
lily-1907228445 lily-377968127 lily-1981907779 lily-1952518784
lily-296821422 lily-1064980618 lily-139147634 lily-858354388
lily-1240303342 lily-498961823 lily-1694754041 lily-857966633
lily-493128987 lily-245227244 lily-770896615 lily-1448347512
lily-162159653 lily-1603151806 lily-1161024345 lily-947083620
lily-482565241 lily-1816870688 lily-343486016 lily-1529693362
lily-1704233046 lily-362840184 lily-1734808312 lily-1799759304
lily-1621151921 lily-850643794 lily-1609345100 lily-352285370
lily-252861281 lily-232868728 lily-63578255 lily-373727935
lily-1427729978 lily-1188889802 lily-866891020 lily-908890961
lily-1776155966 lily-1016581711 lily-983259310 lily-1848091594
lily-855065904 lily-1138436460 lily-1062812767 lily-125312593
lily-780636626 lily-539801114 lily-261950139 lily-191185564
lily-373332627 lily-39505122 lily-976759915 lily-1070982198
lily-1429775931 lily-2527433 lily-766876457 lily-2081475428
lily-1223182807 lily-1936381098 lily-628203336 lily-2011624244
lily-1620427779 lily-794222677 lily-332798096 lily-1884238327
lily-1813999370 lily-476022491 lily-2024668293 lily-632739817
lily-1001146307 lily-1617161696 lily-1436518898 lily-876180898
lily-1404555604 lily-1287166836 lily-855408736 lily-1942493832
lily-575851766 lily-1851433247 lily-436848402 lily-1638955211
lily-1653274601 lily-133335100 lily-1908151971 lily-1875117730
lily-1863607283 lily-1316904541 lily-1628817120 lily-336699223
lily-1676075135 lily-1542879372 lily-967376154 lily-1591568183
lily-1740870191 lily-1652220718 lily-834776204 lily-610972333
lily-1215771598 lily-108435641 lily-279451553 lily-914251638
lily-1661964650 lily-476654158 lily-538104078 lily-1847231957
lily-1946369396 lily-1177065164 lily-1060604839 lily-1102558477
lily-1905465911 lily-2025868162 lily-780131354 lily-643373098
lily-497589969 lily-1297535747 lily-2140837669 lily-386880419
lily-1223961634 lily-2066792483 lily-1791812667 lily-799536040
lily-657953591 lily-515849639 lily-742203680 lily-1910919554
lily-529444259 lily-1463262684 lily-1279720688 lily-1518238277
lily-1942520041 lily-403053247 lily-2000308022 lily-1498292452
lily-1844790090 lily-1803304010 lily-712428169 lily-1670943197
lily-1660798746 lily-1491869096 lily-47031704 lily-615229588
lily-2040733997 lily-254475965 lily-1228731626 lily-552612306
lily-732519969 lily-170651499 lily-1154054209 lily-1689952094
lily-2125322335 lily-221674396 lily-2019941713 lily-1621205123
lily-2056885282 lily-125491874 lily-1500690887 lily-90537264
lily-1731984094 lily-177930174 lily-112872452 lily-1722901935
lily-1895314988 lily-766142012 lily-257688149 lily-89712742
lily-284108407 lily-1672239646 lily-1209119933 lily-189772630
lily-1274192680 lily-1584654252 lily-1246571364 lily-778611536
lily-860222334 lily-1244506898 lily-1636255472 lily-646564080
lily-1265617817 lily-1473971792 lily-1010047680 lily-2025075965
lily-533842382 lily-2102532812 lily-228412531 lily-1772801278
lily-1263064352 lily-753391318 lily-370495876 lily-1740255689
lily-480595953 lily-554537706 lily-1332325516 lily-795061770
lily-1375785811 lily-1979140482 lily-2133215477 lily-494422170
lily-1705487128 lily-1655398536 lily-1950885198 lily-1977189579
lily-1641325963 lily-994618906 lily-1509490801 lily-355056062
lily-554243517 lily-2103745845 lily-1680550564 lily-1407142345
lily-1433726746 lily-1846489953 lily-1475385907 lily-171777049
lily-1171411721 lily-983870597 lily-1712773793 lily-1337166240
lily-1738786581 lily-1310022710 lily-1833079482 lily-1705974693
lily-641794372 lily-2084205395 lily-140538283 lily-718114653
lily-42424608 lily-719980939 lily-871877723 lily-1753075033
lily-663862715 lily-153457027 lily-402163154 lily-748943569
lily-1114183060 lily-711533762 lily-715505532 lily-1772987465
lily-1266556733 lily-391069729 lily-726220920 lily-838213450
lily-798161864 lily-992015228 lily-591999069 lily-47384707
lily-2113223659 lily-713981480 lily-381973821 lily-504789110
lily-1895614935 exited unsuccessfully.
Removing `collated-files.texi'
Traceback (most recent call last):
  File "../../scripts/lilypond-book.py", line 1616, in ?
    main ()
  File "../../scripts/lilypond-book.py", line 1599, in main
    ly.exit (1)
  File "/usr/src/lilypond/share/lilypond/2.5.32/python/lilylib.py", line
132, in exit
    raise _ ('Exiting (%d)...') % i
Exiting (1)...
make[3]: *** [out-www/collated-files.texi] Error 1
make[3]: Leaving directory `/usr/src/lilypond/input/regression'
make[2]: *** [WWW] Error 2
rm out-www/sakura-sakura.ly out-www/typography-demo.ly out-www/puer-
fragment.ly out-www/les-nereides.ly out-www/wilhelmus.ly
make[2]: Leaving directory `/usr/src/lilypond/input'
make[1]: *** [WWW] Error 2
make[1]: Leaving directory `/usr/src/lilypond'
make: *** [web] Error 2
You have new mail in /var/spool/mail/root
address@hidden lilypond]#

[NB I recall that it was taking a Real Long Time to convert to PNG in
these last two instances (i.e. for mm-rests2.ly and multi-measure-rest-
center.ly).  Could the memory leak be leading to the failure?]

> > 
> >>>While it was working I saw that it would take a lot of time at the
> >>>points where it was converting to PNG.  I didn't actually time any of
> >>>these events but it was on the order of minutes.
> >>
> >>PNG generation has become a bit slower, because we render at higher 
> >>resolution and then scale down the image. However, this should not make 
> >>that much of a difference.
> > 
> > 
> > I *think* I recall that the really slow PNG generation began some ways
> > into the build.  Can't be sure about that, though.
> 
> oh yeah, there's another thing: GUILE has a memory leak, which means 
> that lilypond-book run slower the farther you get.   I already fixed 
> this in GUILE, but the GUILE guys are kind of slow with doing releases.

That would explain it.  I opened up another terminal and ran 'top' to
see what was using so many resources.  I saw that LilyPond seemed to be
using loads of memory at times (ca. 70% of system memory).

-David