Re: [Monotone-devel] monotone log is too slow

[Nathaniel Smith]

On Sat, Mar 11, 2006 at 04:34:10PM +0100, Benoît Dejean wrote:
>       Hi, i'm using monotone 0.25 at work. It's sometimes a bit slow on my
> 2GHZ computer, but that's fine. Except when i need to 'monotone log
> --brief <somefile>', sometime it take up to a minute. No matter, that's
> fine. Database is 50MB.
> 
>       So today, at home, on my PPC 1GHz with a devel version of monotone
> (pulled today), i wanted to do a diff between recent update of po/fr.po.
> I first did a 'monotone log po/fr.po' but canceled it thinking 'it's too
> slow, i only need the revision names, i don't care about commit
> messages'. I tried a simple 'monotone log --brief po/fr.po' on

Commit messages are unlikely to make any noticeable difference in
speed :-).  It's probably unpacking every roster to figure out which
revisions the file changed in... ("Did it change in this rev?  no.
Okay, did it change in the next rev back?  no.  Okay, did it change in
the next rev back?  no. Okay, ...")

> net.venge.monotone. Ouch. --last 10 saved my life. It took about 15s.
> 
> So i added timings (attached patch) and ran a full log. It took me half
> an hour to print the full brief log for a single file. Hot cache,
> monotone binary and database in ram. No I/O. 100% CPU during 1900
> seconds. I've attached the trace and made a nice chart with gnumeric.
> What's worrying me is the shape of the line ... it's exponential. Would
> it take me 2H to log a file in a few months ?

Hmm, strange -- I would have expected steep linear growth, not
anything exponential.  Profiling seems called for...

Oh, wait -- but perhaps log is traversing the history graph in a
really suboptimal order.  In fact, I know that it does this
breadth-first thing, that turns out to be somewhat annoying for
reading logs (this is why changes can appear out of order in log, for
instance)... and this is also a pretty pessimal order for
reconstructing versions.  This _identical_ issue actually came up in
the netsync speedup discussions, since current netsync uses a similar
traversal order; it's been fixed on the vlog branch, anyway, by
09cadd59949a605d41d1fd1c6550be73a5735edf.

Yet another reason to fix log's traversal ordering.

The Real answer, though (assuming this analysis is correct at all,
profiling is still a useful sanity check :-)), is to finish Emile's
work on per-file DAGs, and then teach log to use it... (which might
require Derek's work on more cleanly rosterifying restrictions...)

-- Nathaniel

-- 
"...All of this suggests that if we wished to find a modern-day model
for British and American speech of the late eighteenth century, we could
probably do no better than Yosemite Sam."