Re: [Monotone-devel] Re: [PATCH] pre-cache database file to improve response time

[Nathaniel Smith]

On Mon, Mar 20, 2006 at 07:10:54PM -0800, Joe Wilson wrote:
> >> BenoÃt Dejean wrote:
> >i think this is irrelevant. sqlite does random writes/reads. moreover,
> >if you cat'ed your dvd iso before burning it, that wouldn't speed at all
> >the burning process. but you get the point, if you're running out of
> >ram, the kernel destroys cache pages that have just been read by your
> >'preload' :)
> 
> The DVD image (which is several times the size of most machine's 
> RAM) had to be read into memory in the first place in order to be 
> burned to the DVD. Pages will be evicted and replaced as result 
> (all be it at a slower pace). But your OS still manages to recover 
> somehow.

Well, no, assuming a reasonably smart kernel, it will notice that you
are doing a linear read and aggressively drop the old pages, with the
result that mostly your DVD reader will just re-use the same pages
over and over without affecting the rest of the cache.

Some other considerations:
  -- some people have quite large databases; significantly larger than
     100MB.  Even significantly larger than main memory, in entirely
     plausible cases.
  -- there's really no reason to have any confidence that this will
     help more than it hurts; certainly it helps you on your
     benchmark, but there are a vast number of situations out there,
     and my experience with this sort of trick is that it's pretty
     random and unreliable; sometimes it can cause really
     disproportionate slowdowns.  For this case, how do we know?
     (Random example of a different situation: running monotone on a
     multi-user system.)
  -- 'checkout' is the best case for this patch, and a fairly uncommon
     operation; the patch actually pessimizes for 'status', 'diff',
     'commit', etc., which are far more common.  (Because these are
     bound by reading the workspace, not the db, and this patch
     will tend to trash the OS's cache of the workspace.)
  -- we're putting significant work into making monotone's disk
     format less seeky, which is the real solution to these sorts of
     issues, so in the best case this is only a short-term patch
     anyway... if that turns out not to be the case, then hacks like
     this can be re-evaluated.
  -- it's "bad style" -- which, regardless of whether it works or not,
     tends to give people a bad taste in their mouth, and bias them
     against monotone.  I think there's a reasonable chance that
     people would actually rather use a slower program that doesn't
     do things they perceive as offensive, than a faster one that
     does...

Have you tried running 'vacuum' and 'analyze' on your db?  It would be
interesting to know if you still see similar speedup.

You might also try the 8192 page_size... that was committed to
mainline recently as well.  (Old db's won't be automatically migrated
and will continue to work fine, anyone who cares can do a dump/reload
to get the new page size.)

Cheers,
-- Nathaniel

-- 
"If you can explain how you do something, then you're very very bad at it."
  -- John Hopfield