[Monotone-devel] times to load various things from the database
From:
Derek Scherger
Subject:
[Monotone-devel] times to load various things from the database
Date:
Sat, 10 Jan 2009 23:58:46 -0700
In 70230559a9f88f2df47217c4cc6a27d8a3160a45 I've added a few hidden commands that load various things from the database. These commands do nothing more than load every revision, roster, file or cert from the database (in topo-sorted order for revs and rosters) which involves unzipping, delta reconstruction, hash verification, etc.. Timing these commands gives some moderately interesting results:
Sat Jan 10 09:24 PM ~/monotone/mainline $ time ./mtn db load_revisions mtn: loading revisions mtn: revisions mtn: 14,464/14,499
real 0m3.288s user 0m3.120s sys 0m0.168s
Sat Jan 10 09:24 PM ~/monotone/mainline $ time ./mtn db load_rosters
mtn: loading rosters mtn: rosters mtn: 14,499/14,499
real 25m11.219s user 24m20.875s sys 0m44.263s
Sat Jan 10 09:49 PM ~/monotone/mainline $ time ./mtn db load_files mtn: loading files
mtn: files mtn: 46,646/46,646
real 3m38.611s user 3m8.468s sys 0m24.502s
Sat Jan 10 09:56 PM ~/monotone/mainline $ time ./mtn db load_certs mtn: loading certs mtn: loaded 58,304 certs
real 0m0.566s user 0m0.500s sys 0m0.064s
I think there has been some talk in the past around the idea of writing a binary revision format (which would be a big deal from a rewriting history point of view) or a hand-coded parser to speed up revision parsing. Based on the results above it would seem that there is very little to gain from this. Reading and parsing revisions is quite fast and there are probably lots of other things that would need to go much faster before this became a real problem. Reading and reconstructing files doesn't seem to be too bad either and reading certs (unconditionally, and not verifying their signatures) is also fast. However, reading rosters is comparably slow.
A long time ago I did write a binary roster_delta encoder/decoder but failed to commit it. I've finally committed it to net.venge.monotone.experiment.binary-roster-deltas. This was intended to be able to work with a database containing old
roster_deltas but there seems to be a problem that I haven't tracked
down yet.
Unfortunately it doesn't make much difference to roster reading speed:
Sat Jan 10 10:46 PM ~/monotone/mainline $ time ./mtn --db ../binary.db db load_rosters mtn: loading rosters mtn: rosters
mtn: 14,500/14,500
real 21m34.217s user 20m52.442s sys 0m40.195s
There is some improvement here but not all that much. It seems that long delta chains are likely a bigger factor. After reconstructing an old roster with a long delta chain (600 deltas), the roster for the next revision is reconstructed by rebuilding essentially the same delta chain (599 deltas) and applying it again.