Re: [Monotone-devel] Speedup chances

[Eric Anderson]

Markus Schiltknecht writes:
 > Some questions that arise here: Why is that specialized hex decoder so 
 > much faster that what's in botan? Does it have to be limited to 40 
 > digits? Can the botan decoder be sped up?

Probably the generality of the botan one being a full pipe, the
special case is arbitrary string -> string, not just 40 digits, that
buys you nothing since the rest of monotone wants strings.

 > Another thing to keep in mind: instead of trying hard to squeeze out the 
 > last bits of performance from the decoder, it might rather be possible 
 > to avoid en- and decoding steps completely by using a binary roster 
 > format - thus also storing binary hashes.

Revision: 464e510af4959231ff63352c902c689b0f1687aa
Branch: net.venge.monotone.experiment.performance

Modified files:
        cmd_merging.cc fake_pthread.c roster.cc roster.hh

ChangeLog: 

   * Patch to add in binary rosters; substantial (1.2x) speed
     improvement for the client on pull, some speed improvement on
     annotate (only informally tested, matters much more when annotating
     a file near the end of the roster than the beginning).  A wash on the
     server, although I haven't tested serving with an all-binary roster
     database.

Lapo Luchini writes:
 > Christof Petig wrote:
 > > Botan seems to call memset on buffer free. And since we call hex_decode
 > > some 100k times per OE roster read, this easily adds up.
 > > 
 > > I don't know whether botan's hex_decode is slow or fast, but using
 > > dynamic buffering here kills performance.
 > 
 > I don't think it's a problem of raw speed, but of different scopes: 
 > should every free of a cryptography library memset to zero in order to 
 > reduce probability of disclosure of something possibly secret?
 > I'd say yes.
 > 
 > But do monotone revision numbers need to be cleaned that way?
 > Most probably not.

See the below revision and changelog; I assume it could be extended to
other functions.

Revision: e1a721eb1b1bf8d64229419ac1f73bda0a855590
Branch: net.venge.monotone.experiment.performance

Modified files:
        botan/gzip.cpp botan/init.cpp botan/init.h monotone.cc

ChangeLog: 

Remove zeroing of memory used by Botan to do compression. 1.06x
reduction in client time, 1.02x in server time.

gzip.cpp: if we are not paranoid, don't use the paranoid memory
allocation functions.

init.{cpp,h}: define a variable for how paranoid we are going to be
about zeroing memory.

monotone.cc: set that we are not paranoid.
        -Eric