Re: [Gnu-arch-users] ANNOUNCEMENT -- "timestamps" optimization

[Jason McCarty]

Tom Lord wrote:
> 
> 
> The "timestamps optimization" (more properly, the "inode siganture
> optimization") is now available in tla--devo--1.1.

Great!

> 
> Here's what it does:
[snip]
> I'm seeing a little better than a 2x speed-up in `what-changed' for my
> tla trees.  I expect that the speed-up will be greater for trees using
> explicit tagging and for larger trees (and on file systems that are
> less efficient about small files).

I did a little speed testing on Linux 2.4 (ext3 fs), with patch-160 and
patch-162. Times reported are total time (including IO/system time).

When I have plenty of free memory and have emptied the kernel's file
cache, I also see a 2x speedup (13s vs. 7s), but only on the first time
what-changed is called. After that, everything is in the kernel's cache
and they run at the same speed (2.8s each).

> There's still a bit more overhead in some places -- the new code isn't
> bashful about tossing in an extra tree-traversal with stat calls on
> `commit' or `get'.
> 
> I'll be interested to hear timing change results, especially on larger
> and explicit mdoe trees.

I tested with a really tight memory situation. I need about 29MB of free
memory to keep from OOM (out-of-memory) erroring, while du reports my
tla tree as 25 MB in size. In that case, I'm seeing a 3-4x speed
improvement (11.9s vs. 3.2s) on your tla tree, but I don't have anything
bigger to test on. When I increase the amount of free memory, the
optimization's advantage is reduced. So for this to be any improvement,
my free memory has to be too small to hold the entire tree (including
pristines) in the kernel's cache.

User time was 2.7-2.8s on every run, so it looks like tree-traversal
costs trump file-comparing costs, regardless of how many files are
compared. So the real effect of the inode-signature optimization is to
reduce kernel cache usage, rather than reducing cpu time significantly.
Now only the source tree meta-data needs to be cached, instead of the
meta-data and file contents for both pristines and source. I guess I
would tout it as a reduction in soft memory requirements, with the
beneficial side-effect that it improves the kernel cache hit-rate for
large trees, and thus overall speed.

> Hopefully I haven't broken anything in the process -- very
> conservative users may want to wait a little while for braver users to
> try it out first.

I didn't do any correctness testing, but it didn't appear to do anything
wrong :)

Jason