[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Gnu-arch-users] [PATCH] arch speedups on big trees
From: |
Andrew Suffield |
Subject: |
Re: [Gnu-arch-users] [PATCH] arch speedups on big trees |
Date: |
Fri, 19 Dec 2003 19:19:21 +0000 |
User-agent: |
Mutt/1.5.4i |
On Fri, Dec 19, 2003 at 11:08:45AM -0800, Tom Lord wrote:
>
>
> > From: Chris Mason <address@hidden>
>
> > I've been playing around with a few ideas to improve arch performance on
> > large source trees, mostly in the area of applying changesets, and
> > creating changesets. I've got a sample archive here with 100 changesets
> > on top of the linux 2.6 kernel, and vanilla arch takes a number of
> > minutes to apply them all (15-30 seconds per changeset via tla replay)
>
> > This is primarily because arch is doing an inventory of the source tree
> > before each changeset, my patch changes things to inventory only the
> > files touched by the changeset instead. It sends a table of the
> > candidate files to the inventory funcs, and this brings the time to
> > replay my 100 changesets to ~4 seconds.
>
> Holy crap! Really?
This feels intuitively right to me (ie, order of magnitude is
reasonable). I'm fairly sure I've noted before now that inventory and
the inode-sigs check are still the things that tla spends most of its
time doing on non-trivial trees - it's better than it was, and now
"fast enough" for small trees, but there's still some way to
go. stat() is slow, and open()/read() even slower.
Some rough numbers:
On a smallish tree with 740 files, here's the top of strace -c when I
apply about 20 changesets in one run:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
33.87 0.505745 6 79916 write
15.30 0.228506 6 40300 3239 lstat64
15.04 0.224609 7 33808 18 open
10.63 0.158737 3 55474 read
6.12 0.091420 3 33837 close
(Total runtime, about 3 seconds; percentages are of the total syscall
time, not the total runtime)
The lstat64s are more or less entirely inode-sigs checks, and the
open/read/closes are mostly reading explicit tag files.
The writes are mostly writing inode-sigs files (why is it writing so
many? looks to be one-per-changeset-applied).
All of these things are happening too damn often, and even on a
smallish tree they're taking up a fair amount of time. On a really big
tree, like linux, they're going to dominate. For long series of
changeset application, it's going to be *horrible*.
(Note, this is on a development box which is probably about as optimal
for tla as you are going to get, although I could sneak another *1.5
or so by using reiserfs; only server-class hardware is likely to be
significantly better. Most people wouldn't be anywhere near this
fast).
--
.''`. ** Debian GNU/Linux ** | Andrew Suffield
: :' : http://www.debian.org/ |
`. `' |
`- -><- |
signature.asc
Description: Digital signature
- [Gnu-arch-users] [PATCH] arch speedups on big trees, Chris Mason, 2003/12/19
- Re: [Gnu-arch-users] [PATCH] arch speedups on big trees, Chris Mason, 2003/12/19
- Re: [Gnu-arch-users] [PATCH] arch speedups on big trees, Tom Lord, 2003/12/19
- Re: [Gnu-arch-users] [PATCH] arch speedups on big trees, Tom Lord, 2003/12/19
- Re: [Gnu-arch-users] [PATCH] arch speedups on big trees, Chris Mason, 2003/12/19
- Re: [Gnu-arch-users] [PATCH] arch speedups on big trees, Tom Lord, 2003/12/19
- Re: [Gnu-arch-users] [PATCH] arch speedups on big trees, Andrew Suffield, 2003/12/19
- Re: [Gnu-arch-users] [PATCH] arch speedups on big trees, Tom Lord, 2003/12/19