Re: Surely 'bzr update' shouldn't be this slow?

[Stephen J. Turnbull]

Alan Mackenzie writes:

 > OK.  Hopefully the other branches are referred to via a relative path
 > rather than an absolute one.

Paths are resolved in the usual way on GNU/Linux and other POSIX
systems (ie, the shell globs, evaluates environment variable
references, and interprets special path elements like ~), and bzr
checks for an absolute path and otherwise resolves relative to CWD.

Windows is more complex but that's not relevant to you, right?

 > On Thu, Jan 07, 2010 at 02:56:02PM +0100, Óscar Fuentes wrote:
 > > Alan Mackenzie <address@hidden> writes:

 > > > Why does Bazaar need to "process" this data?  It's essentially doing
 > > > copying, with some accompanying administrivia.
 > 
 > > It is not copying, [...].
 > 
 > It IS copying, conceptually

Alan, don't be contrary.  Óscar is just telling you the facts, and
AIUI *he* would not have chosen bzr if it were up to him.  He's just
trying to help you and others make the best use of the official VCS.

 > - the content of the branched repository contains a portion of the
 > original branch, and nothing else.  So it has to do a bit of
 > filtering on the files' logs.

No, it has to do a lot more than that.  What it is doing is
conceptually most like garbage collection.  The actual content of a
repository is stored as a set of compressed archives of file objects,
revision objects, and patches, plus some indicies.  (These archives
are called "packs.")  The "processing" goes through those objects,
throws out the ones not needed for the branch you request, and
reconstructs minimal packs and indicies for them.

 > There have been, perhaps, ~100 files updated since I first downloaded the
 > repository last Friday.  For each changed file, bzr transferred ~2Mb
 > between my PC and savannah.  Why?  This is ludicrous.
 > 
 > Hopefully it was an exceptional case, but I'd not changed my .../trunk at
 > all since downloading it, so anything exceptional was at the savannah
 > end.

Exactly!  At this date, the main cause of that particular slowness is
not bzr, it's savannah-hackers, which chose the absolute worst
available way to implement bzr support.  (They had their reasons,
security-related, which given Savannah's history as a target seem
reasonably important to me.)

 > I'm about to fix a bug which will involve ~100 bytes change to a C file
 > and ~200 bytes log message and ChangeLog addition.  How much will the bzr
 > commit operation transfer?  Hopefully, several kilobytes, no more.

As much as it needs to, but until the issue with Savannah gets
addressed, the minimum you should hope for is one pack and the index
file, typically a few MB.

You'll know when it gets fixed because there will be an announcement
that you should use "bzr+ssh" URLs, not "http" or "sftp".

 > Any chance you could point out that other thread to me?

The issue that bzr can be damned slow if you don't tune your workflow
has been pretty well constantly active since late 2007.  I guess it
just faded into the rest of the background noise for you. :-)

 > Yes, bzr is too slow for me.  My first checkout took, perhaps, an hour
 > and a half, but I can cope with that.

By "checkout" you mean "bzr branch http://savannah.../path/to/emacs/trunk";,
right?  ("bzr checkout" does something completely different and should
not take more than a handful of seconds.  Please don't remark about the
randomness of bzr command names, this particular choice makes sense in
the appropriate context.)

The 90-minute initial branch is something we'll just have to deal
with (probably you personally won't actually need to do it again!)

BTW, "bzr branch http://bzr.savannah.gnu.org/r/emacs/trunk emacs" just
now took 31:30 on my MacBook Pro (2.4GHz Intel Core2 Duo, 2GB RAM, Mac
OS X 10.5.8, Python 2.6.4) with bzr 2.0.1.  As others report, this is
CPU-bound, using 97% of one core.

 > 'bzr branch' (to a random place) took 40 minutes.

Are you sure the "random place" wasn't your CD-RW?  Just joking, but
that seems awfully long to me ("bzr branch emacs emacs.1" took 584
seconds, under 10 minutes for me, same box as above).  The only recent
reports I've seen on the Bazaar list in that league involve Windows
file systems or NFS mounts, so I wonder if there's something unusual
about your setup.  Are you on NFS?  If not, may I ask you to confirm
that your disk performance doesn't normally noticably suck, etc?

 > 'bzr branch' to the Right Place took a few seconds, and
 > this is the only bzr thing I've done so far which has been
 > reasonable.

It's also one of the things you're likely to do fairly frequently, and
minimizing its cost removes an important barrier to adopting best
practice.  So that's a good sign, even if so far it's rare in your
experience.

 > 'bzr update' took 23 minutes,

This is probably a one-off.  See the "BZR error!" thread, where Stefan
explains.

And the underlying problem is probably the Savannah misconfiguration,
which will eventually get fixed.

In any case, once you get the branch(es) you need locally, most
operations can be done locally, and with a little bit of care are all
reasonably fast.  Even while the Savannah issue remains open, once
Stefan et al finish tuning the arrangement of stuff in the main repo
(probably already done, actually), if you update your local branch
mirrors at least a couple of times a week you can probably hope to
limit each download to 10 or 20MB.

Or you can use the Launchpad mirror, as others have suggested.

 > Their basic assumptions don't match the Emacs project, for whatever
 > reason.

bzr was chosen for political reasons (it was the only GNU project
among the contenders), not because it was the best fit to existing
Emacs project practice.  The Emacs project therefore must adapt, and
that's part of the deal for Emacs developers.

It really shouldn't be that hard, but managing transitions in the GNU
Project is hard because the admins are spread so thinly.  Expect a few
more bumps, but really, there's no reason to expect that problems will
continue to be this frequent for more than a couple more weeks.