[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: how can I not have to spend 40 minutes _waiting_ in themorningfor ea
From: |
eric.berg |
Subject: |
RE: how can I not have to spend 40 minutes _waiting_ in themorningfor each of 40 updates? |
Date: |
Fri, 10 Oct 2008 10:13:26 -0400 |
Hi, Chaitanya,
If the primary goal of this exercize is to get a copy of
the same files onto each machine, then rync (http://rsync.samba.org) may well be the right
tool for the job. I've used rsync for many years now to address a number
of issues from syncing source trees on multiple machines to deploying static
content and even for backups to remote machines. It supports ssh and
compression. I consider it to be fundamental to my toolkit, right behind
ssh and perl.
I recommend that you do some research into how long it
takes to rsync on a few days with varying degrees of change in your code.
The first thing that rsync does it to assess the files on both sides in order to
determine what needs to be copied. This can be an expensive operation --
in fact, the most expensive part for minor changes. It may
While it can do partial file diffs, it may be a good ide to
set it to transfer entire files ( -W param). Compression is an option to
be explored as well. There are a number of options that might work to
optimize this.
Also, and I've never done this, but it may be helpful to
add a level to the distribution by first distributing to a set of machines that
then push to the others to spread out the load. Consider geography here to
minimize bandwidth.
And one more point that I'll make: I've run into some
barriers when attempting to rsync directory structures with many files ( >
60,000). Rsync croaked, so I wrote a short script to break it out by doing
an separate rsync for individual top-level directories. This may or may
not be an issue, and it may actually reduce the overall time it takes to do the
rysnc. YMMV.
Helpful rsync development params:
-n - dry run. no files are
changed.
-stats - as you would expect....
--progress
Eric
Hi Eric,
I have the code on 40 different machines from the
trunk or t-o-t or head or whatever you call it. So, no different branches
involved here. I will have a go at understanding rsync and using it for
syncing the changed files from 1 machine to the 39 others.
Btw, did you
ever try this or run into similar
problems?
Regards,
Chaitanya
On Fri, Oct 10, 2008 at 7:06 PM,
<address@hidden>
wrote:
Has
the suggestion to use rsync been raised here? I'm not sure
if
you're updating to different branches for the various platforms,
but
exporting to one or a few locations and then running rsync to
transfer
just the diffs without having to repeatedly go back to CVS might
be a
workable solution.
As for time servers, you'll want to sync
to several servers. NTP's
algorithm takes travel time into account,
though closer is
better.
Eric
_______________________________________________
This e-mail may contain
information that is confidential, privileged or otherwise protected from
disclosure. If you are not an intended recipient of this e-mail, do not
duplicate or redistribute it by any means. Please delete it and any attachments
and notify the sender that you have received it in error. Unless specifically
indicated, this e-mail is not an offer to buy or sell or a solicitation to buy
or sell any securities, investment products or other financial product or
service, an official confirmation of any transaction, or an official statement
of Barclays. Any views or opinions presented are solely those of the author and
do not necessarily represent those of Barclays. This e-mail is subject to terms
available at the following link: www.barcap.com/emaildisclaimer.
By messaging
with Barclays you consent to the foregoing. Barclays Capital is the investment
banking division of Barclays Bank PLC, a company registered in England
(number 1026167) with its registered office at 1 Churchill Place, London, E14
5HP.
This email may relate to or be sent from other members of the Barclays
Group.
_______________________________________________