Re: [rdiff-backup-users] rdiff-backup performance -- slow operation on initial backup?

[Derek Atkins]

Hi,

Thank you for taking the time to look at this..

On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote:
> Is this really your first rdiff-backup to this location? If you have any
> previous rdiff-backup runs to this repository then the situation is
> complicated by rdiff-backup's need to create a new set of reverse diff
> files to be able to regress to previous file contents.

Yes, this s really the first rdiff-backup to this location.

A second backup run shortly after the first one completed finished in 55
minutes.

> What is your /tmp location? rdiff-backup uses this location for some
> operations though not AFAIK for standard backup runs. Still, if /tmp is on
> encfs maybe it could be a culprit; you can override rdiff-backup's
> temporary file location with --tempdir and --remote-tempdir.

If it is truly /tmp then no; /tmp is a ramdisk on the backup server and is
on the root disk on the target server.  Neither are being run through
encfs.

If, however, you mean the rdiff-backup-data.tmp files, those ARE being run
through encfs.

> Might also be worth trying --ssh-no-compression.

I already have "Compression no" set in ~/.ssh/config so I'm not sure what
this would add?

> Dominic
> http://www.timedicer.co.uk

-derek

PS: I'm running a raw rsync command now just to see how it behaves -- so
far I'm only seeing about 2MB/s, but it's only been running for 10 minutes
or so.

>
> On 28 March 2016 at 14:37, Derek Atkins <address@hidden> wrote:
>
>> Just a quick update:  I tried making these changes on both sides and it
>> really didn't make a difference.  Full backup of 202852072 Kbytes
>> required 2267m25.913s (previously it took 2346m57.800s, so it only sped
>> up a factor of 3%.
>>
>> Only thing I have not yet tried is running a raw rsync to see how fast
>> that runs.  I'll do that next.
>>
>> So, back to my orignal question: anyone have any idea how to get initial
>> transfers to run faster (or indeed any significant data transfers)?
>>
>> Thanks,
>>
>> -derek
>>
>> "Derek Atkins" <address@hidden> writes:
>>
>> > Hi,
>> >
>> > I'm trying to use rdiff-backup to backup a bunch of servers.  One
>> > particular server contains about 160GB of data, but when I try to
>> perform
>> > the rdiff-backup it's saving the data at a measly 1MB/s.
>> >
>> > Here's my configuration:
>> >
>> >   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>> >
>> > I ran a bunch of tests to try to figure out my bottlenecks.
>> >
>> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on
>> the
>> > backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I
>> get
>> > 50.2MB/s.  If I run dd directly on the backup server (through encfs) I
>> get
>> > 20.1MB/s.  If I go over SSH from the backup server to the target
>> server
>> > and run the dd on the target server, then write to FreeNas through
>> encfs
>> > declines to 7.6MB/s.
>> >
>> > Note that in those SSH tests, however, I forgot to turn off
>> compression.
>> > When I do that, the throughput for the dd test reduced to 6.6BM/s.
>> (Note
>> > that this is running simultaneously with a running rdiff-backup, so
>> it's
>> > possible that they are reducing performance).
>> >
>> > Then I ran an scp test to the same target server; copying about 1.4GB
>> of
>> > photos.  Files ranged in size from 10KB to 5MB.  When run in standard
>> mode
>> > (displaying each file status) I got 4.4MB/s.  Running in quiet mode I
>> get
>> > 5.1MB/s.
>> >
>> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO)
>> should
>> > not be significantly slower than the last dd-over-ssh test.  It
>> appears
>> > rdiff-backup is slowing me down by a factor of 5x throughput versus
>> scp.
>> >
>> > I found a message from Ben from 2005 where he suggests increasing the
>> > blocksize and conn_bufsiz settings in Globals.py:
>> >
>> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>> >
>> > What he didn't say was whether this needed to be changed on the target
>> > server, the backup server, or both.  Nor do I know if that would
>> actually
>> > help this situation.
>> >
>> > Do you have any ideas?
>> >
>> > Thanks,
>> >
>> > -derek
>> >
>> > PS: According to rpm, both systems are running version 1.2.8.
>>
>> --
>>        Derek Atkins                 617-623-3745
>>        address@hidden             www.ihtfp.com
>>        Computer and Internet Security Consultant
>>
>> _______________________________________________
>> rdiff-backup-users mailing list at address@hidden
>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> Wiki URL:
>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>>
>


-- 
       Derek Atkins                 617-623-3745
       address@hidden             www.ihtfp.com
       Computer and Internet Security Consultant