[rdiff-backup-users] rdiff-backup performance -- slow operation on initial backup?

[Derek Atkins]

Hi,

I'm trying to use rdiff-backup to backup a bunch of servers.  One
particular server contains about 160GB of data, but when I try to perform
the rdiff-backup it's saving the data at a measly 1MB/s.

Here's my configuration:

  [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]

I ran a bunch of tests to try to figure out my bottlenecks.

I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on the
backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I get
50.2MB/s.  If I run dd directly on the backup server (through encfs) I get
20.1MB/s.  If I go over SSH from the backup server to the target server
and run the dd on the target server, then write to FreeNas through encfs
declines to 7.6MB/s.

Note that in those SSH tests, however, I forgot to turn off compression. 
When I do that, the throughput for the dd test reduced to 6.6BM/s.  (Note
that this is running simultaneously with a running rdiff-backup, so it's
possible that they are reducing performance).

Then I ran an scp test to the same target server; copying about 1.4GB of
photos.  Files ranged in size from 10KB to 5MB.  When run in standard mode
(displaying each file status) I got 4.4MB/s.  Running in quiet mode I get
5.1MB/s.

So clearly the bottleneck is in rdiff-backup -- performance (IMHO) should
not be significantly slower than the last dd-over-ssh test.  It appears
rdiff-backup is slowing me down by a factor of 5x throughput versus scp.

I found a message from Ben from 2005 where he suggests increasing the
blocksize and conn_bufsiz settings in Globals.py:
https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html

What he didn't say was whether this needed to be changed on the target
server, the backup server, or both.  Nor do I know if that would actually
help this situation.

Do you have any ideas?

Thanks,

-derek

PS: According to rpm, both systems are running version 1.2.8.

-- 
       Derek Atkins                 617-623-3745
       address@hidden             www.ihtfp.com
       Computer and Internet Security Consultant