Re: [rdiff-backup-users] rdiff-backup performance -- slow operation on initial backup?

[Dominic Raferd]

It doesn't sound like rdiff-backup is the culprit here. You could try hpn-ssh https://sourceforge.net/projects/hpnssh/ ?

On 29 March 2016 at 21:03, Derek Atkins <address@hidden> wrote:

Hi,

On Mon, March 28, 2016 10:56 am, Derek Atkins wrote:
> Hi,
>
> Thank you for taking the time to look at this..
>
> On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote:
>> Is this really your first rdiff-backup to this location? If you have any
>> previous rdiff-backup runs to this repository then the situation is
>> complicated by rdiff-backup's need to create a new set of reverse diff
>> files to be able to regress to previous file contents.
>
> Yes, this s really the first rdiff-backup to this location.
>
> A second backup run shortly after the first one completed finished in 55
> minutes.
>
>> What is your /tmp location? rdiff-backup uses this location for some
>> operations though not AFAIK for standard backup runs. Still, if /tmp is
>> on
>> encfs maybe it could be a culprit; you can override rdiff-backup's
>> temporary file location with --tempdir and --remote-tempdir.
>
> If it is truly /tmp then no; /tmp is a ramdisk on the backup server and is
> on the root disk on the target server.  Neither are being run through
> encfs.
>
> If, however, you mean the rdiff-backup-data.tmp files, those ARE being run
> through encfs.
>
>> Might also be worth trying --ssh-no-compression.
>
> I already have "Compression no" set in ~/.ssh/config so I'm not sure what
> this would add?
>
>> Dominic
>> http://www.timedicer.co.uk
>
> -derek
>
> PS: I'm running a raw rsync command now just to see how it behaves -- so
> far I'm only seeing about 2MB/s, but it's only been running for 10 minutes
> or so.

My rsync backup just finished.  It copied 202688912 KB in 1599m55.255s  so
about 2.1MB/s.  Still significantly slower than SCP, but faster than
rdiff-backup.

The command I ran was:

rsync -art -e "ssh -l root -i /dev/null -o Compression=no"
address@hidden:/var/www/ /backups/server

:-(

-derek

>
>>
>> On 28 March 2016 at 14:37, Derek Atkins <address@hidden> wrote:
>>
>>> Just a quick update:  I tried making these changes on both sides and it
>>> really didn't make a difference.  Full backup of 202852072 Kbytes
>>> required 2267m25.913s (previously it took 2346m57.800s, so it only sped
>>> up a factor of 3%.
>>>
>>> Only thing I have not yet tried is running a raw rsync to see how fast
>>> that runs.  I'll do that next.
>>>
>>> So, back to my orignal question: anyone have any idea how to get
>>> initial
>>> transfers to run faster (or indeed any significant data transfers)?
>>>
>>> Thanks,
>>>
>>> -derek
>>>
>>> "Derek Atkins" <address@hidden> writes:
>>>
>>> > Hi,
>>> >
>>> > I'm trying to use rdiff-backup to backup a bunch of servers.  One
>>> > particular server contains about 160GB of data, but when I try to
>>> perform
>>> > the rdiff-backup it's saving the data at a measly 1MB/s.
>>> >
>>> > Here's my configuration:
>>> >
>>> >   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>>> >
>>> > I ran a bunch of tests to try to figure out my bottlenecks.
>>> >
>>> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on
>>> the
>>> > backup server.  Going directly to FreeNAS via NFS (bybassing encfs) I
>>> get
>>> > 50.2MB/s.  If I run dd directly on the backup server (through encfs)
>>> I
>>> get
>>> > 20.1MB/s.  If I go over SSH from the backup server to the target
>>> server
>>> > and run the dd on the target server, then write to FreeNas through
>>> encfs
>>> > declines to 7.6MB/s.
>>> >
>>> > Note that in those SSH tests, however, I forgot to turn off
>>> compression.
>>> > When I do that, the throughput for the dd test reduced to 6.6BM/s.
>>> (Note
>>> > that this is running simultaneously with a running rdiff-backup, so
>>> it's
>>> > possible that they are reducing performance).
>>> >
>>> > Then I ran an scp test to the same target server; copying about 1.4GB
>>> of
>>> > photos.  Files ranged in size from 10KB to 5MB.  When run in standard
>>> mode
>>> > (displaying each file status) I got 4.4MB/s.  Running in quiet mode I
>>> get
>>> > 5.1MB/s.
>>> >
>>> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO)
>>> should
>>> > not be significantly slower than the last dd-over-ssh test.  It
>>> appears
>>> > rdiff-backup is slowing me down by a factor of 5x throughput versus
>>> scp.
>>> >
>>> > I found a message from Ben from 2005 where he suggests increasing the
>>> > blocksize and conn_bufsiz settings in Globals.py:
>>> >
>>> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>>> >
>>> > What he didn't say was whether this needed to be changed on the
>>> target
>>> > server, the backup server, or both.  Nor do I know if that would
>>> actually
>>> > help this situation.
>>> >
>>> > Do you have any ideas?
>>> >
>>> > Thanks,
>>> >
>>> > -derek
>>> >
>>> > PS: According to rpm, both systems are running version 1.2.8.
>>>
>>> --
>>>        Derek Atkins                 617-623-3745
>>>        address@hidden             www.ihtfp.com
>>>        Computer and Internet Security Consultant
>>>
>>> _______________________________________________
>>> rdiff-backup-users mailing list at address@hidden
>>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>>> Wiki URL:
>>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>>>
>>
>
>
> --
>        Derek Atkins                 617-623-3745
>        address@hidden             www.ihtfp.com
>        Computer and Internet Security Consultant
>
>
> _______________________________________________
> rdiff-backup-users mailing list at address@hidden
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>


--
       Derek Atkins                 617-623-3745
       address@hidden             www.ihtfp.com
       Computer and Internet Security Consultant