Re: [rdiff-backup-users] rdiff-backup performance -- slow operation on initial backup?

[Derek Atkins]

On Wed, March 30, 2016 12:47 pm, Greg Troxel wrote:
>
> "Derek Atkins" <address@hidden> writes:
>
>>> I would up the send/receive socket buffers (because it's easy, not
>>> because I think that's the problem), and watch disk/cpu on both sides,
>>> and also run netstat to see if data is piling up in the transmit socket
>>> buffer.
>>
>> Do you mean within rdiff-backup, or at some other level?
>
> On NetBSD I mean bumping up these sysctls
>
> net.inet.tcp.sendspace = 131072
> net.inet.tcp.recvspace = 131072
> net.inet6.tcp6.sendspace = 131072
> net.inet6.tcp6.recvspace = 131072
>
> and presumably that's similar on other systems.

I'll look for the Linux equivalent, however...

> But, if your socket buffers aren't full, that's probably not your
> problem.

... according to repeated netstat the socket buffers are GENERALLY 0,0.  I
did see it get up to about 1000, at one point...  OH, I take that back,
the SEND Queue value was just up to 243816 on the target system....

>>> FWIW, I used to use rdiff-backup but found it to be nonrobust on
>>> machines with limited (only a few GB) RAM and hundreds of GB of backup.
>>> I have switched to bup.
>>
>> Unfortunately bup is not available on all my target platforms.
>
> bup is python with a little C, and thus seems pretty portable.   Where
> isn't it working?

Didn't say it wasn't working.  I said "it is not available", meaning there
is no prepackaged RPM for some of my systems.  I could certainly try to
piece it together, but of course I prefer to use distro-supplied software
wherever possible.  It makes upgrading much easier.

> There is also attic and borg which are similar to bup.
>
>> Maybe I should consider amanda or bacula?
>
> amanda is basically wrappers around dump and tar.  If you have 50
> machines and want to do level 0/1/2 to tape and take tapes offsite, it
> works great, after you pay for the LTO tape drive.

I don't have 50 machines.  I do have ~10-15.  Pretty much all Linux boxes.
 Not using tape; backups are all on spinning media.

> Two things to think about:
>
>   do you care about deduplication?  bup does not only per-file but
>   within-file deduplication, so if multiple boxes have the same data it
>   doesn't take up extra space

No.  At least, not at the block level.  I would care about file-level
dedup, but honestly I only care about that from within one server.  If I
have multiple systems that happen to have the exact same config file I
don't particularly care about dedupping that.  I just don't want a daily
copy of /etc/passwd :)

>   Do you really need to back up all platforms, or could you sync from
>   some (Android?) to a machine with more disk and back that up?  I have
>   been using syncthing, which seems to be pretty solid, for syncing
>   among Android and regular computers (BSD and OS X).  (It is written in
>   go so it's not in practice that portable.)

Pretty much all the systems I want to backup are Linux, but different
vintages and such.

Android is different and I back that up differently.  I'm just trying to
maximize performance.  I've got a GigE network and relatively plenty of
encryption performance; I'd like to leverage that in my backup (and
restore) operations.

Thanks,

-derek

-- 
       Derek Atkins                 617-623-3745
       address@hidden             www.ihtfp.com
       Computer and Internet Security Consultant