Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.

[Piotr Karbowski]

On Tue, Sep 22, 2009 at 4:36 PM, Dominic Raferd <address@hidden> wrote:
> Piotr Karbowski wrote:
>>
>> On Tue, Sep 22, 2009 at 10:51 AM, Dominic Raferd <address@hidden>
>> wrote:
>>
>>>
>>> Piotr Karbowski wrote:
>>>
>>>>
>>>> On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd <address@hidden>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> Piotr Karbowski wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller <address@hidden>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> local rdiff-backup dir with remote server but how? If I will use for
>>>>>>>> example rsync it still need to check whole files for changes (read,
>>>>>>>> download it) and upload only new. I hope you will understand what I
>>>>>>>> need and help me.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> rsync won't check whole files unless you give the -c flag. Otherwise,
>>>>>>> it
>>>>>>> just compares metadata. I don't know if that's also the case with
>>>>>>> rdiff-backup, but I assume so.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> So I need to know how rdiff default compares data, if by size and
>>>>>> mod-time, it will not be so painful but still itefficient  will
>>>>>> download
>>>>>> changed
>>>>>> files to generate diff.
>>>>>>
>>>>>>
>>>>>
>>>>> Rdiff-backup is designed to be ultra-efficient at this activity. It
>>>>> only
>>>>> sends the changes in a file over the wire, not the whole file. To do
>>>>> this
>>>>> it
>>>>> uses the librsync library which is effectively the same as rsync. You
>>>>> can
>>>>> read more about the technique at http://en.wikipedia.org/wiki/Rsync.
>>>>> rdiff-backup does not use file times to determine whether to do
>>>>> backups.
>>>>> It
>>>>> can backup very large files with small changes very quickly.
>>>>>
>>>>> Dominic
>>>>>
>>>>>
>>>>>
>>>>
>>>> You dont understand me, rdiff-backup is efficient, but to make diff it
>>>> must read WHOLE file, on remote nfs or sshfs it is SLOOOOW and
>>>> painful
>>>>
>>>
>>> Sorry I get it now. But I think rdiff-backup and rsync require a separate
>>> computer at the remote end in order to optimise transfers, so if you are
>>> just accessing a remote share using sshfs or similar then they can still
>>> work of course but as you realise they will be slow. I guess it is not
>>> possible for you to run rdiff-backup (or rsync) at the remote end as
>>> well?
>>>
>>> You could run rdiff-backup locally to create a backup store and then
>>> mirror
>>> this store to the remote share using rcp. Still it will be slow because
>>> rdiff-backup always stores the latest copy of each file in full and so if
>>> this changes even slightly then the whole file will must be transferred
>>> by
>>> rcp.
>>>
>>> Duplicity http://duplicity.nongnu.org/ might work better for you, because
>>> it
>>> uses forward diffs. Also its archives are secure.
>>>
>>> Although not directly relevant I found a page here
>>> http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to
>>> greatly speed up OpenSSH in some situations.
>>>
>>
>> Duplicity is interesing project. What you think about using
>> rdiff-backup to create local backup, for example in /backups and then
>> send this /backups to remote server by duplicity? As far as I know
>> duplicity is encrypted so I DONT need using encfs, dmcrypt or other -
>> only ssh access is needed (realy I dont need duplicity on remote
>> server?).
>>
>> I just want be able to send _ENCRYPTED_ backups to remote server where
>> I have only ssh access (sftp/scp work).
>
> I have not used duplicity myself, I use rdiff-backup. But I am not sure you
> need to run rdiff-backup first, I think duplicity may make its own local
> copies of backup increments so that it can send future increments without
> having to access the earlier increments from the remote share. And yes
> duplicity sends encrypted files so you don't need other encryption.
>
> Because duplicity uses forward diffs you have to keep all backups forever,
> and if there is any corruption of a file you lose all backups that occurred
> *after* the date of this file. Rdiff-backup uses reverse diffs so
> corruption, if it occurs, affects backups *before* the date of the corrupted
> backup. With rdiff-backup you can delete backups before a certain date
> (though in my experience the storage is so efficient it is not usually worth
> bothering), which is not possible with duplicity.
>
> Still it sounds like duplicity would better suit your needs.
>

So best will be using duplicity to make local backup and run rsync to
send new files (diffs) to remote server. and if backup will be TOO big
just mv backup backup_old and start new backup (every 4 weeks for
example)