Re: [Duplicity-talk] some usage questions regarding full/incremental backups

[Kenneth Loafman]

Mathijs Kwik wrote:
> Hi,
> 
> I currently use rdiff-backup to backup a few servers.
> They all backup to 1 central backup-machine.
> That machine thus has backup for all servers, including history.
> This is all unencrypted, but this isn't an issue since the machine is
> in the office and is secured by itself (disks are encrypted).
> After all machines have backed-up to this central location, it uses
> regular rsync to upload the archive to an off-site machine.
> (I don't use rdiff-backup for this, since I don't need history of history :)
> The off-site machine is in a fully secured, trusted environment.
> 
> Now,
> I would like to change the off-site machine to a regular cheap
> backup-hosting service (current location is _very_ expensive).
> So I need encryption since I can't trust the backup-hosting provider.

I would stay with the local server for quick recovery when needed and
not go directly to the backup-hosting service.

> So I found duplicity, it looks perfect, just the way I need it.
> 
> One thing that isn't clear to me is the full/incremental stuff.
> Currently, using rdiff-backup, I can remove all stuff that's older than a 
> month.
> I saw duplicity can do this too, but the man-page mentions it won't
> delete old stuff that has newer (incremental) stuff depending on it.
> Maybe rdiff-backup has the same issue, didn't look into that yet.
> So, if I only use incremental backups, the archive will fill-up more
> and more even when cleaning old stuff.

Yes, that is the way it works.  It is not a good practice to only do
incremental backups, especially with files that change a lot.

> So I need to use full backups every now and then.
> Will this mean everything gets transfered fully?

Only on a full backup.

> Or is there some intelligent algorithm that justs 'merges' the
> increments into a full backup?

No, but the idea has been put out before.

> Since our total backup is close to 1TB, I don't feel like transferring
> everything every month or so.
> Also, storage space might be an issue.
> Say I want to keep a month of backups,
> I would do a full backup every 2 weeks and clean up all but the last 2
> full's right after that.
> Does this mean that (right after doing the full backup, before
> cleaning up), there are 3 full copies?
> needing 3TB+ storage feels a bit much when the 'real' data is 'only' 1TB.
> Am I miscalculating here? Or is there some smart strategy I can try?
> All servers combined is about 1TB. daily changes are between 500Mb and 1Gb.

I usually do the cleanup of old backups prior to backing up so I don't
run into the situation you describe.  As long as you leave the latest
full backup, you should be in good shape.

> Like I said, maybe rdiff-backup has the same issue, but I can't
> remember a full/incremental option on it.
> As far as I remember rdiff-backup chose a reverse-incremental
> strategy, so the most current backup is always full, and every
> previous backup is decremented from that.

This is similar to the reverse-delta method that CVS uses.  It works and
has no need of full/incremental backups.  Purging can be done at the end
of the chain.

> So I really hope some smart merging strategy can be used when doing full.
> Otherwise a reverse/decremental strategy would be very helpful to save
> space / bandwidth.

There is no merging.  The space is saved because the incremental backups
only backup the changed portions of the files.  It's possible to do the
merging (its done during a restore anyway), but not yet.

If you google "rdiff-backup encryption" there are a fair number of
articles on how to do this.  Perhaps that is all you need?  I think
duplicity will do what you need, but maybe not what you want.

...Ken

signature.asc
Description: OpenPGP digital signature