Re: [Duplicity-talk] some usage questions regarding full/incremental backups

[Damon Timm]

Hi - I am not a Duplicity developer, but I do use it, and can share my
experience ... I do a full backup every 14 days and remove everything
older than 31 days (which should keep two full backups at a time
before things start being removed ... this means I have at least
double my source file size at any time).

I use this script to do off-site backups of my home NAS:

http://blog.damontimm.com/bash-script-incremental-encrypted-backups-duplicity-amazon-s3/

Answered your other questions, below:

On Fri, Jan 2, 2009 at 11:02 PM, Mathijs Kwik <address@hidden> wrote:
> One thing that isn't clear to me is the full/incremental stuff.
> Currently, using rdiff-backup, I can remove all stuff that's older than a 
> month.
> I saw duplicity can do this too, but the man-page mentions it won't
> delete old stuff that has newer (incremental) stuff depending on it.
> Maybe rdiff-backup has the same issue, didn't look into that yet.
> So, if I only use incremental backups, the archive will fill-up more
> and more even when cleaning old stuff.
>
> So I need to use full backups every now and then.
> Will this mean everything gets transfered fully?

Yes.  In my case, every 14 days, the entirety of my backup is uploaded
again.  I have less than 1GB of data, though.

> Or is there some intelligent algorithm that justs 'merges' the
> increments into a full backup?

No - I think the issue is that everything is encrypted ... so,
duplicity has to start with a "base" to build its incremental backups
on, otherwise it wouldn't be able to utilize rsync-like goodness when
doing the incrementals ... there isn't any easy way, I would imagine,
to unencrypt the entire backup offsite, then remove the parts no
longer needed, then re-encrypt everything.

At least, that is how I understand it.  It is a trade-off between
rsync and encryption ...

> Since our total backup is close to 1TB, I don't feel like transferring
> everything every month or so.
> Also, storage space might be an issue.
> Say I want to keep a month of backups,
> I would do a full backup every 2 weeks and clean up all but the last 2
> full's right after that.
> Does this mean that (right after doing the full backup, before
> cleaning up), there are 3 full copies?

Yea, I think so.  In my case, I have two fulls, plus some incrementals
before things are cleaned up.

> needing 3TB+ storage feels a bit much when the 'real' data is 'only' 1TB.
> Am I miscalculating here? Or is there some smart strategy I can try?
> All servers combined is about 1TB. daily changes are between 500Mb and 1Gb.

Others may have a strategy, but I think it is the nature of the beast.
 You could space out your "fulls" more and only save one of them ...
then you wouldn't have to keep duplicate snapshots ... but you would
still have to use the bandwidth.

> Like I said, maybe rdiff-backup has the same issue, but I can't
> remember a full/incremental option on it.
> As far as I remember rdiff-backup chose a reverse-incremental
> strategy, so the most current backup is always full, and every
> previous backup is decremented from that.

Duplicity, as I understand it, is the opposite of that (due to
encryption).  First is a full, everything else is incremental.  I
think it makes sense considering ...

> So I really hope some smart merging strategy can be used when doing full.
> Otherwise a reverse/decremental strategy would be very helpful to save
> space / bandwidth.

Others may have insight on this, but I think the smartness would have
to reside off-site ... so it could examine what was changed and merge
it to the encrypted ... I think.

Good luck!

Damon

>
> Any help is greatly appreciated.
>
> Thanks,
> Mathijs
>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>