Re: [Duplicity-talk] full and incrementals

[Paul Harris]

2009/4/23 Jacob <address@hidden>:
> On Thu, 23 Apr 2009 09:40:21 +0800
> Paul Harris <address@hidden> wrote:
>
>> The full backup set is broken up into 5mb files, so it would be nice if it
>> were able to figure out which of those full backup files it could delete -
>> eg the original files have been deleted from the home directory.   This
>> isn't currently possible... right?
>
> This is nearly impossible. An incremental or full backup is basically an 
> encrypted tarball that has been split into "volumes". This means you could 
> have multiple files in each volume. Thus, in order to get rid of a single 
> volume, all the files in that tarball would have to be deleted on your home 
> system too. Even then, the sigtar and manifest files each have their own 
> records stating that those files once existed. If you wanted to remove all 
> traces, those remote, compressed, encrypted files would have to be edited.
>

If i could wave a magic wand and make the code changes (and skip past
the technical devils in details such as encryption algorithm
requirements), it would work like so...

Each volume could be decrypted independently.   The volumes that
contain ONLY parts of files that are not required anymore are deleted.
  The sigtar and manifest files would remain stating that the files
existed, I don't mind that they remain as they are relatively small.

When it comes to restoring a file that is still available, the
appropriate volumes are opened and the file is pieced together from
the ranges saved in each volume.

When it comes to restoring a file that is no longer available,
duplicity would know those files are gone for good because the volumes
don't exist in the folder.

> Removing deleted files from the backup seems counter-productive. In most, if 
> not all, cases, I'd imagine you'd want to save those files in case you figure 
> out later you really needed that deleted file.
>

Yes and no.  Yes, you want to save the files for a period of time.
After a long time, that file is deleted and you really don't need it
anymore.  The backup disks get full of incrementals because its got
loads of files I just don't need.

Good example, I take RAW photos with an SLR camera.  I back them up.
When I get the time, I process them and generate colour corrected
JPEGs.   I then throw away the RAWs because they are really too big to
keep a lot of them around.
I also move those images around the harddisk (hence the request to
support file moves and renames).  There may be multiple copies, and
some different attempts.

If I kept everything, I'd need a lot more backup space.

One possible solution is to remove the volumes from the backup set (as
described above in my magical pixie land where software can be written
in one night like in Swordfish using "unix" and flying graphical
blocks).

The only other alternative is to do a fresh full backup and then
delete the old backup set.

A fresh, full backup is fine if the backup set is small and the backup
media is nearby.  But if the backup set is large distant and media is
remote, then its a pain in the arse and a whole night or two of
uploading.


Which brings me to the final bit of my email.  Applause, because I
can't believe how fast duplicity scans and backs up my files... I love
it.

cheers,
Paul