[Duplicity-talk] duplicity with s3 and glacier, lifecycling might result in old manifests not being deleted

[Tobias Vollmer]

Hi,

tl;dr: Moving archives results in not deleted manifests of full backups.
Bug?


I started to play around with duplicity. I would like to create a setup
where I have one full backup (and incrementals) in S3, and a second in
glacier.

To find out the proper setup, I played around with local file storage
and moved "old" backups to a different folder.

I run this command every Minute
> duplicity --encrypt-key $KEYID  --volsize 100 --full-if-older-than 5m 
> --file-prefix-archive archive. $DATAPATH file://$BACKUPPATH

Additionally I run the following command after the backup to clear old
Backups
> duplicity  remove-all-but-n-full 1 --force --file-prefix-archive archive. 
> file://$BACKUPPATH
and move the archive.* files to a second location.

If I skip the moving part, everything works as expected, if I move the
archive files, old manifests are not deleted.

My setup is a Debian Box with the back ported 0.6.24 Version of duplicity.

I added the following line to the definition of the delete function in
/usr/share/pyshared/duplicity/collections.py, line 147:
> log.Notice("Deleting files:" + str(rfn))
to get some debugging information.

The first listing shows which files are send to the backend to delete,
if I do _not_ move the archives manually:

> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Tue Jan  6 18:09:01 2015
> Deleting backup chain at time:
> Tue Jan  6 18:08:01 2015
> Deleting complete signature chain Tue Jan  6 18:08:01 2015
> Deleting complete signature chain Tue Jan  6 18:08:01 2015
> Deleting complete backup chain Tue Jan  6 18:08:01 2015
> Deleting 
> files:['duplicity-inc.20150106T170701Z.to.20150106T170801Z.manifest.gpg', 
> 'archive.duplicity-inc.20150106T170701Z.to.20150106T170801Z.vol1.difftar.gpg']
> Deleting 
> files:['duplicity-inc.20150106T170602Z.to.20150106T170701Z.manifest.gpg', 
> 'archive.duplicity-inc.20150106T170602Z.to.20150106T170701Z.vol1.difftar.gpg']
> Deleting 
> files:['duplicity-inc.20150106T170501Z.to.20150106T170602Z.manifest.gpg', 
> 'archive.duplicity-inc.20150106T170501Z.to.20150106T170602Z.vol1.difftar.gpg']
> Deleting 
> files:['duplicity-inc.20150106T170401Z.to.20150106T170501Z.manifest.gpg', 
> 'archive.duplicity-inc.20150106T170401Z.to.20150106T170501Z.vol1.difftar.gpg']
> Deleting 
> files:['duplicity-inc.20150106T170301Z.to.20150106T170401Z.manifest.gpg', 
> 'archive.duplicity-inc.20150106T170301Z.to.20150106T170401Z.vol1.difftar.gpg']
> Deleting files:['duplicity-full.20150106T170301Z.manifest.gpg', 
> 'archive.duplicity-full.20150106T170301Z.vol2.difftar.gpg', 
> 'archive.duplicity-full.20150106T170301Z.vol1.difftar.gpg']

The second listing is from a run after the archives are moved manually:

> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Tue Jan  6 18:15:01 2015
> Deleting backup chain at time:
> Tue Jan  6 18:14:02 2015
> Deleting complete signature chain Tue Jan  6 18:14:02 2015
> Deleting complete signature chain Tue Jan  6 18:14:02 2015
> Deleting complete backup chain Tue Jan  6 18:14:02 2015
> Deleting 
> files:['duplicity-inc.20150106T171301Z.to.20150106T171402Z.manifest.gpg']
> Deleting 
> files:['duplicity-inc.20150106T171201Z.to.20150106T171301Z.manifest.gpg']
> Deleting 
> files:['duplicity-inc.20150106T171101Z.to.20150106T171201Z.manifest.gpg']
> Deleting 
> files:['duplicity-inc.20150106T171002Z.to.20150106T171101Z.manifest.gpg']
> Deleting 
> files:['duplicity-inc.20150106T170901Z.to.20150106T171002Z.manifest.gpg']

As you can see, the incrementals are completely removed, but the full
backup is missing.

After the removal, the directory listing shows the additional manifest:

> ls -a
> total 2848
> .
> ..
> backup.log # <- Expected, logfile :-)
> duplicity-full.20150106T170901Z.manifest.gpg
> duplicity-full.20150106T171501Z.manifest.gpg
> duplicity-full-signatures.20150106T171501Z.sigtar.gpg

This might be a problem, as the number of manifests will grow over time
on a long running backup system.

Could this be a bug? Or is it expected behaviour?


Cheers,

Tobias
-- 
Please watch your privacy! Encrypt your emails!
Achten Sie auf Ihre Privatsphäre! Verschlüsseln Sie Ihre Emails!
PGP-Key: 8AE12233 (http://key.tvollmer.de/)
75D8 DA44 69B5 9621 1262 A41D 1952 B22F 8AE1 2233