[Duplicity-talk] Losing incremental backups

[Greg Hartman]

I have been using duplicity for about six months.
Recently I have encountered a problem where my backups
suddenly fail. I am able to reproduce this bug on the
0.4.3 release. I have pasted the error messages
returned by duplicity below.

I did some investigation, and it looks to me like the
problem is related to an assumption that listdir()
will return pathnames in the same order that the files
were originally written to disk. This seems to be the
assumption in set_values in collections.py, but I'm
not very familiar with the Ruby and the code, so I may
be wrong.

Some UNIX filesystems, including ext3 with the
dir_index option set, use specialized data structures
to speed up path name resolution. This often means
that the order of existing files can change when files
are added to directories. I'm fairly convinced that
this is what happened to my backups.

>From what I can tell the sigtar files may come before
the difftar files in the list. The error messages seem
to indicate that duplicity ignores the sigtars when
this happens and then complains that the difftar files
are orphaned.

This seems to cause problems with the backups, which
isn't surprising since duplicity probably believes
that the chain of incremental backups is broken.

Is my analysis of the failure correct? If so, what is
the right solution? I'm assuming that this problem
could affect all of the backends, so perhaps some
sorting or a more robust matching algorithm in
set_values in collections.py would be in order.



      
____________________________________________________________________________________
Be a better pen pal. 
Text or chat with friends inside Yahoo! Mail. See how.  
http://overview.mail.yahoo.com/

duplicity -v 9 --no-encryption --collection-status 
file:///archives/systems/gs2064/duplicity
Main action: collection-status
Ignoring file 
'duplicity-new-signatures.2007-10-28T01:00:15-04:00.to.2007-10-29T01:00:10-04:00.sigtar.gz'
Ignoring file 'duplicity-full-signatures.2007-10-08T01:00:13-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-24T01:00:17-04:00.to.2007-10-25T01:00:12-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-12T01:00:14-04:00.to.2007-10-13T01:00:14-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-11T01:00:13-04:00.to.2007-10-12T01:00:14-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-15T01:00:14-04:00.to.2007-10-16T01:00:14-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-27T01:00:15-04:00.to.2007-10-28T01:00:15-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-11-01T01:00:15-04:00.to.2007-11-02T01:00:14-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-14T01:00:15-04:00.to.2007-10-15T01:00:14-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-30T01:00:12-04:00.to.2007-10-31T01:00:10-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-29T01:00:10-04:00.to.2007-10-30T01:00:12-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-09T01:00:12-04:00.to.2007-10-10T01:00:10-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-21T01:00:14-04:00.to.2007-10-22T01:00:16-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-11-03T01:00:14-04:00.to.2007-11-04T01:00:16-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-17T01:00:12-04:00.to.2007-10-18T01:00:15-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-08T01:00:13-04:00.to.2007-10-09T01:00:12-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-22T01:00:16-04:00.to.2007-10-23T01:00:15-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-31T01:00:10-04:00.to.2007-11-01T01:00:15-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-13T01:00:14-04:00.to.2007-10-14T01:00:15-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-18T01:00:15-04:00.to.2007-10-19T01:00:14-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-23T01:00:15-04:00.to.2007-10-24T01:00:17-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-19T01:00:14-04:00.to.2007-10-20T01:00:10-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-11-02T01:00:14-04:00.to.2007-11-03T01:00:14-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-25T01:00:12-04:00.to.2007-10-26T01:00:09-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-16T01:00:14-04:00.to.2007-10-17T01:00:12-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-10T01:00:10-04:00.to.2007-10-11T01:00:13-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-26T01:00:09-04:00.to.2007-10-27T01:00:15-04:00.sigtar.gz'
Ignoring file 
'duplicity-new-signatures.2007-10-20T01:00:10-04:00.to.2007-10-21T01:00:14-04:00.sigtar.gz'
Added incremental Backupset (start_time: Mon Oct  8 01:00:13 2007 / end_time: 
Tue Oct  9 01:00:12 2007)
Added incremental Backupset (start_time: Tue Oct  9 01:00:12 2007 / end_time: 
Wed Oct 10 01:00:10 2007)
Added incremental Backupset (start_time: Wed Oct 10 01:00:10 2007 / end_time: 
Thu Oct 11 01:00:13 2007)
Added incremental Backupset (start_time: Thu Oct 11 01:00:13 2007 / end_time: 
Fri Oct 12 01:00:14 2007)
Added incremental Backupset (start_time: Fri Oct 12 01:00:14 2007 / end_time: 
Sat Oct 13 01:00:14 2007)
Added incremental Backupset (start_time: Sat Oct 13 01:00:14 2007 / end_time: 
Sun Oct 14 01:00:15 2007)
Added incremental Backupset (start_time: Sun Oct 14 01:00:15 2007 / end_time: 
Mon Oct 15 01:00:14 2007)
Added incremental Backupset (start_time: Mon Oct 15 01:00:14 2007 / end_time: 
Tue Oct 16 01:00:14 2007)
Added incremental Backupset (start_time: Tue Oct 16 01:00:14 2007 / end_time: 
Wed Oct 17 01:00:12 2007)
Added incremental Backupset (start_time: Wed Oct 17 01:00:12 2007 / end_time: 
Thu Oct 18 01:00:15 2007)
Added incremental Backupset (start_time: Thu Oct 18 01:00:15 2007 / end_time: 
Fri Oct 19 01:00:14 2007)
Added incremental Backupset (start_time: Fri Oct 19 01:00:14 2007 / end_time: 
Sat Oct 20 01:00:10 2007)
Added incremental Backupset (start_time: Sat Oct 20 01:00:10 2007 / end_time: 
Sun Oct 21 01:00:14 2007)
Added incremental Backupset (start_time: Sun Oct 21 01:00:14 2007 / end_time: 
Mon Oct 22 01:00:16 2007)
Added incremental Backupset (start_time: Mon Oct 22 01:00:16 2007 / end_time: 
Tue Oct 23 01:00:15 2007)
Added incremental Backupset (start_time: Tue Oct 23 01:00:15 2007 / end_time: 
Wed Oct 24 01:00:17 2007)
Added incremental Backupset (start_time: Wed Oct 24 01:00:17 2007 / end_time: 
Thu Oct 25 01:00:12 2007)
Added incremental Backupset (start_time: Thu Oct 25 01:00:12 2007 / end_time: 
Fri Oct 26 01:00:09 2007)
Added incremental Backupset (start_time: Fri Oct 26 01:00:09 2007 / end_time: 
Sat Oct 27 01:00:15 2007)
Added incremental Backupset (start_time: Sat Oct 27 01:00:15 2007 / end_time: 
Sun Oct 28 01:00:15 2007)
Added incremental Backupset (start_time: Sun Oct 28 01:00:15 2007 / end_time: 
Mon Oct 29 01:00:10 2007)
Added incremental Backupset (start_time: Mon Oct 29 01:00:10 2007 / end_time: 
Tue Oct 30 01:00:12 2007)
Added incremental Backupset (start_time: Tue Oct 30 01:00:12 2007 / end_time: 
Wed Oct 31 01:00:10 2007)
Added incremental Backupset (start_time: Wed Oct 31 01:00:10 2007 / end_time: 
Thu Nov  1 01:00:15 2007)
Added incremental Backupset (start_time: Thu Nov  1 01:00:15 2007 / end_time: 
Fri Nov  2 01:00:14 2007)
Added incremental Backupset (start_time: Fri Nov  2 01:00:14 2007 / end_time: 
Sat Nov  3 01:00:14 2007)
Added incremental Backupset (start_time: Sat Nov  3 01:00:14 2007 / end_time: 
Sun Nov  4 01:00:16 2007)
Ignoring incremental Backupset (start_time: Sun Nov  4 01:00:16 2007; needed: 
Sun Nov  4 01:00:16 2007)
Warning, found signatures but no corresponding backup files
Warning, found the following orphaned backup files:
[duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.manifest, 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol1.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol2.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol3.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol4.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol5.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol6.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol7.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol8.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol9.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol10.difftar.gz,
 duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-!
 11-15T16:55:07-04:00.vol11.difftar.gz, 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol12.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol13.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol14.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol15.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol16.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol17.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol18.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol19.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol20.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol21.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol22.diff!
 tar.gz, duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07
-04:00.vol23.difftar.gz, 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol24.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol25.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol26.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol27.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol28.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol29.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol30.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol31.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol32.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol33.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol34.difftar.gz,
 duplici!
 
ty-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol35.difftar.gz, 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol36.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol37.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol38.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol39.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol40.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol41.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol42.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol43.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol44.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol45.difftar.gz,
 duplicity-inc.2007-11-04T01:00:16-04:0!
 0.to.2007-11-15T16:55:07-04:00.vol46.difftar.gz, duplicity-inc.2007-11
-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol47.difftar.gz, 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol48.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol49.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol50.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol51.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol52.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol53.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol54.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol55.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol56.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol57.difftar.gz,
 duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15!
 T16:55:07-04:00.vol58.difftar.gz, 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol59.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol60.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol61.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol62.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol63.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol64.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol65.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol66.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol67.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol68.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol69.difftar.g!
 z, duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:0
0.vol70.difftar.gz, 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol71.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol72.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol73.difftar.gz,
 
duplicity-inc.2007-11-04T01:00:16-04:00.to.2007-11-15T16:55:07-04:00.vol74.difftar.gz]