[Duplicity-talk] Fwd: duplicity including files in backup that have not changed?

[Chris Newton]

Over the past several months I have slowly switched over about 600 machines from rdiff-backup to duplicity with an S3 back end at Soft Layer.  Overall it has been great.  

For some reason on four of my machines(all el5) incremental backups with duplicity sync about 1.2 million of 1.6 million files that I do not believe have changed.  When I switch back to rdiff-backup the files do not get sync’d.  

I can’t run duplicity currently as it uses about 8G of RAM and kills the machine, so I can’t do a verbose run to see exactly which files are changing.  Would anyone have an idea as to a situation where duplicity would treat files as changed when rdiff-backup would not?

Another side effect of this is the memory footprint incremental backup duplicity process gets gigantic.  I would expect it to store all files in the latest backup chain since the latest full, but it appears to read in all volumes from all chains?  Here is a graph of the size of the duplicity memory footprint growing.  Note full backups happened on the 1st/2nd and 15th/16th:

https://docs.google.com/document/d/1Rc0R0sotsUu8WLzt3e6mj2hPzaIJ75cOPcpd4I7FNQI/edit?usp=sharing

Thanks in advance.

Version info:

Version 0.7.11 released December 31, 2016

Python 2.7.9

The file list that both rdiff and duplicity use is the exact same file.

Rdiff backup command:

/usr/bin/rdiff-backup -b --print-statistics --tempdir /backups/tmp  --remote-schema "/bin/nice -n 19 ssh -q -o StrictHostKeyChecking=no -C -o ServerAliveInterval=15 -o ServerAliveCountMax=30 -i ${IDENTITY_FILE} %s \"rdiff-backup --server\" || echo error: rdiff-backup failure" --include-globbing-filelist ${GLOBBING_FILELIST} --exclude '*' / address@hidden::${REMOTE_BACKUPS_DIR}/${SERVERNAME}/

Duplicity backup command:

/opt/duplicity/bin/duplicity --include-filelist /etc/duplicity/conf/files-nightly.conf / s3://s3-api.xxx-us-geo.objectstorage.service.sl.com/machine__name-nightly-s3

Summary of some of the backup log:

Starting with rdiff:

Apr  7 00:20:14 zxy-app-01-p rdiff-backup: SourceFiles 1543774

Apr  7 00:20:14 zxy-app-01-p rdiff-backup: ChangedFiles 839

Apr  8 00:05:44 zxy-app-01-p rdiff-backup: SourceFiles 1548166

Apr  8 00:05:44 zxy-app-01-p rdiff-backup: ChangedFiles 11186

To duplicity:

Apr 11 06:11:16 zxy-app-01-p duplicity_backup: SourceFiles 1545039

Apr 11 06:11:16 zxy-app-01-p duplicity_backup: ChangedFiles 0

Apr 11 23:02:25 zxy-app-01-p duplicity_backup: SourceFiles 1545051

Apr 11 23:02:25 zxy-app-01-p duplicity_backup: ChangedFiles 1181203

Apr 13 01:34:44 zxy-app-01-p duplicity_backup: SourceFiles 1550345

Apr 13 01:34:44 zxy-app-01-p duplicity_backup: ChangedFiles 1181867

Apr 13 23:06:00 zxy-app-01-p duplicity_backup: SourceFiles 1550041

Apr 13 23:06:00 zxy-app-01-p duplicity_backup: ChangedFiles 1181203

Apr 14 23:14:08 zxy-app-01-p duplicity_backup: SourceFiles 1549683

Apr 14 23:14:08 zxy-app-01-p duplicity_backup: ChangedFiles 1181556

Back to rdiff backup:

Apr 29 10:56:00 zxy-app-01-p rdiff-backup: SourceFiles 1537895

Apr 29 10:56:00 zxy-app-01-p rdiff-backup: ChangedFiles 1

Apr 29 20:22:42 zxy-app-01-p rdiff-backup: SourceFiles 1537873

Apr 29 20:22:42 zxy-app-01-p rdiff-backup: ChangedFiles 89

Apr 30 20:23:21 zxy-app-01-p rdiff-backup: SourceFiles 1537850

Apr 30 20:23:21 zxy-app-01-p rdiff-backup: ChangedFiles 87