Re: [Duplicity-talk] Duplicity full backup has become slow

[edgar . soldin]

On 11/12/2017 19:19, Matthew Harrison via Duplicity-talk wrote:
> Hi,
> 
> I've been happy using duplicity to backup my Ubuntu PC to my NAS for around
> 18 months now, using rsync.
> 
> I do a full back up every 2 weeks, and and incremental daily.
> 
> I noticed in August that the full backup started taking around twice as
> long, from
> 
> --------------[ Backup Statistics ]--------------
> StartTime 1502355166.95 (Thu Aug 10 09:52:46 2017)
> EndTime 1502373067.73 (Thu Aug 10 14:51:07 2017)
> ElapsedTime 17900.78 (4 hours 58 minutes 20.78 seconds)
> SourceFiles 1361618
> SourceFileSize 212181777813 (198 GB)
> NewFiles 1361618
> NewFileSize 212181777813 (198 GB)
> DeletedFiles 0
> ChangedFiles 0
> ChangedFileSize 0 (0 bytes)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 1361618
> RawDeltaSize 211041882531 (197 GB)
> TotalDestinationSizeChange 166597720163 (155 GB)
> Errors 0
> -------------------------------------------------
> 
> to
> 
> --------------[ Backup Statistics ]--------------
> StartTime 1503684459.99 (Fri Aug 25 19:07:39 2017)
> EndTime 1503718307.60 (Sat Aug 26 04:31:47 2017)
> ElapsedTime 33847.61 (9 hours 24 minutes 7.61 seconds)
> SourceFiles 1361798
> SourceFileSize 212042512113 (197 GB)
> NewFiles 1361798
> NewFileSize 212042512113 (197 GB)
> DeletedFiles 0
> ChangedFiles 0
> ChangedFileSize 0 (0 bytes)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 1361798
> RawDeltaSize 210902807446 (196 GB)
> TotalDestinationSizeChange 166567853868 (155 GB)
> Errors 0
> -------------------------------------------------
> 
> As you can see, pretty much the same statistics except for the time taken.
> 
> Now in between these 2 full backups, probably the most significant change
> to my system was an upgrade from Ubuntu 16.10 to 17.04.
> 
> I don't think network speed is an issue because some of the (4GB) vol files
> can take as little as 5 minutes (looking at the timestamps) which I assume
> includes creation and transfer time.
> I've noticed that the slowest archive files to be produced took around 1h
> 25m, and either consisted of lots and lots of small files, or very large
> binary files.
> 
> Is anyone aware of any change in the versions of the Ubuntu packages that
> duplicity relies on, between Ubuntu 16.10 and 17.04 that might have caused
> this slowdown?
> 
> I haven't looked yet at adding extra logging to help see if my thoughts on
> creation vs transfer time are correct.
> 
> On the plus side, this has nudged me to look more closely at what
> unnecessary cruft I'm backing up :D
> 
> Many thanks, and best regards,

Matthew,

what are the respective duplicity versions that were/are used?

..ede/duply.net