Re: [Duplicity-talk] Big bandwidth bill from Rackspace

[edgar . soldin]

On 10.06.2013 23:47, Joseph D. Wagner wrote:
> On 06/10/2013 12:20 pm, address@hidden wrote:
> 
>> On 10.06.2013 21:10, Lou wrote:
>>
>>> I recently got hit with a 200 usd bill from Rackspace for bandwidth
>>>
>>>> while using duply against their cloudfiles backend. I'd like to find
>>>> out why - the backup source didn't change much, and reading the
>>>> random daily e-mail reports that duply generates don't show any big
>>>> numbers. Does anyone know if perhaps the "verify" part of
>>>> backup_verify_purge is doing something traffic intensive?
>>
>> yes. essentially it restores any file of the latest backup and compares
>> it to the original path in the local file system. therefore pretty much
>> all the last full and the incremental volumes have to be downloaded
>> again.
> 
> Why does it need to download the entire file?  Why not just download a list 
> of files and their checksums (CRC32/64 or if paranoid SHA1/256/512)?  This 
> would minimize bandwidth usage for verify operations and be just as effective 
> as comparing the whole file.
> 

that's the way it is currently implemented. you are totally right. the 
signatures contain enough information for incrementing without having to 
restore the previous state of a file, why does verify need it! because nobody 
hacked it in until now ;)

feel free to step up.. ede/duply.net