[Duplicity-talk] Checkpoints during backup

[Robin Smidsrød]

I have a lot of files (about 280GB) that I need to backup. I have a 
reasonably slow upstream connection (ADSL, 550kbps). I have a server 
somewhere on the internet with lots of storage space and ssh/sftp 
connection. I want to backup my files to that location. With duplicity 
this is a breeze. There is only one catch. It takes too long time to do 
the initial full backup.

I think my situation is pretty common.

Would it be possible to enable some kind of checkpoint method when 
uploading the files, so that it is possible to continue a stopped backup 
without having to transfer everything again? I know basic rsync allows 
this, but rsync is not an option for my setup (need remote encrypted files).

I was thinking: What if every XX minutes (say 30) or XX megabytes there 
is written a sort of manifest of what you have done so far, so that if 
the process is cancelled, the next time you start the process you can 
skip the data that was already written to the (partial, but verified) 
manifest.

With something like this it would be unproblematic if the process was 
stopped, the computer was shut down, or the connection disappeared while 
running the backup. The way it is now the process have to be restarted 
from scratch if anything happens.

If my calculations are correct, my initial backup should take 
approximately 50 days to complete. And of course, during 50 days, a lot 
of interruptions could happen. Because I have a 6Mbps downlink a restore 
wouldn't take more than 5 days to complete, so it's definitely within 
reason to do something like this for complete disaster recovery.

Is this kind of thing incredibly complicated to implement with the 
current codebase?

-- Robin

PS: If it is doable, I might be interested in sponsoring development of 
this feature.