Re: [Duplicity-talk] Always reads manifest from remote

[edgar . soldin]

On 10.12.2013 20:49, address@hidden wrote:
> I was not aware that truncated remote manifests are a thing to look out for.  
> My immediate reaction is that the current state of the code is inconsistent 
> with solving that problem. As it is, the code silently accepts having the 
> manifest only on the remote, but throws an assertion failure if the manifest 
> is only local. That seems odd if the local manifest is the more trusted one. 
> It also, as I mentioned, silently ignores the absence of the key needed to 
> compare the manifests.

he probably means that the manifest was uploaded incompletely to the backend
 
> Seems to me that if you want to solve the problem of the remote manifest 
> being truncated, you should read it back and check it right after writing it 
> -- not wait until the next backup, when the local copy might be missing. 

we deal with untrustworthy backends in duplicity. so no. if we stat this way we 
have to check every file uploaded, leading to multiplied bandwidth and longer 
execution times.

it is up to the user to 'verify' his backup. not doing always includes a risk.

>It should also insist on the key being available. What do you think about this 
>change?

already pointed that out years ago. read the bug report
https://bugs.launchpad.net/duplicity/+bug/687295

i also proposed a cleaner solution, but haven't had time to tackle it.

> I agree that making the secret key available during inc backups is no big 
> deal.  The docs should probably make clear that this is necessary.  I'll 
> review them and try to make a more specific suggestion.

they do not, 'cause it works on english systems (bugreport).
eventually duplicity is supposed to work even without a secret key present. 
unfortunately for now it is a very tinkered solution.
 
> About Glacier, I just want to comment that there is no need for a backend. 
> I'm using the S3 backend and using lifecycle transition rules to send things 
> to Glacier, and I'm not the only one. (And, incidentally, checking the 
> manifest right after writing it would be Glacier-friendly -- not that I'm 
> suggesting that should be a big consideration.)

currently duplicity acts up if some manifest/volume is corrupted, so i ask 
myself. why do we have this extra test? 

..ede
 
> Thanks,
> Dave Cottingham
> 
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From: *"Kenneth Loafman" <address@hidden>
> *To: *"Discussion of the backup program duplicity" <address@hidden>
> *Sent: *Tuesday, December 10, 2013 11:59:45 AM
> *Subject: *Re: [Duplicity-talk] Always reads manifest from remote
> 
> It's needed since we sometimes have truncated manifests on the remote and 
> this one is critical.  We should probably extend the check for truncation to 
> all the files we write.
> 
> If you have encrypted backups, you will need your key to increment those 
> backups.  I don't see a problem here.
> 
> I think we need to keep the check, but condition it on the Glacier backend, 
> when/if that comes along.
> 
> ...Ken
> 
> 
> On Fri, Dec 6, 2013 at 12:06 PM, <address@hidden <mailto:address@hidden>> 
> wrote:
> 
>     I've figured out why we have bug 1170113, which is that on incremental 
> backups duplicity always reads the latest manifest from the remote even if 
> there is a copy in the local cache. This is usually innocuous, but if the 
> remote is Amazon S3 transitioning to Glacier, the remote can't be read 
> (without a previous arrangements) so it's fatal.
> 
>     The question is what would be the preferred fix. I'll just copy here my 
> latest entry from the bug report page, and I'd be interested in what you guys 
> think:
> 
>     I have identified why this bug occurs. For incremental backups duplicity 
> always calls BackupSet.check_manifests(), described as "Make sure remote 
> manifest is equal to local one." It reads the manifest from the remote and 
> checks that it is identical to the local copy. I believe this is the only 
> file that duplicity checks in this way.
> 
>     Of course, the usual case is that the remote files are encrypted and 
> duplicity is not in possession of the secret key. After reading the file from 
> remote, if duplicity finds this is the case then it just proceeds as if the 
> comparison had succeeded.
> 
>     As I mentioned above, this behavior is fatal if the remote is Amazon S3 
> transitioned to Glacier. But there are several ways one could imagine 
> patching it:
> 
>     1. Get rid of check_manifests() altogether. I don't see why in this one 
> case we should be paranoid about a remote file not matching the one in the 
> local cache.
> 
>     2. Instead of ignoring only the exception that the secret key is not 
> available, also ignore the exception of not being able to read from the 
> remote.
> 
>     3. Add a command line flag to disable check_manifests().
> 
> 
>      - Dave Cottingham
> 
> 
> 
>     _______________________________________________
>     Duplicity-talk mailing list
>     address@hidden <mailto:address@hidden>
>     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>