Re: [Duplicity-talk] Duplicity and put only backend permissions

[edgar . soldin]

On 25.04.2015 02:08, Christian Saga wrote:
> Hi there,
> so I tested a bit and came to the following result:
> - You can remove the deleteObject rights from S3 and by that it is not 
> possible anymore to delete old created backups via access to S3
> - You can still do a full and incremental backups with duplicity/duply (At 
> least I could not find any errors in the log, the files were uploaded)
> - You can regularly restore files from the backup
> - You canNOT do a purge and delete old backups
> - You can transfer the config to another machine and initiate a purge from 
> there after switching the S3 user to one with deleteObject rights
> 
> I wasn't able to test yet a pickup of an interrupted backup, will do that on 
> the weekend.

please do as these are probably your no-go candidates
 
> Two things to mention, which did not seem to be correct?
> - when deleting: Duplicity actually throws a row of nasty errors/exceptions 
> (in which you can spot the S3 denial) and goes into a loop of retries. I 
> killed duplicity after a few mins, so I am not sure how long it would try. 
> Some better exception handling would be nice here ;-)

its the backend retrying. keep in mind you asked duplicity to purge, so it 
atually tries to ;). see manpage, parameter --num-retries 
 http://duplicity.nongnu.org/duplicity.1.html

> - when purging from a different machine: It kills ALL signature files on the 
> backend. I always thought a backup set contains (manifest, difftar and 
> sigtar). After the purge however only manifest and difftar remains. Is that 
> supposed to happen?
> 

no, purge should remove everything.  run your purge without --force to see what 
it wants to delete first. check that all file types are listed there. then run 
the 'purge --force' and the delete is incomplete again check the log for errors.
if all that doesn't help, run it again with max verbosity '-v9' and post the 
log (zipped here or pastebin). obfuscate stings you deem private in it.

btw. for tests like that you can easily reuse a once created duplicity 
repository by copying into a new folder, so that you do not have to recreate it 
for every test.

..ede/duply.net

> Regards
>   Christian
> 
> 
> Am Freitag, den 24.04.2015, 17:28 +0200 schrieb address@hidden:
>> ;).. we did (talk) years ago, when the resuming was considered new and 
>> unsafe, but it was never implemented.. ede
>>
>> On 24.04.2015 16:57, Kenneth Loafman wrote:
>> > I could have sworn we had a --no-resume option.  We probably just talked 
>> > about it.
>> > 
>> > ...Ken
>> > 
>> > 
>> > On Fri, Apr 24, 2015 at 8:37 AM, <address@hidden <mailto:address@hidden> 
>> > <mailto:address@hidden>> wrote:
>> > 
>> >     not exactly.
>> > 
>> >     you will not be able to resume or retry, but provided your connection 
>> > is good and you can afford the traffic you'll have a one shot backend, 
>> > where
>> >     - you'll start a new backup on every run*
>> >     - a hacked source cannot destroy/manipulate your older backups
>> > 
>> >     ..ede
>> > 
>> >     * (you will still need a way to disable resuming in duplicity)
>> > 
>> >     On 24.04.2015 15 <tel:24.04.2015%2015>:20, Kenneth Loafman wrote:
>> >     > I guess that's right.  If they could not overwrite, they probably 
>> > could not delete either, so kinda useless as a target.
>> >     >
>> >     > On Fri, Apr 24, 2015 at 7:24 AM, <address@hidden 
>> > <mailto:address@hidden> <mailto:address@hidden> <mailto:address@hidden 
>> > <mailto:address@hidden>>> wrote:
>> >     >
>> >     >     ok, in conclusion that would mean that resume will build a 
>> > potentially partially uploaded volume and try to upload it. if there is 
>> > already a file with the same name, if only partial, the backup will fail 
>> > in case it cannot  overwrite mentioned file.
>> >     >
>> >     >     that would mean the initial idea of a write once backend would 
>> > require a possibility to disable resuming, which are potentially doomed to 
>> > fail.
>> >     >
>> >     >     ..ede/duply.net <http://duply.net> <http://duply.net>
>> >     >
>> >     >     On 24.04.2015 13 <tel:24.04.2015%2013> <tel:24.04.2015%2013>:24, 
>> > Kenneth Loafman wrote:
>> >     >     > No, that's not what I meant.  When duplicity resumes, it looks 
>> > at the manifest.  If the volume is in the manifest, it's assumed to have 
>> > uploaded.  If not, it restarts at the next volume.  Whatever volume may 
>> > have been partially built (in temp space), or uploaded (partially), will 
>> > be ignored.
>> >     >     >
>> >     >     > ...Ken
>> >     >     >
>> >     >     >
>> >     >     > On Fri, Apr 24, 2015 at 5:04 AM, <address@hidden 
>> > <mailto:address@hidden> <mailto:address@hidden> <mailto:address@hidden 
>> > <mailto:address@hidden>> <mailto:address@hidden <mailto:address@hidden> 
>> > <mailto:address@hidden <mailto:address@hidden>>>> wrote:
>> >     >     >
>> >     >     >     so resumes only happen on local incomplete volumes? but 
>> > not if an upload was interrupted?
>> >     >     >
>> >     >     >     on a sidenote: retrying uploads will of course fail as 
>> > well with Christian's approach.
>> >     >     >
>> >     >     >     ..ede
>> >     >     >
>> >     >     >     On 24.04.2015 11 <tel:24.04.2015%2011> 
>> > <tel:24.04.2015%2011> <tel:24.04.2015%2011>:50, Kenneth Loafman wrote:
>> >     >     >     > It would only be able to resume by rewriting and 
>> > resending the incomplete volume and continuing from there.  The upload 
>> > part is in a temporary volume and is not saved.
>> >     >     >     >
>> >     >     >     > ...Ken
>> >     >     >     >
>> >     >     >     >
>> >     >     >     > On Fri, Apr 24, 2015 at 4:00 AM, <address@hidden 
>> > <mailto:address@hidden> <mailto:address@hidden> <mailto:address@hidden 
>> > <mailto:address@hidden>> <mailto:address@hidden <mailto:address@hidden> 
>> > <mailto:address@hidden <mailto:address@hidden>>> <mailto:address@hidden 
>> > <mailto:address@hidden> <mailto:address@hidden <mailto:address@hidden>> 
>> > <mailto:address@hidden <mailto:address@hidden> <mailto:address@hidden 
>> > <mailto:address@hidden>>>>> wrote:
>> >     >     >     >
>> >     >     >     >     On 23.04.2015 18:13, Christian Saga wrote:
>> >     >     >     >     > Hi there,
>> >     >     >     >     > I was just thinking about a new setup. Basically 
>> > allowing duplicity to only write new objects to S3 and not allow deletion 
>> > of old items.
>> >     >     >     >     > This would allow me to have the backups secured, 
>> > in case the server gets hijacked.
>> >     >     >     >     >
>> >     >     >     >     > Do you know if this would pose problems with 
>> > duplicity?
>> >     >     >     >
>> >     >     >     >     it should not
>> >     >     >     >
>> >     >     >     >     >I feel, it is only writing new objects for full and 
>> > incremental backups to s3. Only the purge would delete old versions? If I 
>> > would turn the purge off, then it should work?
>> >     >     >     >
>> >     >     >     >     the purge would either fail or succeed, but in 
>> > reality not have done anything depending of the implementation in the 
>> > backend code. either way not an issue according to your write only 
>> > strategy.
>> >     >     >     >
>> >     >     >     >     > Or is there any other action, that would need the 
>> > possibility to change files in the backend instead of just uploading?
>> >     >     >     >     >
>> >     >     >     >
>> >     >     >     >     be aware of interrupted uploads. currently duplicity 
>> > might try to resume a backu0p and in turn overwrite a volume on the 
>> > backend, which would defeat your purpose.
>> >     >     >     >
>> >     >     >     >     Ken: what does happen if the upload part during a 
>> > resumed backup fails? would duplicity fail but try resuming again on the 
>> > next run?
>> >     >     >     >
>> >     >     >     >     ..ede/duply.net <http://duply.net> 
>> > <http://duply.net> <http://duply.net> <http://duply.net>
>> >     >     >     >
>> >     >     >     >
>> >     >     >
>> >     >     >
>> >     >
>> >     >     _______________________________________________
>> >     >     Duplicity-talk mailing list
>> >     >     address@hidden <mailto:address@hidden> <mailto:address@hidden> 
>> > <mailto:address@hidden <mailto:address@hidden>>
>> >     >     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>> >     >
>> >     >
>> >     >
>> >     >
>> >     > _______________________________________________
>> >     > Duplicity-talk mailing list
>> >     > address@hidden <mailto:address@hidden> <mailto:address@hidden>
>> >     > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>> >     >
>> > 
>> >     _______________________________________________
>> >     Duplicity-talk mailing list
>> >     address@hidden <mailto:address@hidden> <mailto:address@hidden>
>> >     https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>> > 
>> > 
>> > 
>> > 
>> > _______________________________________________
>> > Duplicity-talk mailing list
>> > address@hidden <mailto:address@hidden>
>> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>> > 
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden <mailto:address@hidden>
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>