Re: [Duplicity-talk] Version 4 authorisation for Amazon S3 buckets

[Christian Saga]

Hi there,
I tried to do the things outlined by Mikko and I can't make it work the way I need to. I cannot activate the virtual environment automatically before running duply/duplicity. So when I use duply manually, this seems to work, however automated backup from cron just fails, as if it using the old debian package boto.

Second issue I have with this, is that duplicity is removed from Debian package management, hence also from automated updates. Isn't there a way to link the regular package into the virtualenv?

And last small addition: To install duplicity via pip install, you will also need do have python-dev installed via apt-get.

Regards
  Christian

First of all, you will need python-dev as well for the installation of duplicity out of 

Am Dienstag, den 28.04.2015, 17:24 +0300 schrieb Mikko Ohtamaa:
>     
> >         i would love to only use packaged python scripts, but failed to grasp how this can be combined and kept up-to-date? So far it looks like the pip packages are all ending up in /usr/local/lib and then being preferred above other packages
> >         
> >     
>     
>     
>     The process would be something like this
>     
>     
>     Install some .deb dependencies:
>     
>     sudo apt-get install librsync-dev
>     
>     
>     
>     First create isolated virtual environment folder called duplicity-venv. You can do this a as normal user, but use user who will take the backups (root?).
>     
>     First get virtualenv command: 
>     
>     sudo apt-get install python-virtualenv:
>     
>     
>     Then create a new virtualenv
>     
>     
>     
>     cd $HOME
>     
>     virtualenv duplicity-venv 
>     
>     
>     Then activate it - this will change PATH in the current shell session:
>     
>     
>     source duplicity-venv/bin/activate
>     
>     
>     Now you can do clean installs with pip and everything is kept nicely in duplicity-venv folder and duplicity-venv/bin preferred ove /usr/bin and /usr/local/bin
>     
>     
>     Let's install duplicity (this is a tricky one because duplicity is not distributed on pypi.python.org):
>     
>     pip install lockfile
>     
>     pip install https://launchpad.net/duplicity/0.7-series/0.7.02/+download/duplicity-0.7.02.tar.gz
>     
>     
>     Check we now use locally installed duplicity:
>     
>     
>     which duplicity
>     
>         /root/duplicity-venv /bin/duplicity
>     
>     
>     duplicity --version
>        
>         duplicity 0.7.02
>     
>     
>     
>     To update duplicity later
>     
>     
>     source $HOME/duplicity-venv/bin/activate
>     
>     pip install -https://launchpad.net/duplicity/0.7-series/0.7.02/+download/duplicity-0.7.NEWVERSION.tar.gz
>     
>     
>     
>     Note: If duplicity would be distributed on PyPi one could update just by doing
>     
>     
>     pip install -U duplicity
>     
>     
>     Hope this helps,
>     
>     Mikko
>     
>      
> >         
> >         Regards
> >           Christian
> >     
> >         
> >         
> >         Am Dienstag, den 28.04.2015, 15:39 +0300 schrieb Mikko Ohtamaa: 
> > >             
> > >             
> > >             On 28 April 2015 at 15:36, Jacob Godserv <address@hidden> wrote:
> > > >                 For what it's worth, using a second package manager like pip will cause long term upgrade conflicts with your primary package manager in the future. You should avoid using it unless it respects your primary package manager and installs to non-default locations.
> > > >                 
> > > >                 
> > > >             
> > >             Please create virtualenv isolated environments installing Python packages using pip command from PyPi.
> > >             
> > >             
> > >             The official guide: https://packaging.python.org/en/latest/projects.html#virtualenv
> > >             
> > >             
> > >             You should almost never need to run sudo pip and replace your system packages unless you know what you are doing.
> > >             
> > >             
> > >             
> > >             Just my Python developer advice,
> > >             
> > >             
> > >             Cheers,
> > >             
> > >             Mikko
> > >             
> > >             
> > >              
> > > >                 
> > > >                 On Mon, Apr 27, 2015, 12:08 Christian Saga <address@hidden> wrote: 
> > > > >                     Hi Claudio,
> > > > >                     thanks for sharing, I tried that as well, but it did not help. It was only the updated version that made it work for me.
> > > > >                     
> > > > >                     Regards
> > > > >                       Christian 
> > > > >                     
> > > > >                     
> > > > >                     Am Montag, den 27.04.2015, 16:06 +0100 schrieb Cláudio Gil: 
> > > > > >                         Hi,
> > > > > >                         
> > > > > >                         
> > > > > >                         I've been using duplicity 0.6.25 and a Frankfurt bucket as target. The only thing* I needed to do was:
> > > > > >                         
> > > > > >                         
> > > > > > >                             export S3_USE_SIGV4="True" 
> > > > > > >                         
> > > > > >                         
> > > > > >                         * I may have upgraded python-boto (through the system packages) but the support for this is there since 2.24.
> > > > > >                         
> > > > > >                         
> > > > > >                         Cheers,
> > > > > >                         Cláudio
> > > > > >                         
> > > > > >                         2015-04-27 15:41 GMT+01:00 Christian Saga <address@hidden>:
> > > > > > >                             Sorry, took a while to put this together. I put up a little Webpage to store this solutions for me. You find it here: sowhatisthesolution.wordpress.com
> > > > > > >                             
> > > > > > >                             Also, not to be a traffic leech, here the solution ;-): If you are planning to use the new EU (Frankfurt) location called eu-central-1 you will run into problems as this location only supports the V4 authentication.
> > > > > > >                             
> > > > > > >                             You might encounter the following error when using duply (even if the config worked on other locations before)
> > > > > > > >                                 The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256
> > > > > > > >                             
> > > > > > >                             So what is the solution
> > > > > > >                             To solve this you will need to upgrade your duplicity and boto framework as following:
> > > > > > >                             
> > > > > > >                             Package duplicity needs version > 0.7.02-1
> > > > > > >                             This version is available within the unstable branch of debian. I used package pinning to get this package. You can easily find articles for this via Google, I found this one helpful.
> > > > > > >                             Be careful to understand pinning, as you might confuse APT with a wrong configuration. You should always use sudo apt-cache policy [packagename] and the -s parameter on upgrades first to check what will happen.
> > > > > > >                             
> > > > > > >                             Package python-boto needs version >= 2.36.0
> > > > > > >                             This is a bit tricky, as the Debian packages, even within the unstable branch only give you version 2.34.0-2.
> > > > > > >                             To get the newest version from PyPI (what is that?) you need to install the python-pip package
> > > > > > > >                                 sudo apt-get install python-pip
> > > > > > > >                             
> > > > > > >                             After that you can install the newest version of boto
> > > > > > > >                                 sudo pip install --upgrade boto
> > > > > > > >                             
> > > > > > >                             This will install the newest version of boto in parallel to your debian package. I haven’t encountered any problems with the parallel installation. The PIP installed one will take precedence to the debian package. If you want to uninstall the PIP version, use sudo pip uninstall boto.
> > > > > > >                             To get an overview of PIP packages, use pip freeze.
> > > > > > >                             
> > > > > > >                             Package duply did not need additional changes, I am using version 1.9.1-1 
> > > > > > >                             
> > > > > > >                             Am Montag, den 02.02.2015, 22:13 +0100 schrieb address@hidden: 
> > > > > > > > could you please post a short summary with the error and steps to solve it. others with this problem can pick it up then via internet search engine in the mailing list archives ;)
> > > > > > > >
> > > > > > > > ..ede/duply.net
> > > > > > > >
> > > > > > > > On 02.02.2015 21:33, Christian Saga wrote:
> > > > > > > > > Hi Ken, Hi Ed
> > > > > > > > > yep, that did the trick. EU central region is working now.
> > > > > > > > > Got boto 2.36.0 now on python 2.7.
> > > > > > > > > 
> > > > > > > > > Thanks for your help
> > > > > > > > >   Christian
> > > > > > > > > 
> > > > > > > > > Am Montag, den 02.02.2015, 07:04 -0600 schrieb Kenneth Loafman:
> > > > > > > > > 
> > > > > > > > >> sudo pip install --upgrade boto
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > _______________________________________________
> > > > > > > > > Duplicity-talk mailing list
> > > > > > > > > address@hidden
> > > > > > > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > > > > > > > 
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > Duplicity-talk mailing list
> > > > > > > > address@hidden
> > > > > > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > > > > > >                             
> > > > > > >                             
> > > > > > >                             
> > > > > > >                             
> > > > > > >                             _______________________________________________
> > > > > > >                             Duplicity-talk mailing list
> > > > > > >                             address@hidden
> > > > > > >                             https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > > > > >                             
> > > > > > >                         
> > > > > >                         
> > > > > >                         
> > > > > > _______________________________________________
> > > > > > Duplicity-talk mailing list
> > > > > > address@hidden
> > > > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > > > >                     
> > > > >                     
> > > > >                     
> > > > >                     _______________________________________________
> > > > >                     Duplicity-talk mailing list
> > > > >                     address@hidden
> > > > >                     https://lists.nongnu.org/mailman/listinfo/duplicity-talk 
> > > > >                 
> > > >                 
> > > >                 _______________________________________________
> > > >                 Duplicity-talk mailing list
> > > >                 address@hidden
> > > >                 https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > >                 
> > > >             
> > >             
> > >             
> > >             
> > >             -- 
> > >             Mikko Ohtamaa
> > >             http://opensourcehacker.com
> > >             http://twitter.com/moo9000
> > >             
> > >             
> > > _______________________________________________
> > > Duplicity-talk mailing list
> > > address@hidden
> > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > >         
> >         
> >         
> >     
> >         
> >         _______________________________________________
> >         Duplicity-talk mailing list
> >         address@hidden
> >         https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >         
> >     
>     
>     
>     
>     -- 
>     Mikko Ohtamaa
>     http://opensourcehacker.com
>     http://twitter.com/moo9000
>     
>     
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk