Re: [Duplicity-talk] A duplicity backend for Amazon S3

[Brian Sutherland]

On Sun, Apr 30, 2006 at 06:45:27PM -0500, Ben Escoto wrote:
> >>>>> Brian Sutherland <address@hidden>
> >>>>> wrote the following on Thu, 13 Apr 2006 20:28:38 +0200
> > Hi,
> > 
> > I've written a backend for duplicity that writes to the Amazon S3
> > service [1]. It uses the bitbucket.py module [2].
> > 
> > It allows you to do something like this:
> > 
> >     duplicity / s3+http://${access_key}:${secret_key}/${bucket}
> > 
> > and so on to store you backups on S3. Which, well is very cheap. If
> > others start using this, it would be nice for the code to be included in
> > duplicity.
> > 
> > (Beware, bitbucket.py is a fairly young module and may have some bugs)
> > 
> > [1] http://www.amazon.com/gp/browse.html/103-1203077-2066220?node=16427261
> > [2] http://www.other10percent.com/?p=18
> 
> Thanks, this sounds like a great idea, and maybe even something I
> might use once my current remote-storage contract expires.  I couldn't
> download the bitbucket module though at the link you provided.

Perhaps here is better:
    http://cheeseshop.python.org/pypi/BitBucket

> I added the code in your message and changed the web page to mentioned
> the Amazon S3 link.

After testing the code for some time, I found it didn't work as well as
I hoped. It would fail to write to the connection often. Also the
BitBucket module changed it's interface in a backwards incompatible way.

Below is a new bitbucket backend [1] that I am running the first test on
right now (It takes a while). It will try to re-connect and re-try an
operation if one fails. Also included is a patch [2] to fix a bug in
bitbucket 0.3b when running without a config file.

> I'd like to add an S3 example to the
> documentation, but don't know what ${access_key} et al look like.  If
> you give me an S3 URL with some realistic looking values I'll add it
> to the web and man pages.

Here is the example from the amazon web pages:

access key: 44CF9590006BF252F707
secret key: OtxrzxIsfpFjA7SwPzILwy8Bw21TLhquhboDYROV
resulting url: 
s3+http://44CF9590006BF252F707:OtxrzxIsfpFjA7SwPzILwy8Bw21TLhquhboDYROV/bucket_name

A big problem with putting the keys in the url seems to be that the keys
can contain almost any character, my secret key contains a + and /. I am
not sure what other characters can appear or what to do about it.

[1]
class BitBucketBackend(Backend):

        def __init__(self, parsed_url):
                import bitbucket
                self.module = bitbucket
                parts = parsed_url.suffix.split('/')
                self.bucket_name = parts[-1]
                self.access_key, self.secret_key = 
'/'.join(parts[:-1]).split(':')
                self._connect()
                self.debug = False

        def _connect(self):
                self.connection = 
self.module.connect(access_key=self.access_key,
                                                      
secret_key=self.secret_key)
                self.bucket = self.connection.get_bucket(self.bucket_name)

        def put(self, source_path, remote_filename = None):
                """Transfer source_path (Path object) to remote_filename 
(string)

                If remote_filename is None, get the filename from the last
                path component of pathname.

                """
                if not remote_filename:
                    remote_filename = source_path.get_filename()
                bits = self.module.Bits(filename=source_path.name)
                if self.debug:
                    print 'Putting %s' % source_path.name
                try:
                    self.bucket[remote_filename] = bits
                except:
                    self._connect()
                    self.bucket[remote_filename] = bits


        def get(self, remote_filename, local_path):
                """Retrieve remote_filename and place in local_path"""
                local_path.setdata()
                bits = self.bucket[remote_filename]
                if self.debug:
                    print 'Getting %s' % local_path.name
                try:
                    bits.to_file(local_path.name)
                except:
                    self._connect()
                    bits.to_file(local_path.name)
                local_path.setdata()

        def list(self):
                """Return list of filenames (strings) present in backend"""
                try:
                    self.bucket.fetch_all_keys()
                    keys = self.bucket.keys()
                except:
                    self._connect()
                    self.bucket.fetch_all_keys()
                    keys = self.bucket.keys()
                if self.debug:
                    print 'Getting Keys: %s' % keys
                return keys

        def delete(self, filename_list):
                """Delete each filename in filename_list, in order if 
possible"""
                for file in filename_list:
                    if self.debug:
                        print 'Deleting: %s' % file
                    try:
                        del self.bucket[file]
                    except:
                        self._connect()
                        del self.bucket[file]


[2]

--- src/BitBucket-0.3b/bitbucket.py     2006-04-23 16:02:33.000000000 +0200
+++ pylib/bitbucket.py  2006-05-01 03:07:21.000000000 +0200
@@ -558,11 +558,15 @@
 #      SecretAccessKey: YourSecretAccessKeyHere
 #      Debug: 1
 #-------------------------------------------------------
-BB_DEFAULTS = {'AccessKeyID': '',
-              'SecretAccessKey': '',
-              'BucketNamePrefix': '',
-              'PageSize': 100,
-              'Debug': 1}
+_BB_DEFAULTS = {'AccessKeyID': '',
+               'SecretAccessKey': '',
+               'BucketNamePrefix': '',
+               'PageSize': '100',
+               'Debug': '1'}
+
+BB_DEFAULTS = {}
+for i in _BB_DEFAULTS:
+    BB_DEFAULTS[i.lower()] = _BB_DEFAULTS[i]

 #
 # Just wanted to subclass to include an implementation of HEAD

-- 
Brian Sutherland

Metropolis - "it's the first movie with a robot. And she's a woman.
              And she's EVIL!!"