Re: [Duplicity-talk] data set sizes

[mike]

> Question: Is there any limit to the size of data sets that can be backed up.
> I have a need to backup 30 gig of data which include both small files
> like openoffice docs as well as larger > a gig in size. Can some of the
> list members maybe pass on what they are doing with Duplicity?

I have not fully deployed it yet, but it looks like it will work; I
will be using duplicity to do offsite (as well as encrypted,
compressed, etc.) backups of some of my clients for my small hosting
business. It may be up to 200 gigs of data right now, file sizes
ranging from a few bytes to a gig or two max. From what I can tell
Duplicity will handle this like a champ. I'll be uploading it to
Amazon's S3 service using the s3+http connector (requires CVS + a
python bitbucket module)

I poked around to see what spanning options are allowed; there is no
option by default (would love to see one, and if I knew python better
I might try to make a patch/submit it) to define how large each volume
should be. I found around line 80 or so in the "duplicity-bin" Python
script (from CVS) that you can modify the parameters I've filled in
with a "filesize" variable:

if globals.encryption:
        at_end = gpg.GPGWriteFile(tarblock_iter, tdp.name,

globals.gpg_profile, filesize)
               else: at_end = gpg.GzipWriteFile(tarblock_iter,
tdp.name, filesize)

I made my own "filesize" variable - which currently I have set to:
filesize = 50 * 1024 * 1024

Which means 50 meg segments. By default it uses 5 meg segments. Which
with a lot of data (200+ gigs) would mean a LOT of volumes.

Perhaps there's reasoning around it but I see no reason why a command
line option couldn't be thrown in quick to override this. I could have
set even larger, but it seems like this was a good size.

> Also I would like to automate the backups. Does anyone have any
> experience with gpg-agent. If so is there a quick and dirty howto for
> Duplicity and gpg-agent?

I tried using this on a completely gpg-free machine. It would not
work. However, the minute I installed gpg, it worked; I did not even
have to setup any keys or anything. I noticed the first time it ran I
think it autogenerated some keyring for my user, but it did not prompt
me, or use the keys as far as I can tell; I believe duplicity uses the
gpg engine but uses the passcode you define if you define one:

> From the man page: "Duplicity will read the PASSPHRASE environment
variable to find the passphrase to give to GnuPG. If this is not set,
the user will be prompted for the passphrase."

So I think it is as simple as defining the same passphrase when you do
your archiving and then again on the extraction. Otherwise I believe
you can use GPG keys. That's been my experience at least.

Hope something here helps. I'm not a Python guru nor a duplicity pro,
but figured I would give you my thoughts/experiences :)

- mike