Re: [Duplicity-talk] PAR2 for duplicity

[Peter Schuller]

> This has come up before, but I'm wondering if anyone is really interested in 
> it.

I am definitely interested in it. I have semi-planned on contributing
something at some point, but I was hoping to first do something about
the way file uploads are handled (there is IMO an abstraction missing
between the front-end backup code and the backend). This also relates
to the somewhat broken handling of local archive files.

> There are two basic approaches: create PAR2 per backup run (i.e. for all 
> files) or per individual .tar.gz.gpg. The former has the advantage that the a 
> whole file may be missing and recovery possibly still would work. But for 
> recovery to work, you'd have to have all files available which is obviously 
> not the case if PAR2 is done per archive. 

I considered this; the major argument against doing par2 over the
entire archive, is that, unless the par2 command line tools have
features I have missed, you will need a copy of the full backup in
order to generate the par2 information. This does fundamentally change
the resource requirements for duplicity.

That said, some kind of compromise could be reached. Given a certain
percentage redundancy, one could generate par2 files over *batches* of
files sufficiently large that one can recover from a single file being
lost (so eg for 5% redundancy you might use 20 file batches).

The best thing would probably be if there was a FEC library of some
kind (assuming we don't want to actually implement the FEC
ourselves). That might give us some more flexibility (single pass
required AFAIK, as long as the redundant information fits in
RAM). Anyone aware of something like this?

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <address@hidden>'
Key retrieval: Send an E-Mail to address@hidden
E-Mail: address@hidden Web: http://www.scode.org

pgp0k6SJR4HJ_.pgp
Description: PGP signature