Re: [Duplicity-talk] Following up on an extended attributes question

[Arjun Krishnan]

Hi Ken

I'd be happy to work on it, with a couple of caveats: I'm a research mathematician by day, and have little professional programming experience.

So as long as people are willing to indulge slow progress and trivial questions as I get familiar with the codebase, I'd be willing to give it a shot!

Currently we use the GNU_FORMAT, which is the default format, so there is no pax_header in that format.  For sake of backwards compatibility, we'd need to be able to read GNU_FORMAT, and the PAX_FORMAT would need to be readable by GNU tar for manual recovery of a corrupted backup.  Perhaps it's time to change formats?  I don't know all the tradeoffs we'll need to make and what the complexity would be.

gnu tar appears to support pax_format with `--xattrs` flag. From what I've seen so far, tarfile happily reads both pax and gnu format

files, but it neither appends extended attributes when creating files, nor does it extract files properly and set extended attributes. 

Maybe they didn't want to add a dependency on the xattr module, or maybe this is for cross platform compatibility.

Since pax_headers are key/value pairs, we need to track their size and be prepared for a value that may exceed our max block size.  I've read of cases where key/value pairs exceed the size of the file itself, performing the functional equivalent of a dictionary file.  We would have to allow for cases where a large value crosses a volume boundary.

I can see that the blocksize is a diff process parameter,

and that volsize affects the difftar size. I'm still figuring out what's being compared during the diff process,

and how the diffs are being created, so I don't understand how the xattrs fit in to this process. 

Is there a brief technical overview of what duplicity is doing?

Arjun