Re: gpg: [don't know]: invalid packet (ctb=14) - WAS: [Duplicity-talk] Rollup Functionality and Parity?

[Peter Schuller]

> as I have to use a ftp backup space and had now the "gpg ctb 14" error 
> for the 4th time.. I am interested in a best practice or some other 
> solution except doing a full backup, which I do all the time now if this 
> occurs.

Well, my take on it is that if you are having any kind of repeated
issues like that it would either be because of some bug or
incompatibility that systematically introduces incorrect data, or
actual corruption (bitrot, bitflip in transit, etc).

If there is a bug in duplicity that should be fixed, but if there is
actual corruption going on the question is whether you might want to
switch backup location to some place that doesn't corrupt files.

This does not mean I am not 100% for improving error recovery in
duplicity.

> I like the idea of parity as this could help if there is minor damage. 
> But a maybe even easier way could be like this (at least easier to 
> implement)
> a) doing a backup to a remote filesystem but keeping a copy of the 
> backup files locally
> b) a verify of the backup files against their signature
> c) retransfer of defect files only (and do step b again for a limited 
> count, and telling a severe error if it failed multiple times)

[snip]

> Does this all make sense to you guys .. any advice how to circumvent the 
> gpg error at all, except of not using ftp space?

Ignoring the cause of the problem, keeping local files might be a good
way to make recovery more efficient. But I am really scepticle to the
usefulness of the backup if you are systematically having corruption
or incorrect creation of files.

Particularly if it is corruption, since it would indicate exiting full
backups would get blown away. If it is software bug that causes files
to be written incorrectly, at least it only affects things insofar as
incrementals are made. Your problem in this case, if I remember
correctly from previous posts, is that ideally you would just like to
be able to "forget" a previous incremental and re-do it, instead of
re-doing a full backup? That is, the problem for you isn't that the
initial full backup becomes corrupt after initial creation?

One possibility is to run your duplicity session locally and use the
features of a client like lftp to upload the files (mirror -R on the
directory and it acts similarly to rsync in that it figures out which
files are missing and uploads them).

This could be a good way to find out what's going on as well. For
example, if you still run into the same problem when backing up
locally, that means the problem is not with any kind of server-side
corruption (unless you're massively unlucky and this happens in both
cases). Alternatively if everything works well you should be able to
discover that the contents of the remote and local files have suddenly
changed.

That's if you'd be willing to investigate a bit more to try to narrow
down the problem and see if it is a duplicity bug or not.

With regards to supporting keeping a local copy of files, and
implementing par2, it is on my personal todo list. Not sure when I
will look at it.

Hmm. In the mean time, one possible improvement could be to simply
emit the md5sums of uploaded files as part of regular duplicity
logging. Say that gets emitted at verbosity 5, you can run your
backups at verbosity 5 with output logged to some file (e.g. using
script(1)). The next time you have this problem, you will be able to
go back and find the checksum of the offending file at the time it was
uploaded. This would perhaps be the easiest way to get a feel for
whether the contents of the file actually changed. It won't however
say whether the file was corrupted in transit or corrupted server-side
after the transfer.

Would that be of interest to you?

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <address@hidden>'
Key retrieval: Send an E-Mail to address@hidden
E-Mail: address@hidden Web: http://www.scode.org

pgppt6NZQSaBp.pgp
Description: PGP signature