Re: [Duplicity-talk] Duplicity hanging after errors

[edgar . soldin]

On 19.12.2014 00:49, E.B. wrote:
> Hi,
> I'm running duplicity from a cron script on two ubuntu 12.04 servers
> (installed from apt-get, so duplicity version 0.6.25) and they both
> stopped working lately. Not sure if the problems are related or just
> coincidence.
> 
> Big problem that the error causes duplicity to hang and not exit.
> Wondering why that is? If I add "--progress" then it will die (the
> trace goes through dummy_backup first where it dies, but
> write_multivol just hangs). Is my only solution here to add
> "--progress" to my cron script?

no, usually you would raise --verbosity=debug (-v9) and run from command line 
to see if you get the same error, and more details.
read further below.

> But the errors themselves I don't understand and would appreciate
> feedback. Is the only fix to wipe the archive completely and start
> with new full backup (so I lose any backup histories)?
> 
> Error trace below with control-c to cancel at the end (run from cli
> but same trace showing in my cron log). Both machines errors appear
> during normal diff processing of files on the server (A /some/path...
> A /some/path M /some/path.... M /some/path... etc).
> 
> REALLY appreciate the software and the help.
> 
> 
> First machine:
> 
> Traceback (most recent call last):
>   File "/usr/local/bin/duplicity", line 1509, in <module>
>     with_tempdir(main)
>   File "/usr/local/bin/duplicity", line 1503, in with_tempdir
>     fn()
>   File "/usr/local/bin/duplicity", line 1352, in main
>     do_backup(action)
>   File "/usr/local/bin/duplicity", line 1485, in do_backup
>     incremental_backup(sig_chain)
>   File "/usr/local/bin/duplicity", line 613, in incremental_backup
>     dummy_backup(tarblock_iter)
>   File "/usr/local/bin/duplicity", line 202, in dummy_backup
>     while tarblock_iter.next():
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 518, in next
>     result = self.process(self.input_iter.next())
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 190, in get_delta_iter
>     for new_path, sig_path in collated:
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 281, in collate2iters
>     relem2 = riter2.next()
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 350, in combine_path_iters
>     refresh_triple_list(triple_list)
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 336, in refresh_triple_list
>     new_triple = get_triple(old_triple[1])
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 322, in get_triple
>     path = path_iter_list[iter_index].next()
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 234, in sigtar2path_iter
>     for tarinfo in tf:
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/tarfile.py", line 
> 2470, in next
>     tarinfo = self.tarfile.next()
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/tarfile.py", line 
> 2319, in next
>     self.fileobj.seek(self.offset)
>   File "/usr/lib/python2.7/gzip.py", line 429, in seek
>     self.read(1024)
>   File "/usr/lib/python2.7/gzip.py", line 256, in read
>     self._read(readsize)
>   File "/usr/lib/python2.7/gzip.py", line 320, in _read
>     self._read_eof()
>   File "/usr/lib/python2.7/gzip.py", line 342, in _read_eof
>     hex(self.crc)))
> IOError: CRC check failed 0x43872847 != 0x7d505e15L
> 
> ^Cclose failed in file object destructor:
> IOError: [Errno 32] Broken pipe
> Exception KeyboardInterrupt in <module 'threading' from 
> '/usr/lib/python2.7/threading.pyc'> ignored
> 
> 
> 
> 
> Second machine:
> 
> Traceback (most recent call last):
>   File "/usr/local/bin/duplicity", line 1509, in <module>
>     with_tempdir(main)
>   File "/usr/local/bin/duplicity", line 1503, in with_tempdir
>     fn()
>   File "/usr/local/bin/duplicity", line 1352, in main
>     do_backup(action)
>   File "/usr/local/bin/duplicity", line 1485, in do_backup
>     incremental_backup(sig_chain)
>   File "/usr/local/bin/duplicity", line 613, in incremental_backup
>     dummy_backup(tarblock_iter)
>   File "/usr/local/bin/duplicity", line 202, in dummy_backup
>     while tarblock_iter.next():
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 518, in next
>     result = self.process(self.input_iter.next())
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 190, in get_delta_iter
>     for new_path, sig_path in collated:
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 281, in collate2iters
>     relem2 = riter2.next()
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 350, in combine_path_iters
>     refresh_triple_list(triple_list)
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 336, in refresh_triple_list
>     new_triple = get_triple(old_triple[1])
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 322, in get_triple
>     path = path_iter_list[iter_index].next()
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 
> 234, in sigtar2path_iter
>     for tarinfo in tf:
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/tarfile.py", line 
> 2470, in next
>     tarinfo = self.tarfile.next()
>   File "/usr/local/lib/python2.7/dist-packages/duplicity/tarfile.py", line 
> 2319, in next
>     self.fileobj.seek(self.offset)
>   File "/usr/lib/python2.7/gzip.py", line 429, in seek
>     self.read(1024)
>   File "/usr/lib/python2.7/gzip.py", line 256, in read
>     self._read(readsize)
>   File "/usr/lib/python2.7/gzip.py", line 307, in _read
>     uncompress = self.decompress.decompress(buf)
> error: Error -3 while decompressing: invalid stored block lengths
> 
> ^Cclose failed in file object destructor:
> IOError: [Errno 32] Broken pipe
> Exception KeyboardInterrupt in <module 'threading' from 
> '/usr/lib/python2.7/threading.pyc'> ignored
> 


the error indicates that at least one of the backup's files are corrupted. you 
seem to use compression but no encryption (right?) as gzip tries to decompress 
and fails.

it shouldn't hang though. please file a bug
 https://bugs.launchpad.net/duplicity

usually this is because the backup repository files are really corrupted. check 
in the -v9 output which file it ries to decompress and afterwards try it 
manually. if you can reproduce the error the files are defective.

you should do a new full then and delete the corrupt backup chain.

consider using "par2+", see manpage
 http://duplicity.nongnu.org/duplicity.1.html

..ede/duply.net

PS: on errors please always send the complete output of a maximum verbosity 
(-v9) duplicity run that reproduces the error. obfuscate text that you deem 
private in that log dump and attach-post it to this mailing list.
you may attach a compressed file if the mailing list file size limit is reached.