|
From: | Kenneth Loafman |
Subject: | Re: [Duplicity-talk] Unexpected end of data (2) |
Date: | Mon, 5 Sep 2016 15:32:57 -0500 |
Ok, this could indeed be it. The file reads:2147402630 Sep 2 15:08 duplicity-full-signatures.20160727T201540Z.sigtar.gz
Which is pretty close to 2 GiB exactly. The file on the server (which is a .gpg btw), is slightly larger (2.1 GiB).
The filesystem can handle large files (ext4+ecryptfs) but the CPU is 32 bit, so the bottleneck is there is probably in one of the commands to decrypt the sigtar file. Questions:
1) Is there a simple way for duplicity to split the tarfile?
Probably not, so:2) How does this file "grow"? Solely by number of files? In that case, is it a solution to split the backup in several separate, smaller runs?
Thanks for the quick response!
On Mon, 2016-09-05 at 11:17 -0500, Kenneth Loafman wrote:Usually this means that the input tarfile has been truncated.
This could be the sigtar file in this case. Is it possible that Synology has an upper limit on file size and is silently truncating the signature file? I'm not familiar with your environment, so we're going to have to guess for a bit.
If you saved the log file, please paste it to pastebin.com and reply with the link they supply.
On Sun, Sep 4, 2016 at 4:13 PM, S. Breedveld via Duplicity-talk <address@hidden> wrote:
Dear all,
I want to append to this previous thread, raised by Adam M.:
I am coping with the exact same problem. I initially followed the suggestion to remove the latest incremental backups, but this did not solve the problem. Now I have done a new, full backup, which completed well, but the problem reappears with the first incremental step.
I am running duplicity on a Synology (DS111, running armv5tel CPU), based on the build by SynoCommunity ( https://synocommunity.com/ ). To keep things as simple as possible, I initiated a "local" backup to a remote filesystem, mounted locally (sshfs).
Of course I checked the basics, i.e. free disk space, permissions, etc., and does not seem to be the issue.
Command line:/usr/local/bin/duplicity --exclude-regexp '.*[Nn][Oo][Bb][Aa][Cc][Kk][Uu][Pp].*' --archive-dir=/volume1/homes/s ebastiaan/duplicity/ --tempdir=/volume1/homes/sebas tiaan/duplicity/ --verbosity 9 --asynchronous-upload /volume1/prive/ file:///volume1/homes/sebastia an/remotemountpoint/ Start output:Using archive dir: /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c88 30badf29 Using backup name: 7fd35f7d883ae08eaaf74c8830badf29 Import of duplicity.backends.azurebackend Succeeded Import of duplicity.backends.botobackend SucceededImport of duplicity.backends.cfbackend SucceededImport of duplicity.backends.copycombackend Succeeded Import of duplicity.backends.dpbxbackend SucceededImport of duplicity.backends.gdocsbackend Succeeded Import of duplicity.backends.giobackend SucceededImport of duplicity.backends.hsibackend SucceededImport of duplicity.backends.hubicbackend Succeeded Import of duplicity.backends.imapbackend SucceededImport of duplicity.backends.lftpbackend SucceededImport of duplicity.backends.localbackend Succeeded Import of duplicity.backends.megabackend SucceededImport of duplicity.backends.multibackend Succeeded Import of duplicity.backends.ncftpbackend Succeeded Import of duplicity.backends.onedrivebackend Succeeded Import of duplicity.backends.par2backend SucceededImport of duplicity.backends.pydrivebackend Succeeded Import of duplicity.backends.rsyncbackend Succeeded Import of duplicity.backends.ssh_paramiko_backend Succeeded Import of duplicity.backends.ssh_pexpect_backend Succeeded Import of duplicity.backends.swiftbackend Succeeded Import of duplicity.backends.sxbackend SucceededImport of duplicity.backends.tahoebackend Succeeded Import of duplicity.backends.webdavbackend Succeeded Main action: inc============================================================ ==================== duplicity 0.7.05 (September 15, 2015)Args: /usr/local/bin/duplicity --exclude-regexp '.*[Nn][Oo][Bb][Aa][Cc][Kk][Uu][Pp].*' --archive-dir=/volume1/homes/s ebastiaan/duplicity/ --tempdir=/volume1/homes/sebas tiaan/duplicity/ --verbosity 9 --asynchronous-upload /volume1/prive/ file:///volume1/homes/sebastia an/remotemountpoint Linux HomeStore 2.6.32.12 #7393 Thu Jun 2 19:48:48 CST 2016 armv5tel/usr/local/duplicity/env/bin/python 2.7.11 (default, Dec 17 2015, 11:15:51) [GCC 4.6.4]============================================================ ==================== Using temporary directory /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir Registering (mkstemp) temporary file /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir /mkstemp-ynGqb_-1 Temp has 344093282304 available, backup will use approx 60293120.Local and Remote metadata are synchronized, no sync needed.13117 files exist on backend10 files exist in cacheExtracting backup chains from list of files: [u'duplicity-full.20160727T201540Z.vol1690.difftar. gpg'....
Later informative part:File duplicity-full.20160727T201540Z.vol11895.difftar.gpg is part of known set Found backup chain [Wed Jul 27 22:15:40 2016]-[Wed Jul 27 22:15:40 2016]Last full backup date: Wed Jul 27 22:15:40 2016Collection Status-----------------Connecting with backend: BackendWrapperArchive dir: /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c88 30badf29 Found 0 secondary backup chains.Found primary backup chain with matching signature chain:-------------------------Chain start time: Wed Jul 27 22:15:40 2016Chain end time: Wed Jul 27 22:15:40 2016Number of contained backup sets: 1Total number of contained volumes: 13115Type of backup set:Time: Num volumes: Full Wed Jul 27 22:15:40 2016 13115 -------------------------No orphaned or incomplete backup sets found.Registering (mktemp) temporary file /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir /mktemp-5ZMA51-2 Deleting /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir /mktemp-5ZMA51-2 Forgetting temporary file /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir /mktemp-5ZMA51-2 Using temporary directory /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c88 30badf29/duplicity-Xz3QdP- tempdir Registering (mktemp) temporary file /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c88 30badf29/duplicity-Xz3QdP- tempdir/mktemp-dpdZXg-1 Using temporary directory /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c88 30badf29/duplicity-O4M9YM- tempdir Registering (mktemp) temporary file /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c88 30badf29/duplicity-O4M9YM- tempdir/mktemp-_Hw4en-1 AsyncScheduler: instantiating at concurrency 1Registering (mktemp) temporary file /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir /mktemp-jED6Ib-3 Selecting /volume1/priveComparing . and .Selection: examining path /volume1/prive/#recycleSelection: result: None from function: Regular _expression_: '.*[Nn][Oo][Bb][Aa][Cc][Kk][Uu][Pp].*' Selection: + including fileSelecting /volume1/prive/#recycle...
And then:Selecting /volume1/prive/sebastiaan/Video/07/VIDEO_TS/VTS_01_3.VOB Releasing lockfile <LinkLockFile: '/volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8 830badf29/HomeStore.20136- 1083199126' -- '/volume1/homes/sebastiaan/dup licity/7fd35f7d883ae08eaaf74c8 830badf29/lockfile'> Removing still remembered temporary file /volume1/homes/sebastiaan/duplicity/duplicity-2iR3gQ-tempdir /mktemp-tGnUdl-3 Removing still remembered temporary file /volume1/homes/sebastiaan/duplicity/duplicity-2iR3gQ-tempdir /mkstemp-Ny13R0-1 Traceback (most recent call last):File "/usr/local/bin/duplicity", line 1525, in <module>with_tempdir(main)File "/usr/local/bin/duplicity", line 1519, in with_tempdirfn()File "/usr/local/bin/duplicity", line 1373, in maindo_backup(action)File "/usr/local/bin/duplicity", line 1501, in do_backupincremental_backup(sig_chain) File "/usr/local/bin/duplicity", line 658, in incremental_backupglobals.backend)File "/usr/local/bin/duplicity", line 423, in write_multivolglobals.gpg_profile, globals.volsize)File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplic ity/gpg.py", line 340, in GPGWriteFile data = "">File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplic ity/diffdir.py", line 523, in next result = self.process(self.input_iter.next()) File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplic ity/diffdir.py", line 195, in get_delta_iter for new_path, sig_path in collated:File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplic ity/diffdir.py", line 286, in collate2iters relem2 = riter2.next()File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplic ity/diffdir.py", line 354, in combine_path_iters refresh_triple_list(triple_list) File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplic ity/diffdir.py", line 341, in refresh_triple_list new_triple = get_triple(old_triple[1])File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplic ity/diffdir.py", line 327, in get_triple path = path_iter_list[iter_index].next() File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplic ity/diffdir.py", line 239, in sigtar2path_iter for tarinfo in tf:File "/usr/local/python/lib/python2.7/tarfile.py", line 2507, in next tarinfo = self.tarfile.next()File "/usr/local/python/lib/python2.7/tarfile.py", line 2349, in next raise ReadError("unexpected end of data")ReadError: unexpected end of dataRemoving still remembered temporary file /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c88 30badf29/duplicity-JIQdoh- tempdir/mktemp-SRkW8S-1 Removing still remembered temporary file /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c88 30badf29/duplicity-w85sue- tempdir/mktemp-WxgULW-1
I have done a few runs, and sometimes it hangs at this file, sometimes the next one...
Any hints here?
Sincerely,Sebastiaan
_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
--
Dr.ir. Sebastiaan Breedveld (PhD)
Scientific Researcher
Erasmus MC - Cancer Institute
Department of Radiation Oncology
P.O. Box 5201, 3008 AERotterdam, The Netherlands
Visiting address: GroeneHilledijk 301, 3075 EA Rotterdam, The Netherlands, room Gs-20
E address@hidden | T +31 10 70 42 693
www.erasmusmc.nl/kankerinstituut
http://sebastiaanbreedveld.nl/
[Prev in Thread] | Current Thread | [Next in Thread] |