Re: [Duplicity-talk] duply: Incrementals not shown

[edgar . soldin]

On 03.01.2017 13:39, Florian Lindner via Duplicity-talk wrote:
> Hey,
> 
> Am 03.01.2017 um 12:08 schrieb edgar.soldin--- via Duplicity-talk:
>> hi Florian, comment inline below
>>
>> On 01.01.2017 15:30, Florian Lindner via Duplicity-talk wrote:
>>> Hello,
>>>
>>> Everybody have a Happy New Year!
>>>
>>> I use duplicity though Duply on Debian Testing and have a weird problem. I 
>>> have two profiles, that differ only by the SOURCE setting:
>>>
>>> address@hidden ~ # grep -v "^#" .duply/home/conf | grep -v "^$"
>>>
>>> GPG_KEY='disabled'
>>> GPG_PW='_GPG_PASSWORD_'
>>> TARGET='scp://address@hidden:2022/Backup/home'
>>> SOURCE='/home'
>>> MAX_AGE=1M
>>> MAX_FULLBKP_AGE=2W
>>> DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
>>> VOLSIZE=25
>>> DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
>>> VERBOSITY=5
>>>
>>> The other profile works perfectly. duply status shows all backups I did.
>>>
>>> However, at the home profile does not show any incremental backups I did 
>>> after the initial full backup.
>>>
>>> I do a successfull incremental backup:
>>>
>> SNIP
>>>
>>>
>>>
>>> So everything looks ok, but:
>>>
>>> # duply home status
>>> Start duply v1.11.3, time is 2017-01-01 15:27:29.
>> SNIP
>>
>>> Synchronizing remote metadata to local cache...
>>> Deleting local 
>>> /root/.cache/duplicity/duply_home/duplicity-inc.20161231T112717Z.to.20170101T140824Z.manifest
>>>  (not authoritative at backend).
>>> Deleting local 
>>> /root/.cache/duplicity/duply_home/duplicity-new-signatures.20161231T112717Z.to.20170101T140824Z.sigtar.gz
>>>  (not authoritative at backend).
>>
>> see here your locally cached inremental get's deleted because duplicity does 
>> not seem to see it on your backend.
>>
>> can you browse there and check if the files really ended up there? try 
>> switching to the pexpect+scp:// backend and see if the error resolves. 
>> paramiko sometimes has issues listing files on the backend
> 
> Files on TARGET:
> 
> address@hidden /home/venus/Backup/home # ll *sig*
> -rw------- 1 venus venus 661868201 31. Dez 17:21 
> duplicity-full-signatures.20161231T112717Z.sigtar.gz
> -rw------- 1 venus venus  11327438 31. Dez 17:36 
> duplicity-new-signatures.20161231T112717Z.to.20161231T163047Z.sigtar.gz
> -rw------- 1 venus venus  11345015 31. Dez 17:45 
> duplicity-new-signatures.20161231T112717Z.to.20161231T163906Z.sigtar.gz
> -rw------- 1 venus venus  11763366  1. Jan 02:11 
> duplicity-new-signatures.20161231T112717Z.to.20170101T010504Z.sigtar.gz
> -rw------- 1 venus venus  12878011  1. Jan 15:14 
> duplicity-new-signatures.20161231T112717Z.to.20170101T140824Z.sigtar.gz
> -rw------- 1 venus venus  13268770  1. Jan 20:20 
> duplicity-new-signatures.20161231T112717Z.to.20170101T191037Z.sigtar.gz
> -rw------- 1 venus venus  13271921  1. Jan 20:32 
> duplicity-new-signatures.20161231T112717Z.to.20170101T192657Z.sigtar.gz
> -rw------- 1 venus venus  13526360  2. Jan 02:11 
> duplicity-new-signatures.20161231T112717Z.to.20170102T010504Z.sigtar.gz
> -rw------- 1 venus venus  14738759  3. Jan 02:13 
> duplicity-new-signatures.20161231T112717Z.to.20170103T010511Z.sigtar.gz
> -rw------- 1 venus venus  15035322  3. Jan 13:26 
> duplicity-new-signatures.20161231T112717Z.to.20170103T121953Z.sigtar.gz
> 
> The ones in the night are from the backup cronjob, the last one 13:26 is from 
> the backup I just did. So everything looks in order (for me).
> 
> Trying pexpect+scp:
> 
> 
> # duply home status
> [...]
> Synchronizing remote metadata to local cache...
> Copying duplicity-inc.20161231T112717Z.to.20161231T163047Z.manifest to local 
> cache.
> [...]
> Copying 
> duplicity-new-signatures.20161231T112717Z.to.20161231T163047Z.sigtar.gz to 
> local cache.
> [...]
> Copying 
> duplicity-new-signatures.20161231T112717Z.to.20170103T010511Z.sigtar.gz to 
> local cache.
> Warning, found the following local orphaned signature files:
> duplicity-new-signatures.20161231T112717Z.to.20170101T192657Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170101T010504Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170101T140824Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170101T191037Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170103T121953Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20161231T163047Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20161231T163906Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170102T010504Z.sigtar.gz
> Warning, found the following remote orphaned signature files:
> duplicity-new-signatures.20161231T112717Z.to.20161231T163906Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170101T010504Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170101T140824Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170101T191037Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170101T192657Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170102T010504Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170103T010511Z.sigtar.gz
> duplicity-new-signatures.20161231T112717Z.to.20170103T121953Z.sigtar.gz
> Warning, found signatures but no corresponding backup files
> Last full backup date: Sat Dec 31 12:27:17 2016
> Collection Status
> -----------------
> Connecting with backend: BackendWrapper
> Archive dir: /root/.cache/duplicity/duply_home
> 
> Found 1 secondary backup chain.
> Secondary chain 1 of 1:
> -------------------------
> Chain start time: Sat Dec 31 12:27:17 2016
> Chain end time: Tue Jan  3 13:19:53 2017
> Number of contained backup sets: 2
> Total number of contained volumes: 1282
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Sat Dec 31 12:27:17 2016              1280
>          Incremental         Tue Jan  3 13:19:53 2017                 2
> -------------------------
> 
> Ok, I think this is ok, because the incremental backups didn't see the 
> previous incremental backup. duply home cleanup --force fixed that, except
> Warning, found signatures but no corresponding backup files. I will do a new 
> complete backup later.

yeah, better start clean. and do verifies from time to time. i cannot stress 
that enough.

> So I assume everything is working fine now.

don't assume, verify ;)

> Do you have any idea why only one profile is affected while the other one, 
> same settings, same TARGET is not troublesome?

me neither, but you can have an eye out for max. verbosity messages like this

"
Extracting backup chains from list of files: 
[u'duplicity-full.20161022T121330Z.manifest.gpg', 
u'duplicity-full.20161022T121330Z.vol100.difftar.gpg', 
u'duplicity-full.20161022T121330Z.vol101.difftar.gpg', 
u'duplicity-full.20161022T121330Z.vol102.difftar.gpg', 
u'duplicity-full.20161022T121330Z.vol103.difftar.gpg',
SNIP
full.20161022T121330Z.vol169.difftar.gpg', 
u'duplicity-full.20161022T121330Z.vol16.difftar.gpg', 
u'duplicity-full.20161022T121330Z.vol170.difftar.gpg', 
u'duplicity-full.20161022T121330Z.vol171.difftar.gpg', 
u'duplicity-full.20161022T1213']
"

it's one long line but the end entry seems to be cut, which it is essentially. 
consider joining the paramiko bug report mentioned on
  https://bugs.launchpad.net/duplicity/+bug/1265765

maybe when enough people complain someone from paramiko will have a look at it?
 
> Anyway, big thanks to you!

nP.. ede/duply.net