Re: [Duplicity-talk] Duplicity + asymmetric encryption

[edgar . soldin]

On 09.09.2015 00:03, Marko Božiković wrote:
> On 08.09.2015. 10:37, address@hidden wrote:
>> On 08.09.2015 00:56, Marko Božiković wrote:
>>> On 07.09.2015. 12:08, address@hidden wrote:
>>>> On 07.09.2015 00:24, Marko Božiković wrote:
>>>>> On 03.08.2015. 3:31, address@hidden wrote:
>>>>>>
>>>>>>
>>>>>> On August 2, 2015 11:41:31 PM GMT+02:00, "Marko Božiković" 
>>>>>> <address@hidden> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I've started playing with Duplicity for my cloud backup needs and I
>>>>>>> have a few
>>>>>>> questions regarding asymmetric encryption with gpg. Apologies for a
>>>>>>> long mail,
>>>>>>> but I'm confused by Duplicity's behaviour.
>>>>>>>
>>>>>>> I'm using Debian Jessie, duply 1.9.2, duplicity 0.7.3 and gpg 2.1.6
>>>>>>> (these are
>>>>>>> installed from testing/experimental repos)
>>>>>>>
>>>>>>> I've created a symbolic link gpg->gpg2 in order to force duplicity to
>>>>>>> use gpg2.
>>>>>>>
>>>>>>> I have a gpg2 keyring setup with a master 4096 bit key, three 2048 bit
>>>>>>> subkeys
>>>>>>> (for signing, encryption and authentication) for everyday use that are
>>>>>>> stored
>>>>>>> on a smart card and a newly created 2048 bit (S, E) subkey for
>>>>>>> duplicity backup.
>>>>>>>
>>>>>>> I keep my secret master key offline. The keyring on my laptop only
>>>>>>> contains
>>>>>>> the master public key and stubs for secret subkeys that live on the
>>>>>>> smartcard.
>>>>>>> The keyring on my backup machine only contains the public master key
>>>>>>> and the
>>>>>>> duplicity backup subkey.
>>>>>>>
>>>>>>> Here are the relevant bits from my duply config (I've replaced my
>>>>>>> backup key's
>>>>>>> fingerprint with YYYYYYYY:
>>>>>>>
>>>>>>> GPG_KEY='YYYYYYYY'
>>>>>>> GPG_OPTS='--pinentry-mode loopback'
>>>>>>> VOLSIZE=250
>>>>>>> DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
>>>>>>> DUPL_PARAMS="$DUPL_PARAMS --file-prefix-archive=backup-data-
>>>>>>> --file-prefix-manifest=includes- --file-prefix-signature=signatures-"
>>>>>>>
>>>>>>> Gpg 2.1 always uses gpg-agent for secret key access and it doesn't set
>>>>>>> GPG_AGENT_INFO env. variable. This is the output I get when I run duply
>>>>>>> with
>>>>>>> no GPG_AGENT_INFO set (YYYYYYYY is my backup key's fingerprint and
>>>>>>> XXXXXXXX is
>>>>>>> my master key fingerprint)
>>>>>>>
>>>>>>> -------------------------------
>>>>>>> -- BEGIN no GPG_AGENT_INFO set
>>>>>>> -------------------------------
>>>>>>> Start duply v1.9.2, time is 2015-08-02 22:02:48.
>>>>>>> Using profile '/home/bozho/.duply/photo'.
>>>>>>> Using installed duplicity version 0.7.03, python 2.7.10, gpg 2.1.6
>>>>>>> (Home:
>>>>>>> ~/.gnupg), awk 'mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan',
>>>>>>> grep
>>>>>>> '(GNU grep) 2.20', bash '4.3.30(1)-release (x86_64-pc-linux-gnu)'.
>>>>>>> Autoset found secret key of first GPG_KEY entry 'YYYYYYYY' for signing.
>>>>>>> Cannot use gpg-agent. GPG_AGENT_INFO not set.
>>>>>>> Checking TEMP_DIR '/tmp' is a folder (OK)
>>>>>>> Checking TEMP_DIR '/tmp' is writable (OK)
>>>>>>> Enter passphrase: <MY SECRET KEY PASSWORD>
>>>>>>> Test - Encrypt to 'YYYYYYYY' & Sign with 'YYYYYYYY' (OK)
>>>>>>> Test - Decrypt (OK)
>>>>>>> Test - Compare (OK)
>>>>>>> Cleanup - Delete '/tmp/duply.1629.1438549369_*'(OK)
>>>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>>>
>>>>>>> WARNING:
>>>>>>>
>>>>>>> Backup failed.
>>>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No
>>>>>>> passphrase
>>>>>>> given - skipped
>>>>>>>
>>>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>>>
>>>>>>> WARNING:
>>>>>>>
>>>>>>> Backup failed.
>>>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No
>>>>>>> passphrase
>>>>>>> given - skipped
>>>>>>>
>>>>>>>
>>>>>>> --- Start running command PRE at 22:03:14.018 ---
>>>>>>> Skipping n/a script '/home/bozho/.duply/photo/pre'.
>>>>>>> --- Finished state OK at 22:03:14.122 - Runtime 00:00:00.103 ---
>>>>>>>
>>>>>>> --- Start running command BKP at 22:03:14.135 ---
>>>>>>> Reading globbing filelist /home/bozho/.duply/photo/exclude
>>>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>>>> Last full backup date: none
>>>>>>> GnuPG passphrase: <ANY PASSPHRASE>
>>>>>>> GnuPG passphrase for signing key: <MY SECRET KEY PASSWORD>
>>>>>>> No signatures found, switching to full backup.
>>>>>>> -------------------------------
>>>>>>> -- END no GPG_AGENT_INFO set
>>>>>>> -------------------------------
>>>>>>>
>>>>>>>
>>>>>>> Note that I get asked a passphrase three times:
>>>>>>>
>>>>>>> - "Enter passphrase", which requires my secret backup subkey password -
>>>>>>> why is
>>>>>>> that needed here?
>>>>>>>
>>>>>>> - "GnuPG passphrase:", which can be any password - I'm guessing that's
>>>>>>> used as
>>>>>>> the session key that actually encrypts the data and then gets encrypted
>>>>>>> by my
>>>>>>> public subkey. Why is this necessary? In OpenPGP, the session key
>>>>>>> can/should
>>>>>>> be a long random key - the recipient doesn't need to know it.
>>>>>>>
>>>>>>> - "GnuPG passphrase for signing key" - ok, this one is for signing,
>>>>>>> since it's
>>>>>>> enabled and I'm using the same key.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Now, if I restart gpg-agent in order to reset its caching and actually
>>>>>>> export
>>>>>>> GPG_AGENT_INFO, I get a slightly different output:
>>>>>>>
>>>>>>> -------------------------------
>>>>>>> -- BEGIN GPG_AGENT_INFO set
>>>>>>> -------------------------------
>>>>>>> Start duply v1.9.2, time is 2015-08-02 22:12:42.
>>>>>>> Using profile '/home/bozho/.duply/photo'.
>>>>>>> Using installed duplicity version 0.7.03, python 2.7.10, gpg 2.1.6
>>>>>>> (Home:
>>>>>>> ~/.gnupg), awk 'mawk 1.3.3 Nov 1996, Copyright(C) Michael D. Brennan',
>>>>>>> grep
>>>>>>> '(GNU grep) 2.20', bash '4.3.30(1)-release (x86_64-pc-linux-gnu)'.
>>>>>>> Autoset found secret key of first GPG_KEY entry 'YYYYYYYY' for signing.
>>>>>>> Autoenable use of gpg-agent. GPG_PW or GPG_PW_SIGN (enc != sign key)
>>>>>>> not set.
>>>>>>> Checking TEMP_DIR '/tmp' is a folder (OK)
>>>>>>> Checking TEMP_DIR '/tmp' is writable (OK)
>>>>>>> Enter passphrase: <MY SECRET KEY PASSWORD>
>>>>>>> Test - Encrypt to 'YYYYYYYY' & Sign with 'YYYYYYYY' (OK)
>>>>>>> Test - Decrypt (OK)
>>>>>>> Test - Compare (OK)
>>>>>>> Cleanup - Delete '/tmp/duply.4740.1438549962_*'(OK)
>>>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>>>
>>>>>>> WARNING:
>>>>>>>
>>>>>>> Backup failed.
>>>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No
>>>>>>> passphrase
>>>>>>> given - skipped
>>>>>>>
>>>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>>>
>>>>>>> WARNING:
>>>>>>>
>>>>>>> Backup failed.
>>>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No
>>>>>>> passphrase
>>>>>>> given - skipped
>>>>>>>
>>>>>>>
>>>>>>> --- Start running command PRE at 22:13:22.870 ---
>>>>>>> Skipping n/a script '/home/bozho/.duply/photo/pre'.
>>>>>>> --- Finished state OK at 22:13:22.956 - Runtime 00:00:00.085 ---
>>>>>>>
>>>>>>> --- Start running command BKP at 22:13:22.966 ---
>>>>>>> Reading globbing filelist /home/bozho/.duply/photo/exclude
>>>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>>>> Last full backup date: none
>>>>>>> No signatures found, switching to full backup.
>>>>>>> -------------------------------
>>>>>>> -- END GPG_AGENT_INFO set
>>>>>>> -------------------------------
>>>>>>>
>>>>>>> Notice that I get asked for a password only once, at the beginning
>>>>>>> ("Enter
>>>>>>> passphrase") Why is that, and is the session key being randomly
>>>>>>> generated in
>>>>>>> this case or is my secret subkey password used as a session key (this
>>>>>>> is a bad
>>>>>>> idea)
>>>>>>>
>>>>>>> Thank you,
>>>>>>
>>>>>>
>>>>>> probably 
>>>>>>  https://sourceforge.net/p/ftplicity/feature-requests/36/
>>>>>>
>>>>>> try setting --use-agent manually in conf.
>>>>>>
>>>>>> ..ede/duply.net
>>>>>>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> Sorry for a late reply.
>>>>>
>>>>> I tried using your suggestion and added --use-agent to DUPL_PARAMS, but it
>>>>> didn't help.
>>>>>
>>>>> I've tried with and without setting GPG_AGENT_INFO variable, I get the 
>>>>> same
>>>>> output after being asked for the key password:
>>>>>
>>>>>
>>>>>
>>>>> Start duply v1.9.2, time is 2015-09-05 22:45:50.
>>>>> Using profile '/home/bozho/.duply/photo'.
>>>>> Using installed duplicity version 0.7.03, python 2.7.10, gpg 2.1.6 (Home:
>>>>> ~/.gnupg), awk 'mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan', 
>>>>> grep
>>>>> '(GNU grep) 2.21', bash '4.3.39(1)-release (x86_64-pc-linux-gnu)'.
>>>>> Autoset found secret key of first GPG_KEY entry 'YYYYYYYY' for signing.
>>>>> Autoenable use of gpg-agent. GPG_PW or GPG_PW_SIGN (enc != sign key) not 
>>>>> set.
>>>>> Checking TEMP_DIR '/tmp' is a folder (OK)
>>>>> Checking TEMP_DIR '/tmp' is writable (OK)
>>>>> Test - Encrypt to 'YYYYYYYY' & Sign with 'YYYYYYYY' (OK)
>>>>> Test - Decrypt (OK)
>>>>> Test - Compare (OK)
>>>>> Cleanup - Delete '/tmp/duply.3034.1441489550_*'(OK)
>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>
>>>>> WARNING:
>>>>>
>>>>> Backup failed.
>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No passphrase
>>>>> given - skipped
>>>>>
>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>
>>>>> WARNING:
>>>>>
>>>>> Backup failed.
>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No passphrase
>>>>> given - skipped
>>>>>
>>>>>
>>>>> --- Start running command PRE at 22:45:54.331 ---
>>>>> Skipping n/a script '/home/bozho/.duply/photo/pre'.
>>>>> --- Finished state OK at 22:45:54.410 - Runtime 00:00:00.079 ---
>>>>>
>>>>> --- Start running command BKP at 22:45:54.419 ---
>>>>> Reading globbing filelist /home/bozho/.duply/photo/exclude
>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>> Last full backup date: none
>>>>> No signatures found, switching to full backup.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> However, the backup seems to proceed normally...
>>>>>
>>>>> Thank you,
>>>>>
>>>>
>>>> can i see your duply conf file? obfuscate private strings in it beforehand.
>>>
>>> Here you go.
>>>
>>> Thank you for persevering with this :)
>>>
>>> Just as a note (I don't remember if I've mentioned this before), I moved my
>>> /usr/bin/gpg to /usr/bin/gpg1 and created /usr/bin/gpg symlink to gpg2.
>>>
>>>
>>
>> i notice you do not set the gpg passphrase. this is the classic way to 
>> provide it, the other being gpg-agent. 
>>
>> for testing purposes please try and set GPG_PW and see if it works as 
>> expected. don't be confused that you need the gpg-agent parameters 
>> (allow-loopback-pinentry in your .gnupg/gpg-agent.conf and 
>> GPG_OPTS='--pinentry-mode loopback') for that to work as gpg-agent is now 
>> (since gpg2.1) the "unified" passphrase input interface to gpg2.1.
> 
> Ok, if I set GPG_PW, I get an output with no errors, but duplicity exports my
> keys to files: gpgkey.YYYYYYYY.pub.asc and gpgkey.YYYYYYYY.sec.asc.
> 
>> alternatively you can try the gpg-agent route. therefor you remove the 
>> GPG_OPTS conf param and add DUPL_PARAMS="$DUPL_PARAMS --use-agent". i didn't 
>> come around to implement a senseful route to deal with the gpg2.1 changes so 
>> for for duply.
> 
> If I remove GPG_OPTS (loopback pin entry), I also get no errors and the keys
> are exported to files.
> 

so both setups w/ and w/o gpg-agent work for you now?

> Why is there a need to have keys exported in files? They are protected with a
> password, but is there a practical need for them to be in files, rather than
> requested from the agent when needed?

it's a convenience feature since the early duply days. this way your profile 
folder contains everything necessary to restore your backup on a different 
machine.

> 
>> another thought. not sure that the gpg symlinking does something weird here. 
>> please use duplicity 0.7.04 with the new --gpg-binary parameter.
>>
>> as per why and when the passphrases/keys are needed let's move these 
>> questions to after we solved your current issue.
>>
>> btw. you could always use gpg2.0-- and it should work right out of the box. 
>> why do you insist on gpg2.1?
> 
> Eh, no strong reason. gpg 2.1 has better ssh and security card handling on
> Windows. I also like the exclusive use of gpg-agent for key handling. And
> since I started to use GPG after years of inactivity, I thought I would go
> with the latest version :-)
> 

makes sense.

..ede/duply.net