Re: [Duplicity-talk] Duplicity + asymmetric encryption

[Marko Božiković]

On 08.09.2015. 10:37, address@hidden wrote:
> On 08.09.2015 00:56, Marko Božiković wrote:
>> On 07.09.2015. 12:08, address@hidden wrote:
>>> On 07.09.2015 00:24, Marko Božiković wrote:
>>>> On 03.08.2015. 3:31, address@hidden wrote:
>>>>>
>>>>>
>>>>> On August 2, 2015 11:41:31 PM GMT+02:00, "Marko Božiković" 
>>>>> <address@hidden> wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I've started playing with Duplicity for my cloud backup needs and I
>>>>>> have a few
>>>>>> questions regarding asymmetric encryption with gpg. Apologies for a
>>>>>> long mail,
>>>>>> but I'm confused by Duplicity's behaviour.
>>>>>>
>>>>>> I'm using Debian Jessie, duply 1.9.2, duplicity 0.7.3 and gpg 2.1.6
>>>>>> (these are
>>>>>> installed from testing/experimental repos)
>>>>>>
>>>>>> I've created a symbolic link gpg->gpg2 in order to force duplicity to
>>>>>> use gpg2.
>>>>>>
>>>>>> I have a gpg2 keyring setup with a master 4096 bit key, three 2048 bit
>>>>>> subkeys
>>>>>> (for signing, encryption and authentication) for everyday use that are
>>>>>> stored
>>>>>> on a smart card and a newly created 2048 bit (S, E) subkey for
>>>>>> duplicity backup.
>>>>>>
>>>>>> I keep my secret master key offline. The keyring on my laptop only
>>>>>> contains
>>>>>> the master public key and stubs for secret subkeys that live on the
>>>>>> smartcard.
>>>>>> The keyring on my backup machine only contains the public master key
>>>>>> and the
>>>>>> duplicity backup subkey.
>>>>>>
>>>>>> Here are the relevant bits from my duply config (I've replaced my
>>>>>> backup key's
>>>>>> fingerprint with YYYYYYYY:
>>>>>>
>>>>>> GPG_KEY='YYYYYYYY'
>>>>>> GPG_OPTS='--pinentry-mode loopback'
>>>>>> VOLSIZE=250
>>>>>> DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
>>>>>> DUPL_PARAMS="$DUPL_PARAMS --file-prefix-archive=backup-data-
>>>>>> --file-prefix-manifest=includes- --file-prefix-signature=signatures-"
>>>>>>
>>>>>> Gpg 2.1 always uses gpg-agent for secret key access and it doesn't set
>>>>>> GPG_AGENT_INFO env. variable. This is the output I get when I run duply
>>>>>> with
>>>>>> no GPG_AGENT_INFO set (YYYYYYYY is my backup key's fingerprint and
>>>>>> XXXXXXXX is
>>>>>> my master key fingerprint)
>>>>>>
>>>>>> -------------------------------
>>>>>> -- BEGIN no GPG_AGENT_INFO set
>>>>>> -------------------------------
>>>>>> Start duply v1.9.2, time is 2015-08-02 22:02:48.
>>>>>> Using profile '/home/bozho/.duply/photo'.
>>>>>> Using installed duplicity version 0.7.03, python 2.7.10, gpg 2.1.6
>>>>>> (Home:
>>>>>> ~/.gnupg), awk 'mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan',
>>>>>> grep
>>>>>> '(GNU grep) 2.20', bash '4.3.30(1)-release (x86_64-pc-linux-gnu)'.
>>>>>> Autoset found secret key of first GPG_KEY entry 'YYYYYYYY' for signing.
>>>>>> Cannot use gpg-agent. GPG_AGENT_INFO not set.
>>>>>> Checking TEMP_DIR '/tmp' is a folder (OK)
>>>>>> Checking TEMP_DIR '/tmp' is writable (OK)
>>>>>> Enter passphrase: <MY SECRET KEY PASSWORD>
>>>>>> Test - Encrypt to 'YYYYYYYY' & Sign with 'YYYYYYYY' (OK)
>>>>>> Test - Decrypt (OK)
>>>>>> Test - Compare (OK)
>>>>>> Cleanup - Delete '/tmp/duply.1629.1438549369_*'(OK)
>>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>>
>>>>>> WARNING:
>>>>>>
>>>>>> Backup failed.
>>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No
>>>>>> passphrase
>>>>>> given - skipped
>>>>>>
>>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>>
>>>>>> WARNING:
>>>>>>
>>>>>> Backup failed.
>>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No
>>>>>> passphrase
>>>>>> given - skipped
>>>>>>
>>>>>>
>>>>>> --- Start running command PRE at 22:03:14.018 ---
>>>>>> Skipping n/a script '/home/bozho/.duply/photo/pre'.
>>>>>> --- Finished state OK at 22:03:14.122 - Runtime 00:00:00.103 ---
>>>>>>
>>>>>> --- Start running command BKP at 22:03:14.135 ---
>>>>>> Reading globbing filelist /home/bozho/.duply/photo/exclude
>>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>>> Last full backup date: none
>>>>>> GnuPG passphrase: <ANY PASSPHRASE>
>>>>>> GnuPG passphrase for signing key: <MY SECRET KEY PASSWORD>
>>>>>> No signatures found, switching to full backup.
>>>>>> -------------------------------
>>>>>> -- END no GPG_AGENT_INFO set
>>>>>> -------------------------------
>>>>>>
>>>>>>
>>>>>> Note that I get asked a passphrase three times:
>>>>>>
>>>>>> - "Enter passphrase", which requires my secret backup subkey password -
>>>>>> why is
>>>>>> that needed here?
>>>>>>
>>>>>> - "GnuPG passphrase:", which can be any password - I'm guessing that's
>>>>>> used as
>>>>>> the session key that actually encrypts the data and then gets encrypted
>>>>>> by my
>>>>>> public subkey. Why is this necessary? In OpenPGP, the session key
>>>>>> can/should
>>>>>> be a long random key - the recipient doesn't need to know it.
>>>>>>
>>>>>> - "GnuPG passphrase for signing key" - ok, this one is for signing,
>>>>>> since it's
>>>>>> enabled and I'm using the same key.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Now, if I restart gpg-agent in order to reset its caching and actually
>>>>>> export
>>>>>> GPG_AGENT_INFO, I get a slightly different output:
>>>>>>
>>>>>> -------------------------------
>>>>>> -- BEGIN GPG_AGENT_INFO set
>>>>>> -------------------------------
>>>>>> Start duply v1.9.2, time is 2015-08-02 22:12:42.
>>>>>> Using profile '/home/bozho/.duply/photo'.
>>>>>> Using installed duplicity version 0.7.03, python 2.7.10, gpg 2.1.6
>>>>>> (Home:
>>>>>> ~/.gnupg), awk 'mawk 1.3.3 Nov 1996, Copyright(C) Michael D. Brennan',
>>>>>> grep
>>>>>> '(GNU grep) 2.20', bash '4.3.30(1)-release (x86_64-pc-linux-gnu)'.
>>>>>> Autoset found secret key of first GPG_KEY entry 'YYYYYYYY' for signing.
>>>>>> Autoenable use of gpg-agent. GPG_PW or GPG_PW_SIGN (enc != sign key)
>>>>>> not set.
>>>>>> Checking TEMP_DIR '/tmp' is a folder (OK)
>>>>>> Checking TEMP_DIR '/tmp' is writable (OK)
>>>>>> Enter passphrase: <MY SECRET KEY PASSWORD>
>>>>>> Test - Encrypt to 'YYYYYYYY' & Sign with 'YYYYYYYY' (OK)
>>>>>> Test - Decrypt (OK)
>>>>>> Test - Compare (OK)
>>>>>> Cleanup - Delete '/tmp/duply.4740.1438549962_*'(OK)
>>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>>
>>>>>> WARNING:
>>>>>>
>>>>>> Backup failed.
>>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No
>>>>>> passphrase
>>>>>> given - skipped
>>>>>>
>>>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>>>
>>>>>> WARNING:
>>>>>>
>>>>>> Backup failed.
>>>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No
>>>>>> passphrase
>>>>>> given - skipped
>>>>>>
>>>>>>
>>>>>> --- Start running command PRE at 22:13:22.870 ---
>>>>>> Skipping n/a script '/home/bozho/.duply/photo/pre'.
>>>>>> --- Finished state OK at 22:13:22.956 - Runtime 00:00:00.085 ---
>>>>>>
>>>>>> --- Start running command BKP at 22:13:22.966 ---
>>>>>> Reading globbing filelist /home/bozho/.duply/photo/exclude
>>>>>> Local and Remote metadata are synchronized, no sync needed.
>>>>>> Last full backup date: none
>>>>>> No signatures found, switching to full backup.
>>>>>> -------------------------------
>>>>>> -- END GPG_AGENT_INFO set
>>>>>> -------------------------------
>>>>>>
>>>>>> Notice that I get asked for a password only once, at the beginning
>>>>>> ("Enter
>>>>>> passphrase") Why is that, and is the session key being randomly
>>>>>> generated in
>>>>>> this case or is my secret subkey password used as a session key (this
>>>>>> is a bad
>>>>>> idea)
>>>>>>
>>>>>> Thank you,
>>>>>
>>>>>
>>>>> probably 
>>>>>  https://sourceforge.net/p/ftplicity/feature-requests/36/
>>>>>
>>>>> try setting --use-agent manually in conf.
>>>>>
>>>>> ..ede/duply.net
>>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>> Sorry for a late reply.
>>>>
>>>> I tried using your suggestion and added --use-agent to DUPL_PARAMS, but it
>>>> didn't help.
>>>>
>>>> I've tried with and without setting GPG_AGENT_INFO variable, I get the same
>>>> output after being asked for the key password:
>>>>
>>>>
>>>>
>>>> Start duply v1.9.2, time is 2015-09-05 22:45:50.
>>>> Using profile '/home/bozho/.duply/photo'.
>>>> Using installed duplicity version 0.7.03, python 2.7.10, gpg 2.1.6 (Home:
>>>> ~/.gnupg), awk 'mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan', 
>>>> grep
>>>> '(GNU grep) 2.21', bash '4.3.39(1)-release (x86_64-pc-linux-gnu)'.
>>>> Autoset found secret key of first GPG_KEY entry 'YYYYYYYY' for signing.
>>>> Autoenable use of gpg-agent. GPG_PW or GPG_PW_SIGN (enc != sign key) not 
>>>> set.
>>>> Checking TEMP_DIR '/tmp' is a folder (OK)
>>>> Checking TEMP_DIR '/tmp' is writable (OK)
>>>> Test - Encrypt to 'YYYYYYYY' & Sign with 'YYYYYYYY' (OK)
>>>> Test - Decrypt (OK)
>>>> Test - Compare (OK)
>>>> Cleanup - Delete '/tmp/duply.3034.1441489550_*'(OK)
>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>
>>>> WARNING:
>>>>
>>>> Backup failed.
>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No passphrase
>>>> given - skipped
>>>>
>>>> Export SEC key 'YYYYYYYY' (FAILED)
>>>>
>>>> WARNING:
>>>>
>>>> Backup failed.
>>>> gpg: key XXXXXXXX/YYYYYYYY: error receiving key from agent: No passphrase
>>>> given - skipped
>>>>
>>>>
>>>> --- Start running command PRE at 22:45:54.331 ---
>>>> Skipping n/a script '/home/bozho/.duply/photo/pre'.
>>>> --- Finished state OK at 22:45:54.410 - Runtime 00:00:00.079 ---
>>>>
>>>> --- Start running command BKP at 22:45:54.419 ---
>>>> Reading globbing filelist /home/bozho/.duply/photo/exclude
>>>> Local and Remote metadata are synchronized, no sync needed.
>>>> Last full backup date: none
>>>> No signatures found, switching to full backup.
>>>>
>>>>
>>>>
>>>>
>>>> However, the backup seems to proceed normally...
>>>>
>>>> Thank you,
>>>>
>>>
>>> can i see your duply conf file? obfuscate private strings in it beforehand.
>>
>> Here you go.
>>
>> Thank you for persevering with this :)
>>
>> Just as a note (I don't remember if I've mentioned this before), I moved my
>> /usr/bin/gpg to /usr/bin/gpg1 and created /usr/bin/gpg symlink to gpg2.
>>
>>
> 
> i notice you do not set the gpg passphrase. this is the classic way to 
> provide it, the other being gpg-agent. 
> 
> for testing purposes please try and set GPG_PW and see if it works as 
> expected. don't be confused that you need the gpg-agent parameters 
> (allow-loopback-pinentry in your .gnupg/gpg-agent.conf and 
> GPG_OPTS='--pinentry-mode loopback') for that to work as gpg-agent is now 
> (since gpg2.1) the "unified" passphrase input interface to gpg2.1.

Ok, if I set GPG_PW, I get an output with no errors, but duplicity exports my
keys to files: gpgkey.YYYYYYYY.pub.asc and gpgkey.YYYYYYYY.sec.asc.

> alternatively you can try the gpg-agent route. therefor you remove the 
> GPG_OPTS conf param and add DUPL_PARAMS="$DUPL_PARAMS --use-agent". i didn't 
> come around to implement a senseful route to deal with the gpg2.1 changes so 
> for for duply.

If I remove GPG_OPTS (loopback pin entry), I also get no errors and the keys
are exported to files.

Why is there a need to have keys exported in files? They are protected with a
password, but is there a practical need for them to be in files, rather than
requested from the agent when needed?


> another thought. not sure that the gpg symlinking does something weird here. 
> please use duplicity 0.7.04 with the new --gpg-binary parameter.
> 
> as per why and when the passphrases/keys are needed let's move these 
> questions to after we solved your current issue.
> 
> btw. you could always use gpg2.0-- and it should work right out of the box. 
> why do you insist on gpg2.1?

Eh, no strong reason. gpg 2.1 has better ssh and security card handling on
Windows. I also like the exclusive use of gpg-agent for key handling. And
since I started to use GPG after years of inactivity, I thought I would go
with the latest version :-)

Cheers,
-- 
Marko