Re: [Duplicity-talk] gpg key password asked for backup after verify

[Kenneth Loafman]

OK, let me try again:

1) the TemporaryFile() problems in windows and cygwin have been with us for a long while and it has not been fixed in Python, yet.  The first patch that Howie suggested added darwin to that list, and that was removed by the next commit when he found the problem in _librsyncmodule.c.  I don't know how to fix the problem in Windows, so that issue should probably be handled in the man page and README.  People don't normally partition Windows the same way as Linux, so I doubt it's a real problem for most users.

2) In my response, I outlined why we have problems when we don't have the private key.  Perhaps a solution would be to add a file in the cache called duplicity-md5sums.txt that would contain the md5sum of each file in the cache.  As long as that matched, and the cache was unencrypted, we would not need to hit the remote, except to list filenames.  Thoughts?  As to ignoring errors in the sync between local and remote, I can't even imagine the number of ways we'd get into troubles.

...Ken

On Sun, May 28, 2017 at 7:37 AM, <address@hidden> wrote:

wow, a misunderstanding on both points, that's a new record. :)) let me try again

wrt. 1.
i meant the "too many open files" bug solved by Howie. didn't see it in trunk so far. ok found it now
  http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/revision/1230#duplicity/patchdir.py
patchdir.py still contains

499            if sys.platform.startswith(('cygwin', 'windows')):
500                tempfp = os.tmpfile()
501            else:

though, which should be removed as it potentially creates temp files outside the given duplicity temp folder.

wrt. 2.
somewhere in the thread "backup without private key"
  https://bugs.launchpad.net/duplicity/+bug/687295
i explain how this can be achieved, but that it is dangerous and therefor generally not advisable. admittedly i also spitball some alternative solutions, but think now that they are no precondition to remove this behaviour.
my point now is, that we agreed accept incompatibility to previous versions when we are incrementing the minor version eg. 0.7->0.8. why not simply remove the "exception" (dirty hack) that ignores decryption error when synchronizing the archive folder? people can achieve the same functionality safely by using an additional machine key pair thus not needing their supersecret private key.

sunny sundaily greetings ..ede/duply.net

On 24.05.2017 22:04, Kenneth Loafman wrote:
> I looked it up, it was bug #1416344
> <https://bugs.launchpad.net/duplicity/+bug/1416344>, and was fixed for
> 07.02 release on 2015-01-30, a long time ago.
>
> That was going from 0.96 to 1.0.  We're on 2.01 now and all is well.
>
> ...Ken
>
> On Wed, May 24, 2017 at 10:34 AM, Kenneth Loafman <address@hidden>
> wrote:
>
>> ede,
>>
>> 1) librsync is good up thru version 2.x.  Not sure exactly what version of
>> duplicity it was fixed in, but the logs should have the bug number and we
>> can track it from there.  It's been a long while since it was fixed,
>> perhaps all the way back to the last of the 0.6-series.  We still use the
>> smaller signatures and I'm going to leave that as an option going
>> forwards.  The larger signatures will double the size of the sigtar file,
>> and will force a full backup if we go to them.
>>
>> 2) The decryption/encryption issues during restart and incrementals are:
>>
>>    - Do we leave the cache unencrypted?  Without the manifest in plain
>>    form, or without the ability to decrypt it, we can't continue.
>>    - How do we decrypt if the cache is out of sync with the remote?  It

>>    happens and the remote manifest needs to be decrypted.
>>
>> I see no really good reason to encrypt the manifest since it only contains
>> information readily available from the system itself.  However, on the
>> remote it needs to be encrypted, otherwise it could reveal the filenames of
>> the backed up files.  Some folks are sensitive to that.
>>
>> ...Ken
>>
>>
>>
>> On Wed, May 24, 2017 at 9:47 AM, <address@hidden> wrote:
>>
>>> Ken,
>>>
>>> how is the state wrt. librsync patch, didn't see it in the repo so far?
>>>
>>> *and*, below is once again somebody complaining that suddenly a private
>>> key / passphrase is needed. would you agree that we should remove the dirty
>>> workaround described in
>>>   https://bugs.launchpad.net/duplicity/+bug/687295
>>> for duplicity 0.8?
>>>
>>> it really only "works" with english locale and a backup that does not try
>>> to resume. still users risk to do backups where archive dir is not
>>> synchronized, which is bad for their data.
>>>
>>> ..ede
>>>
>>> -------- Forwarded Message --------
>>> Subject:        Re: [Duplicity-talk] gpg key password asked for backup
>>> after verify
>>> Date:   Wed, 24 May 2017 16:14:31 +0200
>>> From:   Raphael Bauduin <address@hidden>
>>> To:     address@hidden
>>>
>>>
>>>
>>>
>>>
>>> On Wed, May 24, 2017 at 1:49 PM, <address@hidden <mailto:
>>> address@hidden>> wrote:
>>>
>>>     whoops, hit send too fast. read on below.
>>>
>>>     On 24.05.2017 13 <tel:24.05.2017%2013>:17, Raphael Bauduin wrote:
>>>     >
>>>     >
>>>     > On Wed, May 24, 2017 at 12:19 PM, edgar.soldin--- via
>>> Duplicity-talk <address@hidden <mailto:address@hidden.
>>> org> <mailto:address@hiddenorg <mailto:address@hiddenorg>>>
>>> wrote:
>>>     >
>>>     >     On 24.05.2017 11 <tel:24.05.2017%2011>
>>> <tel:24.05.2017%2011>:28, Raphael Bauduin via Duplicity-talk wrote:
>>>     >     > Hi,
>>>     >     >
>>>     >     > I had encrypted backups working fine for weeks on a server.
>>> As the encryption uses the public key, it doesn't ask for a password.
>>>     >     >
>>>     >     > Then I did a duplicity verify, which requires the gpg private
>>> key, and asks for a password.
>>>     >     > The verify went fine, but since then the gpg key password is
>>> also asked for backups, preventing the automation.... I'm nearly sure this
>>> is linked
>>>     >     >
>>>     >     > I have removed the duplicity cache in ~/.cache/duplicity, but
>>> to no avail....
>>>     >     >
>>>     >     > Any suggestion?
>>>     >     >
>>>     >
>>>     >     1.
>>>     >     are you using duply?
>>>     >
>>>     >
>>>     > no
>>>     >
>>>     >
>>>     >
>>>     >     2.
>>>     >     what is your backup command line?
>>>     >
>>>     >
>>>     >  LC_ALL=en_US /bin/duplicity   inc --encrypt-key 'XXXX' --exclude
>>> /root/.cache/duplicity --exclude  /home/backups --exclude /home/restore
>>> --exclude /backups  --include /home/sftp --include /etc --include /home
>>> --include /root --exclude '**' / par2+rsync://rsync/duplicity/
>>>  --verbosity debug
>>>     >
>>>     >
>>>     >
>>>     >
>>>     >     3.
>>>     >     what's the language locale of your os?
>>>     >
>>>     >
>>>     > I'm forcing it to en_US, which worked fine.
>>>     >
>>>     > Investigating further, I think I might have deleted the cache
>>> before I did the verify. So not sure which one causes what.
>>>     > I took a look at the code. Here is the code in question asking for
>>> the password when the cache was empty, where I added a print:
>>>     >             if local_missing and (rem_needpass or loc_needpass):
>>>     >                 if decrypt:
>>>     >                     # password for the --encrypt-key
>>>     >                     print "local_missing = %s,--  %s, -- %s" %
>>> (local_missing, rem_needpass, loc_needpass)
>>>     >                     globals.gpg_profile.passphrase =
>>> get_passphrase(1, "sync")
>>>     >
>>>     > local_missing was a set of .sigtar.gpg files, rem_needpass was True
>>> and loc_needpass was False.
>>>     >
>>>     > Now I have done a backup manually (providing the key password), I
>>> have the else clause below asking for the password although the action is
>>> inc:
>>>     >
>>>     >     elif (action == "inc" and
>>>     >           (globals.gpg_profile.recipients or
>>> globals.gpg_profile.hidden_recipients) and not
>>>     >           globals.gpg_profile.sign_key and not globals.restart):
>>>     >         return ""
>>>     >
>>>     >     # Finally, ask the user for the passphrase
>>>     >     else:
>>>     >         print "action = "" % action
>>>     >         log.Info(_("PASSPHRASE variable not set, asking user."))
>>>     >         use_cache = True
>>>     >
>>>     >
>>>     > globals.gpg_profile.recipients is my encryption key id,
>>> globals.gpg_profile.sign_key is None, but globals.restart=
>>> <__main__.Restart instance at 0x13a8518>
>>>     >
>>>     > So it seems that the globals.restart is set and makes the code skip
>>> the action == "inc" part.
>>>     >
>>>     > Any idea what the problem might be?
>>>     >
>>>     > Thanks
>>>     >
>>>
>>>     ok, your backup is restarting. restarting happens when a backup was
>>> interrupted. restarting _needs_ to decrypt some information from the
>>> backend, which can only be done w/ priv key and passphrase of course.
>>>
>>>
>>> Just confirm that's what I had. Making a full successful backup makes it
>>> run fine without password prompt on subsequent runs.
>>>
>>> Thanks a lot for your help!
>>>
>>>
>>>
>>>
>>>
>>>     what you ran into here is essentially this bug
>>>       https://bugs.launchpad.net/duplicity/+bug/687295 <
>>> https://bugs.launchpad.net/duplicity/+bug/687295>
>>>
>>>     consider using two key pairs in the future. duplicity using gpg can
>>> encrypt to multiple keys. place
>>>
>>>     A. a sec/pub key for the box
>>>     B. your own pub key
>>>
>>>     in your keyring. then backup against both keys and optionally use the
>>> machine key to sign your backups. this way the box can decrypt when needed
>>> w/o needing your very secret personal private key.
>>>
>>>     ..ede/duply.net <http://duply.net>
>>>
>>>
>>>
>>>
>>> --
>>> Web database: http://www.myowndb.com
>>> Free Software Developers Meeting: http://www.fosdem.org
>>>
>>
>>
>