Re: [Duplicity-talk] Backup succeeding but not writing manifest / sigtar

[Mark Grandi]

> On Mar 1, 2016, at 9:13 AM, address@hidden wrote:
> 
> On 01.03.2016 16:48, Mark Grandi wrote:
>> 
>>> On Mar 1, 2016, at 5:04 AM, address@hidden <mailto:address@hidden> wrote:
>>> 
>>> On 01.03.2016 05:50, Mark Grandi wrote:
>>>> I am backing up root, because on Mac OS X, almost all of the applications 
>>>> are stored in /Applications , (shared between all users), so I am backing 
>>>> up root, and excluding all of the other directories besides /Users , but 
>>>> that doesn't seem to stop Duplicity from trying to access them anyway, and 
>>>> failing due to not having the read permissions.
>>>> 
>>>> Anyway, I looked into my problem myself since I had a free afternoon, and 
>>>> found myself honestly quite concerned with what some of the code does, or 
>>>> rather doesn't do. For instance, with Par2Backend, if for some reason the 
>>>> par2 command doesn't return 0, then it just silently doesn't create any 
>>>> par2 files without any notice to the user. In LocalBackend, if any of the 
>>>> methods in there that have try/except blocks, in the except blocks, they 
>>>> are either 'pass' or just 'return False', and often the caller doesn't 
>>>> care about the return result anyway, so any errors that occur during 
>>>> moving/copying files are silently discarded and not presented to the user, 
>>>> nor does it invoke any of the retry logic in Duplicity.
>>>> 
>>>> I mentioned the two above things, because I had trouble compiling the par2 
>>>> binary on my Mac, and upon googling, I found a Mac OS X app that is a GUI 
>>>> frontend to par and par2, and it appears that the author created their own 
>>>> fork of par2, that works on mac (and has improvements such as using grand 
>>>> central dispatch, the website is here: 
>>>> https://gp.home.xs4all.nl/Site/MacPAR_deLuxe.html). Anyway, it sees this 
>>>> version of par2 does not like symlinks, and the Par2Backend class uses 
>>>> symlinks when creating the par2 files. Discovering why the par2 files were 
>>>> not being created required me stepping into the debugger, because no 
>>>> errors are logged to the user.
>>>> 
>>>> Once I changed the symlink to a hardlink, and modified the command line 
>>>> being passed to par2 somewhat, I was stuck on the issue that I originally 
>>>> posted:  the difftar files + par2 files were being moved to the correct 
>>>> spot, but not the manifest or sigtar files and their par2 counterparts. 
>>>> Stepping into the debugger, i found that the difftar files are being 
>>>> written with Par2Backend wrapping LocalBackend.put(), but the sigtar and 
>>>> manifest files are wrapping LocalBackend._move(). However, 
>>>> LocalBackend._move() is using Path.rename() instead of Path.move() ! It 
>>>> even says in the docstring for Path.move(): "Like rename but destination 
>>>> may be on different file system".
>>>> 
>>>> So there lies my problem, my backup destination was on a different file 
>>>> system, but due to LocalBackend._move() calling the wrong underlying Path 
>>>> method, and swallowing exceptions and only returning false (that 
>>>> Par2Backend doesn't even check), it was not moving the sigtar and manifest 
>>>> files + par2 files because of "[Errno 18] Cross-device link".
>>>> 
>>>> The patch to fix this is just simply replacing LocalBackend._move() with:
>>>> 
>>>>   def _move(self, source_path, remote_filename):
>>>>       target_path = self.remote_pathdir.append(remote_filename)
>>>>       try:
>>>>           source_path.setdata() # maybe only needed in Par2Backend?
>>>>           source_path.move(target_path)
>>>>           return True
>>>>       except OSError as e:
>>>>           return False
>>>> 
>>>> This does not solve the concerns I mentioned with swallowing exceptions 
>>>> however.
>>>> 
>>> 
>>> hey Mark,
>>> 
>>> well done debugging the issue! i totally agree w/ you that the duplicity 
>>> code is not in perfect shape. thanks for investing the time to find and fix 
>>> your issue.
>>> 
>>> wrt. to the sym-/hardlink par2 issue. as hardlinks are only avail to root 
>>> users, would you mind to have a look if you can come up w/ a solution that 
>>> works w/o links at all for the par2 backend? if you provide a patch i will 
>>> create a branch that fixes your issues that can be merged into the next 
>>> release by Ken.
>>> 
>>> ..ede
>>> 
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden <mailto:address@hidden>
>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>> 
>> Well, the only reason I changed os.symlink to os.link was because i'm not 
>> using the 'official' par2 I guess you could say, but par2SL? Either way, I 
>> think it should be a separate backend, or the par2 backend should handle 
>> both versions of par2.
> 
> w/o looking into the par2 backend. why is it actually necessary to create 
> links at all?

par2 files have the filename of the file that they contain the recovery 
information for, so from the comments from Par2Backend, it seems that the name 
could possibly be different (possibly because the backend that Par2Backend is 
wrapping has some special naming scheme), so it creates a symlink so that it 
will create the par2 file for the correct filename, and then after that it can 
be renamed/moved to where the wrapped backend wants.
> 
>> Either way, I'm also confused by the statement that hardlinks are only 
>> available to root users? This certainly isn't the case for Mac OS X as well 
>> as linux (at least the version of Ubuntu that I run on my server). Perhaps 
>> you are thinking of windows and NTFS junction points, which require some 
>> policy enabled for that user (which is the bane of every VCS software that 
>> supports symlinks).
> 
> just using hard links to sparsely (pun intended).. with linux you have to be 
> root to hardlink folders, but files work fine - you are correct.
> 
>> 
>> HOWEVER, I do agree that it is probably easiest to just refactor it to not 
>> need a symlink or a hardlink by just doing some renaming magic, so i'll work 
>> on that. I use bazaar myself so i'll create a branch on launchpad for both 
>> this (and the other issue that i posted in a different thread on this 
>> mailing list)
> 
> just looked a bit further into the move issue and path.py:Path.move(...) 
> seems to copy/del all of the time. wouldn't it make sense performancewise to 
> add os.rename() here and only copy/del if the rename failed?
> i guess the author of the code you fixed had the performance advantage of 
> rename in mind when he wrote that code.

I could bzr blame the code and see if there are any insights, but yeah, a lot 
of the path code seems to be very buggy and uses a lot of assumptions. For 
example, the new bug that I found is that when moving the file, it assumes it 
can change the ownership of the files after moving them, but that is returning 
a '[Errno 1] Operation not permitted:' error , possibly because I am mounting a 
drive through AFP, and AFP is doing funky stuff, since while its mounted, its 
not necessarily a real physical drive and seems to not support all the normal 
FS operations...*scratches head*

> wish we had more contributors like you!!.. thanks in advance.. ede
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc
Description: Message signed with OpenPGP using GPGMail