[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] webdavs and large sync volume failing
From: |
address@hidden |
Subject: |
Re: [Duplicity-talk] webdavs and large sync volume failing |
Date: |
Sun, 21 Feb 2016 18:30:48 +0100 |
Hi Edgar,
no reason to apologize, thanks for taking this up again.
to your questions:
> A. where did you get your duplicity? did you build it yourself, which
> repository.
first I used the original raspbian duplicity and duply packages (I guess this
is coming from Debian jessie arm branch)
This created the memory problems with the webdavs backend
I then upgraded to latest duplicity/duply directly downloading from the duply
and duplicity project pages and then manually compiling/installing.
> B. how much ram are we talking about in your raspberrypi? (never played
> around w/ it)
The raspberrypi has 512MB ram (of which 434MB are really available) and a 100MB
swap file
The tmp folder is on a harddisk
> 1. webdav & oom-killing
I will re-run a full backup now with the latest duplicity version self
compiled/installed and test if the memory problem re-apears (this will take a
while as the job runs appr 24hours)
I will come back on this
> 2. lftp not listing
I tried to create a new log using your modified lftpbackend.py but it fails
with ccl_cacert_path error output below.
The file /etc/duplicity/cacert.pem is existing.
Surprisingly the error remains even when reverting back to the original
lftpbackend.py.
The webdavs backend is still working ok for list command.
What I did I do:
- replace the lftpbackend.py in the extracted duplicity archive
- installed it via „python setup.py install“
- run the command: duply photo_2010 list
Did I do something wrong when replacing the file?
I am pretty sure this error was not there when I ran the same duply command
with lftpbackend some week back.
——— output when running the duply command ———
duply photo_2010 list > /var/log/duplydebug
Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 1532, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 1526, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 1364, in main
action = commandline.ProcessCommandLine(sys.argv[1:])
File "/usr/local/lib/python2.7/dist-packages/duplicity/commandline.py", line
1108, in ProcessCommandLine
globals.backend = backend.get_backend(args[0])
File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 223,
in get_backend
obj = get_backend_object(url_string)
File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line 209,
in get_backend_object
return factory(pu)
File
"/usr/local/lib/python2.7/dist-packages/duplicity/backends/lftpbackend.py",
line 111, in __init__
if globals.ssl_cacert_path:
AttributeError: 'module' object has no attribute 'ssl_cacert_path'
18:04:55.610 Task 'LIST' failed with exit code '30'.
——— output end ——
Gruss,
Wolle
> Am 19.02.2016 um 12:35 schrieb address@hidden:
>
> Wolle,
>
> sorry, lost you in my email tsunami!
> there are obviously several issues in the mix here. let's try to tackle them
> in order below
>
> first some more general questions?
>
> A. where did you get your duplicity? did you build it yourself, which
> repository.
> B. how much ram are we talking about in your raspberrypi? (never played
> around w/ it)
>
> usually memory issues in the past were caused by maintainers fiddling w/ our
> gpginterface, that's why i am asking where you've got your duplicity from to
> check that.
>
> What you can try.
>
> 1. webdav & oom-killing
>
> run duplicity w/ a command you know that caused the issue in the past and in
> a second terminal observe memory usage and processes. tyr to find out which
> (sub)processes stuff your memory.
> also make sure that /tmp or the folder you gave for temp files is not mounted
> to a in-memory file system.
>
> 2. lftp not listing
>
> lftp backend probably has an issue in the listing code. please run the
> listing again in max. verbosity but backup/replace
> duplicity/backends/lftpbackend.py w/ the copy attached beforehand.
>
> the resulting -v9 log output might tell us why the returned list is empty.
>
> ..ede/duply.net
>
> On 18.02.2016 21:51, address@hidden wrote:
>> Hi,
>> noone an idea what this problem could be or where I could continue trouble
>> shooting?
>>
>> Thanks for help.
>>
>> BR,
>> Wolle
>>
>>
>>> Am 06.02.2016 um 09:09 schrieb address@hidden:
>>>
>>> Hi,
>>>
>>> I have reproduced the error with verbosity 9 and sent the logfiles to
>>> ede/duply.net.
>>> As stated in that mail it looks like this is a memory issue
>>> (/var/log/messages shows that oom-killer kicks in)
>>>
>>> In parallel I have now upgraded my system to
>>> duplicity 0.7.06
>>> duply 1.11.1
>>>
>>> and re-run the backup with backend lftp+webdavs as proposed below. This
>>> works fine without errors
>>>
>>> but
>>>
>>> If I check the status of this backup via the same backend lftp+webdavs it
>>> shows no backup present.
>>> If I change back to webdavs backend for the status command I can see the 1
>>> successfull full backup I created with the lftp+webdavs backend.
>>> I tested the same with a small backup repository, here the backend
>>> lftp+webdavs works fine both for the backup and the status command
>>>
>>> A quick investigation I did shows that the status command for the big
>>> backup (roughly 14GB of photos) has an issue when retrieving the file info
>>> from the webdav server. There is no data returned - in the log below there
>>> should be a long file list after STDOUT: but this is empty.
>>>
>>> ——— snip from log ——
>>> CMD: lftp -c 'source
>>> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd
>>> 'backup/photo/1971-2005/' || exit 0; ls'
>>> Reading results of 'lftp -c 'source
>>> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd
>>> 'backup/photo/1971-2005/' || exit 0; ls''
>>> STDERR:
>>> ---- Resolving host address...
>>> ---- 1 address found: xxxxxxx
>>> ---- Connecting to sd2dav. xxxxxxx port 443
>>> ---- Sending request...
>>> ---> PROPFIND / HTTP/1.1
>>> ---> Host: sd2dav. xxxxxxx
>>> ---> User-Agent: lftp/4.6.0
>>> ---> Accept: */*
>>> ---> Depth: 0
>>> ---> Authorization: Basic xxxxxxx
>>> ---> Connection: keep-alive
>>> --->
>>> Certificate: xxxxxxx
>>> Trusted
>>> <--- HTTP/1.1 207 Multi-Status
>>> <--- Date: Sat, 06 Feb 2016 06:55:09 GMT
>>> <--- Server: Apache
>>> <--- ETag: "xxxxxxx ="
>>> <--- Content-Length: 1857
>>> <--- Content-Type: text/xml; charset="utf-8"
>>> <--- Vary: Accept-Encoding
>>> <--- Keep-Alive: timeout=3, max=100
>>> <--- Connection: Keep-Alive
>>> <---
>>> ---- Receiving body...
>>> ---- Hit EOF
>>> ---- Closing HTTP connection
>>> ---- Connecting to sd2dav. xxxxxxx port 443
>>> ---- Sending request...
>>> ---> PROPFIND /backup/photo/1971-2005/ HTTP/1.1
>>> ---> Host: sd2dav. xxxxxxx
>>> ---> User-Agent: lftp/4.6.0
>>> ---> Accept: */*
>>> ---> Depth: 0
>>> ---> Authorization: Basic xxxxxxx
>>> ---> Connection: keep-alive
>>> --->
>>> Certificate: xxxxxxx
>>> Trusted
>>> <--- HTTP/1.1 207 Multi-Status
>>> <--- Date: Sat, 06 Feb 2016 06:55:19 GMT
>>> <--- Server: Apache
>>> <--- ETag: "xxxxxxx ="
>>> <--- Content-Length: 1368
>>> <--- Content-Type: text/xml; charset="utf-8"
>>> <--- Vary: Accept-Encoding
>>> <--- Keep-Alive: timeout=3, max=100
>>> <--- Connection: Keep-Alive
>>> <---
>>> ---- Receiving body...
>>> ---- Hit EOF
>>> ---- Closing HTTP connection
>>> ---- Connecting to sd2dav. xxxxxxx port 443
>>> ---- Sending request...
>>> ---> GET /backup/photo/1971-2005/ HTTP/1.1
>>> ---> Host: sd2dav. xxxxxxx
>>> ---> User-Agent: lftp/4.6.0
>>> ---> Accept: */*
>>> ---> Authorization: Basic xxxxxxx
>>> ---> Connection: keep-alive
>>> --->
>>> Certificate: xxxxxxx
>>> Trusted
>>> <--- HTTP/1.1 200 OK
>>> <--- Date: Sat, 06 Feb 2016 06:55:20 GMT
>>> <--- Server: Apache
>>> <--- Last-Modified: Mon, 01 Feb 2016 19:44:00 GMT
>>> <--- ETag: "xxxxxxx ="
>>> <--- Accept-Ranges: bytes
>>> <--- Content-Length: 0
>>> <--- Content-Type: text/html; charset="UTF-8"
>>> <--- Vary: Accept-Encoding
>>> <--- Keep-Alive: timeout=3, max=100
>>> <--- Connection: Keep-Alive
>>> <---
>>> ---- Receiving body...
>>> ---- Received all
>>> ---- Closing HTTP connection
>>>
>>> STDOUT:
>>>
>>> Local and Remote metadata are synchronized, no sync needed.
>>>
>>> ——— snip end from log ——
>>>
>>>> Am 02.02.2016 um 21:27 schrieb address@hidden:
>>>>
>>>> On 02.02.2016 21:18, address@hidden wrote:
>>>>> Hi,
>>>>>
>>>>> I am trying to use Duplicity and duply to backup appr 60GB data to
>>>>> webdavs storage.
>>>>>
>>>>> The script is running on a raspberrypi with raspbian jessie
>>>>>
>>>>> Small backups are working.
>>>>> I have tried to run this backup in one go, it failed; I then split into
>>>>> 15GB chunks it failed again with exit code 137
>>>>>
>>>>> Any ideas what to do / where to look for?
>>>>
>>>> can you run duplicity in max. verbosity '-v9' and send me the _complete_
>>>> output privately?
>>>>
>>>>> Is the webdavs access not stable enough for such backup?
>>>>
>>>> should make no difference. the backup is split into volumes of the same
>>>> size anyway.
>>>>
>>>>> Is it maybe a performance issue with the raspberry
>>>>>
>>>>> btw I started the backup script via a cron job, it was then running
>>>>> several hours before it failed.
>>>>>
>>>>
>>>> what's your duplicity version? with the latest duplicity you can install
>>>> lftp and use that as alernative webdav backend via lftp+webdav://
>>>>
>>>> ..ede/duply.net
>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
> <lftpbackend.py>_______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
- [Duplicity-talk] webdavs and large sync volume failing, address@hidden, 2016/02/02
- Re: [Duplicity-talk] webdavs and large sync volume failing, edgar . soldin, 2016/02/02
- Re: [Duplicity-talk] webdavs and large sync volume failing, address@hidden, 2016/02/06
- Re: [Duplicity-talk] webdavs and large sync volume failing, address@hidden, 2016/02/18
- Re: [Duplicity-talk] webdavs and large sync volume failing, edgar . soldin, 2016/02/19
- Re: [Duplicity-talk] webdavs and large sync volume failing,
address@hidden <=
- Re: [Duplicity-talk] webdavs and large sync volume failing, address@hidden, 2016/02/22
- Re: [Duplicity-talk] webdavs and large sync volume failing, edgar . soldin, 2016/02/23
- Re: [Duplicity-talk] webdavs and large sync volume failing, address@hidden, 2016/02/23
- Re: [Duplicity-talk] webdavs and large sync volume failing, edgar . soldin, 2016/02/23
- Re: [Duplicity-talk] webdavs and large sync volume failing, address@hidden, 2016/02/24
- Message not available
- Re: [Duplicity-talk] webdavs and large sync volume failing, edgar . soldin, 2016/02/23