[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] OSError: [Errno 24] Too many open files
From: |
Kenneth Loafman |
Subject: |
Re: [Duplicity-talk] OSError: [Errno 24] Too many open files |
Date: |
Tue, 12 May 2009 19:00:36 -0500 |
User-agent: |
Thunderbird 2.0.0.21 (X11/20090318) |
No difference, other than my testing was flawed. I finally had to go to
the debugger to stop the process and then count using lsof.
I'm showing 3 file descriptors released with every gpg process after the
waitpid() is completed, so it would take 300+ gpg zombies to overrun the
1024 limit.
This is a problem in the way duplicity and GnuPGInterface are handling
process termination. To fix it, I'll need to add a SIGCHILD signal
handler to GnuPGInterface that will allow the process to terminate when
it ends, rather than making it wait till we get around to checking.
...Ken
Edgar Soldin wrote:
> just curious, could you detect a difference between 32bit & 64bit systems?
> how many incrementals did you need to hit the 1024 ulimit?
>
> @yang: how many incrementals does your backup hold? just to doublecheck
> kens findings.
>
> ..ede
>> I have found the problem and its going to take me a bit to get it fixed.
>>
>> As it works duplicity creates iterators that traverse each of the sig
>> files. The iterators do not close until the process is complete and the
>> files they are using are being held open. The gpg process is held as
>> well since we don't wait for it until we close the file. The zombie
>> processes were just a symptom of the real problem, too many open
>> resources at one time.
>>
>> This hits directly the issue of too long an incremental chain. Each
>> increments sig file is going to have file descriptors open until the
>> backup/incremental/restore completes. The only workaround at this point
>> is to increase the 'ulimit -n' parameter for any long increment chains
>> and perform a full backup on a more frequent basis.
>>
>> I have some ideas about how to clean this up, but need to examine the
>> code a bit closer before committing to a solution.
>>
>> ...Ken
signature.asc
Description: OpenPGP digital signature
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, (continued)
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Ian Barton, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Jeff Singer, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Edgar Soldin, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Andrew Telford, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files,
Kenneth Loafman <=
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, edgar . soldin, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Yang Zhang, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Edgar Soldin, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Yang Zhang, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, edgar . soldin, 2009/05/13
Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/13