[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] OSError: [Errno 24] Too many open files
From: |
Edgar Soldin |
Subject: |
Re: [Duplicity-talk] OSError: [Errno 24] Too many open files |
Date: |
Tue, 12 May 2009 22:41:08 +0200 |
User-agent: |
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.17) Gecko/20080914 Thunderbird/2.0.0.17 Mnenhy/0.7.5.0 |
just curious, could you detect a difference between 32bit & 64bit systems?
how many incrementals did you need to hit the 1024 ulimit?
@yang: how many incrementals does your backup hold? just to doublecheck
kens findings.
..ede
> I have found the problem and its going to take me a bit to get it fixed.
>
> As it works duplicity creates iterators that traverse each of the sig
> files. The iterators do not close until the process is complete and the
> files they are using are being held open. The gpg process is held as
> well since we don't wait for it until we close the file. The zombie
> processes were just a symptom of the real problem, too many open
> resources at one time.
>
> This hits directly the issue of too long an incremental chain. Each
> increments sig file is going to have file descriptors open until the
> backup/incremental/restore completes. The only workaround at this point
> is to increase the 'ulimit -n' parameter for any long increment chains
> and perform a full backup on a more frequent basis.
>
> I have some ideas about how to clean this up, but need to examine the
> code a bit closer before committing to a solution.
>
> ...Ken
>
> Kenneth Loafman wrote:
>
>> As far as I can tell, its just lots of incrementals. duplicity has to
>> open and process each signature file and, in some cases, the gpg process
>> is not being closed completely causing a zombie process.
>>
>> This "seems" to be worse on a 64bit system, but I have been able to
>> partially reproduce it on a 32bit. The most zombie processes I've been
>> able to see are 5-7 and that's with about 60 incrementals. Even then,
>> all of them go away quickly.
>>
>> ...Ken
>>
>> Jeff Singer wrote:
>>
>>> I have a 64bit Ubuntu 9.04 system here that I'd be happy to attempt to
>>> reproduce it over the weekend. Is it basically a backup that involves
>>> lots of incrementals with many volumes that's causing this behavior?
>>>
>>> On Tue, May 12, 2009 at 1:51 PM, Ian Barton <address@hidden
>>> <mailto:address@hidden>> wrote:
>>>
>>> I have got a couple of 64 bit Ubuntu 9.1 (Jaunty) computers. I can
>>> try running a test on my system if you tell me what you would like
>>> me to do.
>>>
>>> FWIW my main computer for running is Debian Lenny 64bit and I have
>>> never seen the problem on it.
>>>
>>> Ian.
>>>
>>>
>>> Thanks for the help!
>>>
>>> I'm not sure at this point. I built a 64bit Ubuntu 9.04 system
>>> and it
>>> seems to be working fine. Debian may be different, but not much.
>>>
>>> As to what's broke, it could even be at the system level, but my
>>> bet is
>>> on gpg.
>>>
>>> ...Ken
>>>
>>> address@hidden <mailto:address@hidden> wrote:
>>>
>>> I am gonna install a 64bit debian in a virtual machine and gonna
>>> doublecheck that ... but what do you think is buggy? python,
>>> gpg ?
>>>
>>> ..ede
>>>
>>> The only real change to GnuPGInterface is a shift right
>>> instead of a
>>> shift left when printing the error code after an
>>> exception. He's not
>>> getting any exceptions, so he's not hitting the change.
>>> Other than
>>> that, its the same code from years ago.
>>>
>>> My best guess is still that its a 32-bit versus 64-bit
>>> issue.
>>>
>>> ...Ken
>>>
>>> Edgar Soldin wrote:
>>>
>>>
>>> true they close every time .. the only thing i see
>>> is that the amount of
>>> gpg processes grows with the amount of incrementals ...
>>>
>>> @yang: how many incrementals does your backup hold?
>>>
>>> the other thing is, maybe gpg itself is buggy ..
>>>
>>> @yang: could you try another gpg version?
>>>
>>> and the third thing
>>>
>>> @ken: is it really impossible for 0.5.17 to use an
>>> oldGnuPGInterface
>>> version, that might be laying around on the system?
>>> How could yang look
>>> for it?
>>>
>>> .. 2cents again :) ..ede
>>> --
>>>
>>>
>>> I tried this with tiny volume sizes of 1M over a
>>> 2GB backup (2K
>>> volumes)
>>> and did a count of the gpg processes (live,
>>> defunct) every 2 seconds
>>> and
>>> never had more than 5 (3 live, 2 defunct). This
>>> was on a very fast
>>> link
>>> between local VMs. Tried both backup and
>>> restore so it would test both
>>> directions. By the end, all were closed.
>>>
>>> It's possible to have some defunct while the
>>> task scheduler is
>>> updating,
>>> but Yang is showing hundreds that are holding
>>> files open. That's the
>>> confusing part.
>>>
>>> ...Ken
>>>
>>> Edgar Soldin wrote:
>>>
>>>
>>> i really have to be fast (a window of 10s
>>> let's say) and it helps to
>>> have a slow backup connection with lots of
>>> incrementals... but I
>>> may be
>>> seeing ghosts here. Can it that it is
>>> perfectly normal for
>>> duplicity to
>>> gpg decrypt something per each incremental
>>> but close all processes
>>> just
>>> a bit later? .. just a guess
>>> ..ede
>>>
>>>
>>> I cannot reproduce this on a 32bit
>>> machine at all. I'll be able
>>> to test
>>> on a 64bit machine sometime this week
>>> and, hopefully, find the
>>> problem.
>>>
>>> ...Ken
>>>
>>> address@hidden
>>> <mailto:address@hidden> wrote:
>>>
>>>
>>> i seem to get closer to it ...
>>> the more incrementals (regardless if
>>> file: or ftp: backend) the more
>>> defunct gpg processes appear .. but
>>> dissappear again while the
>>> backup is
>>> still running
>>>
>>> I did
>>> initial backup of 3GB
>>> touched a new test file to have a
>>> change & did a 2nd backup
>>> touched a new test2 file to have a
>>> change & did a 3nd backup
>>> ...
>>>
>>> and since the 3rd backup the defunct
>>> gpg processes keep appearing
>>>
>>> essentially I say: Take any more
>>> than 3 incrementals in size
>>> backup and
>>> do a new incremental and you will
>>> see those zombie gpg processes ..
>>>
>>> while I am not sure that this is
>>> what happens on the ulimit
>>> problem I
>>> thought I should report these
>>> observations ... ede
>>>
>>> PS:
>>>
>>> here a 'ps xf -u user' shortly after
>>> starting the sixth run .. the
>>> number of gpg zombies seems to be
>>> number of sets (full + incr's)
>>> minus 2 ..
>>>
>>> 9122 pts/0 S+ 0:00
>>> \_ /bin/bash
>>>
>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>>> test backup
>>> 19128 pts/0 S+ 0:00
>>> \_ /bin/bash
>>>
>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>>> backup
>>> 19193 pts/0 S+ 0:00
>>> \_ /bin/bash
>>>
>>> /srv/www/user//release/ftplicity_1.4.2/ftplicity
>>> backup
>>> 19194 pts/0 R+ 0:03
>>> \_ /usr/bin/python
>>>
>>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
>>> --verbosity 4
>>> --encrypt-key 00000000--sign-key
>>> 00000000--gpg-options=--always-trust
>>> --volsize 1 --exclude-globbing-filelist
>>> /srv/www/user//.ftplicity/test/excl
>>> 19196 pts/0 SL+ 0:00
>>> \_ gpg
>>> --logger-fd 4
>>> --passphrase-fd 8 --batch --no-tty
>>> --default-key 00000000--recipient
>>> 00000000--no-secmem-warning
>>> --always-trust --encrypt --sign
>>> 19197 pts/0 SL+ 0:00
>>> \_ gpg
>>> --status-fd 6
>>> --passphrase-fd 11 --logger-fd 5
>>> --batch --no-tty --default-key
>>> 00000000--no-secmem-warning
>>> --always-trust --decrypt
>>> 19199 pts/0 Z+ 0:00
>>> \_ [gpg]
>>> <defunct>
>>> 19200 pts/0 Z+ 0:00
>>> \_ [gpg]
>>> <defunct>
>>> 19201 pts/0 Z+ 0:00
>>> \_ [gpg]
>>> <defunct>
>>> 19202 pts/0 Z+ 0:00
>>> \_ [gpg]
>>> <defunct>
>>> 19203 pts/0 SL+ 0:00
>>> \_ gpg
>>> --logger-fd
>>> 23 --passphrase-fd 28 --batch
>>> --no-tty --default-key
>>> 00000000--recipient
>>> 00000000--no-secmem-warning
>>> --always-trust --encrypt --sign
>>>
>>>
>>>
>>>
>>> hmm .. you are right these
>>> processes are zombie (they appear in
>>> top as
>>> such) .. later on the processes
>>> disappear, while duplicity is still
>>> running ..
>>> but I don't feel this is
>>> normal.. or is it?
>>> just to be more sure, I'll try a
>>> 1MB chunk full backup of my 3GB
>>> home
>>> dir .. let's see with how many
>>> zombies I'll end up ...
>>>
>>> interestingly my ulimit =
>>> unlimited ... seems to be default in
>>> the old
>>> SUSE 10.2, btw also on my debian
>>> 5.0 box both 32bit
>>> can't remember to ever have
>>> touched this setting
>>>
>>> ..ede
>>> --
>>>
>>>
>>> These are not open
>>> processes, they are zombie
>>> processes, so no
>>> real
>>> resources taken. That
>>> simplifies it a bit. What
>>> normally
>>> causes this
>>> is the failure of a parent
>>> process to properly retrieve its
>>> exit status,
>>> but this is not the case or
>>> more people would have this
>>> problem.
>>>
>>> As far as I can tell, the
>>> only systems this is
>>> happening on are
>>> 64bit,
>>> and not even all of those.
>>> Some of my systems are
>>> 64bit and
>>> they don't
>>> show this, so I'm wondering
>>> if this could be limited to a
>>> particular
>>> version of GnuPG, or what.
>>>
>>> Edgar, from your note I
>>> could not tell, is this
>>> happening to
>>> you too?
>>>
>>> ...Ken
>>>
>>> address@hidden
>>> <mailto:address@hidden>
>>> wrote:
>>>
>>>
>>>
>>> just my quick
>>> observation .. a simple
>>> incr backup opens and
>>> leaves over
>>> 20 gpg processes open. I
>>> don't think this is
>>> healthy and also can
>>> imagine that this
>>> multiplies if the
>>> volumes get smaller
>>> (mine are
>>> 50MB).
>>>
>>> regards ede
>>>
>>>
>>> address@hidden:~> ps -u
>>> user xf
>>> PID TTY STAT
>>> TIME COMMAND
>>> 32731 ? S
>>> 0:00 sshd: address@hidden/0
>>> 32732 pts/0 Ss
>>> 0:00 \_ -bash
>>> 1167 pts/0 S
>>> 0:00 \_ /bin/bash
>>> /usr/local/bin/ftplicity
>>> profile backup
>>> 1168 pts/0 S
>>> 0:00 | \_ /bin/bash
>>>
>>> /srv/www//release/ftplicity_1.4.2/ftplicity
>>> profile backup
>>> 1174 pts/0 S
>>> 0:00 | \_
>>> /bin/bash
>>>
>>> /srv/www//release/ftplicity_1.4.2/ftplicity
>>> backup
>>> 1239 pts/0 S
>>> 0:00 |
>>> \_ /bin/bash
>>>
>>> /srv/www//release/ftplicity_1.4.2/ftplicity
>>> backup
>>> 1240 pts/0 S
>>> 0:05 |
>>> \_ /usr/bin/python
>>>
>>> /srv/www/user/_apps/duplicity-0.5.10/bin/duplicity
>>> --verbosity 4
>>> --encrypt-key XXXXXXX
>>> --sign-key B59ECD99
>>> --gpg-options=--always-trust
>>> --full-if-older-than 1M
>>> --volsize 50
>>> --exclude-globbing-filelist
>>> /srv/www/...
>>> 1249 pts/0 SL
>>> 0:00 |
>>> \_ gpg
>>> --logger-fd 4
>>> --passphrase-fd 8
>>> --batch --no-tty
>>> --default-key XXXXXXXX
>>> --recipient
>>> XXXXXXXX
>>> --no-secmem-warning
>>> --always-trust --encrypt
>>> --sign
>>> 1265 pts/0 SL
>>> 0:00 |
>>> \_ gpg
>>> --status-fd 6
>>> --passphrase-fd 11
>>> --logger-fd 5 --batch
>>> --no-tty --default-key
>>> XXXXXXXX
>>> --no-secmem-warning
>>> --always-trust --decrypt
>>> 1267 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1272 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1275 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1278 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1282 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1286 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1289 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1292 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1296 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1299 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1302 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1305 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1308 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1311 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1314 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1317 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1320 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1324 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1327 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1330 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1333 pts/0 Z
>>> 0:00 |
>>> \_ [gpg]
>>> <defunct>
>>> 1334 pts/1 Ss+
>>> 0:00 |
>>> \_
>>> /usr/bin/ncftp -u
>>> ******* backup.server.de
>>> <http://backup.server.de>
>>> 1335 pts/0 R+
>>> 0:00 \_ ps -u user xf
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> <mailto:address@hidden>
>>>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> <mailto:address@hidden>
>>>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> <mailto:address@hidden>
>>>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> <mailto:address@hidden>
>>>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> <mailto:address@hidden>
>>>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> <mailto:address@hidden>
>>>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> <mailto:address@hidden>
>>>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> <mailto:address@hidden>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden <mailto:address@hidden>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden <mailto:address@hidden>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden <mailto:address@hidden>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden <mailto:address@hidden>
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> address@hidden
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, (continued)
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Edgar Soldin, 2009/05/11
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, edgar . soldin, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, edgar . soldin, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Yang Zhang, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Ian Barton, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Jeff Singer, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files,
Edgar Soldin <=
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Andrew Telford, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/12
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, edgar . soldin, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Yang Zhang, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Edgar Soldin, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Yang Zhang, 2009/05/13
- Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, edgar . soldin, 2009/05/13
Re: [Duplicity-talk] OSError: [Errno 24] Too many open files, Kenneth Loafman, 2009/05/13