Re: [rdiff-backup-users] Increased resource usage in rdiff-backup 0.12.3

[Luke Mewburn]

On Thu, Aug 28, 2003 at 11:41:37PM -0700, Ben Escoto wrote:
  | >>>>> "LM" == Luke Mewburn <address@hidden>
  | >>>>> wrote the following on Fri, 29 Aug 2003 10:59:13 +1000
  | 
  |   LM> For example, on one day a backup failed with File
  |   LM> "/usr/pkg/lib/python2.2/gzip.py", line 43, in __init__ IOError:
  |   LM> [Errno 24] Too many open files:
  | 
  | rdiff-backup should only have a few files open at a time.  Isn't there
  | some way to list the open files?  (I seem to recall some obsure way
  | but can't remember the specifics.)  Also I think someone running
  | Solaris had this problem a long time ago, but the resolution was
  | inconclusive.

On NetBSD, fstat(1) is available in the base distribution.
lsof(1) is another tool.

I can run fstat -p pid-of-rdiff-backup during a run to examine the
open file descriptors...  If I find anything interesting I'll let you
know.



  |   LM> A "normal" run for that file system only needs ~ 15MB of RAM,
  |   LM> but a --check-destination-dir regression run was using ~ 170MB
  |   LM> of RAM!  This disparity between the normal and regression runs
  |   LM> is astounding.
  | 
  | I will have to check on this.  Was there something unusual about your
  | directory (lots of files, big files, large directories, etc)?

There *were* a couple of large-ish Maildir/ directories; each with
5000-10000 mail files, but I recently added them to the exclude list.
The source file system has about 30GB and 500000 files.


  |   LM> I don't think I had these issues with rdiff-backup 0.10.x.  Is
  |   LM> there any reason as to why rdiff-backup needs much higher
  |   LM> resource limits now?
  | 
  | At least on the destination side (which is where you seemed to have
  | problems), 0.12.x should be less resource intensive since it shouldn't
  | have to recurse the mirror tree.  

That's good to know.


Cheers,
Luke.