[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[rdiff-backup-users] debugging weird memory error
From: |
Rolf Braun |
Subject: |
[rdiff-backup-users] debugging weird memory error |
Date: |
Thu, 26 Oct 2006 06:08:19 -0400 |
Recently, a system on which I administer backups remotely has not been
able to complete a run of rdiff-backup (the data set is 500 GB but the
backup is completing less than 15% before failing). This is the case
even after I remove the entire directory and retry the backup; in
fact, it started after a server upgrade in which the RAID array was
replaced and the backups were to be restored from the original (there
are also off-site backups). In fact, we have never been able to get a
full backup since the array was replaced. What appears to happen is
that the backup chugs along fine for a time, with some significant
memory usage mainly on the server but no obvious leaks. At some point
the backup starts getting slow and then it hangs altogether for a
while before dying with the memory error listed below. Running "top"
shows less memory usage on the client than the server until around
when the error happens, then the usage balloons on the client while
the server process appears to be idle. It takes about the same amount
of time each try for the backup to fail, and excluding the directory
where it normally fails simply results in it failing elsewhere. A line
from top output on the client:
PID UID REG RPRVT RSHRD RSIZE VPRVT VSIZE TH PRT TIME %CPU
24908 0 4765 1.25G+ 972K 1.07G+ 1.28G+ 1.28G+ 1 27 2h08m 90.4
This error occurred both with our on-site version (hacked to sort the
metadata case insensitive) and with stock 1.1.5, deleting all metadata
in between tests. The only differences between a couple months of
error-free backups and these latest attempts are that the disk array
on the server was replaced and the server OS was upgraded to a new
point release of Mac OS X Tiger, although some other work may have
been done that I'm not aware of. There is no indication that any
hardware failure is occurring and there is plenty of disk space
available on the server.
The following error is when trying to do a backup run with stock
rdiff-backup 1.1.5, all metadata (the rdiff-backup-data directory)
removed in advance. Any help figuring out what is going on here would
be appreciated, as I've worked with the software and hacked around
issues we've had for a little while, but this has me stumped.
Exception '' raised of class 'exceptions.MemoryError':
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/robust.py",
line 32, in check_common_error
try: return function(*args)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rpath.py",
line 878, in listdir
return self.conn.os.listdir(self.path)
Sending back exception of type exceptions.MemoryError:
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 335, in answer_request
result = apply(eval(request.function_string), argument_list)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 483, in readfromid
if length is None: return cls.vfiles[id].read()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 302, in read
if not self.addtobuffer(): break
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 325, in addtobuffer
try: currentobj = self.iter.next()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rorpiter.py",
line 342, in next
next_elem = self.iter.next()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/selection.py",
line 132, in Iterate_fast
try: rpath, val = diryield_stack[-1].next()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/selection.py",
line 118, in diryield
for filename in self.listdir(rpath):
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/selection.py",
line 180, in listdir
dir_listing = robust.check_common_error(error_handler, dir_rp.listdir)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/robust.py",
line 32, in check_common_error
try: return function(*args)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rpath.py",
line 878, in listdir
return self.conn.os.listdir(self.path)
Sending back exception of type exceptions.MemoryError:
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 335, in answer_request
result = apply(eval(request.function_string), argument_list)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 483, in readfromid
if length is None: return cls.vfiles[id].read()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 302, in read
if not self.addtobuffer(): break
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 325, in addtobuffer
try: currentobj = self.iter.next()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py",
line 166, in get_sigs
for src_rorp, dest_rorp in cls.CCPP:
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py",
line 306, in next
source_rorp, dest_rorp = self.iter.next()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rorpiter.py",
line 103, in Collate2Iters
for relem1 in riter1:
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 382, in next
while not type: type, data = self._get()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 414, in _get
if not self.buf: self.buf += self.file.read()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 516, in read
return self.connection.VirtualFile.readfromid(self.id, length)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 448, in __call__
return apply(self.connection.reval, (self.name,) + args)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 370, in reval
if isinstance(result, Exception): raise result
Sending back exception of type exceptions.MemoryError:
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 335, in answer_request
result = apply(eval(request.function_string), argument_list)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 483, in readfromid
if length is None: return cls.vfiles[id].read()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 302, in read
if not self.addtobuffer(): break
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 325, in addtobuffer
try: currentobj = self.iter.next()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py",
line 103, in get_diffs
for dest_sig in dest_sigiter:
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 384, in next
elif type == "r": return self.get_rorp(data)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 394, in get_rorp
rorp.setfile(self.get_file())
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 399, in get_file
type, data = self._get()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 414, in _get
if not self.buf: self.buf += self.file.read()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 516, in read
return self.connection.VirtualFile.readfromid(self.id, length)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 448, in __call__
return apply(self.connection.reval, (self.name,) + args)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 370, in reval
if isinstance(result, Exception): raise result
Sending back exception of type exceptions.MemoryError:
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 335, in answer_request
result = apply(eval(request.function_string), argument_list)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py",
line 216, in patch
for diff in rorpiter.FillInIter(source_diffiter, dest_rpath):
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rorpiter.py",
line 177, in FillInIter
for rp in rpiter:
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 382, in next
while not type: type, data = self._get()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py",
line 414, in _get
if not self.buf: self.buf += self.file.read()
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 516, in read
return self.connection.VirtualFile.readfromid(self.id, length)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 448, in __call__
return apply(self.connection.reval, (self.name,) + args)
File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py",
line 370, in reval
if isinstance(result, Exception): raise result
- [rdiff-backup-users] debugging weird memory error,
Rolf Braun <=