Re: [rdiff-backup-users] Failing on 11GB file - other large files working

[Ben Escoto]

>>>>> "Ty!" == ty  <Ty!>
>>>>> wrote the following on Fri, 18 Jul 2003 12:10:26 -0600

  Ty!> We've been doing nightly backups for a couple of weeks now
  Ty!> without any problems, and then a user throws another kink at
  Ty!> us.  I've attached a trace below of the entire backup run, but
  Ty!> the file it is failing on is an 11GB file, of which about 5GB
  Ty!> shows up on the backup mirror.

  Ty!> We regularly handle files larger than 2GB, so I thought we were
  Ty!> done with the large file problems, and we run with
  Ty!> --no-compression as we still have Python 2.2.1 and are limited
  Ty!> to 2GB zip files if we don't disable compression.

  Ty!> I'm wondering if there may still be a 32 bit offset index that
  Ty!> is used when building the delta.  Granted that I know very
  Ty!> little about the inner workings of rdiff-backup (or librsync),
  Ty!> so that's just a guess.

  Ty!> Any ideas on how to get large files like this (this file will
  Ty!> have very large block-level differences) to get backed up
  Ty!> successfully?

I was just trying to test this.  rdiff-backup worked OK when diffing a
12GB file when there were no changes, but the diff was only about
50MB.  Then I tried diffing two totally different (both constructed
from /dev/urandom) 12GB files.

But I had to terminate---extrapolating from my experience it would
take literally a week straight to finish.  I'm surprised you let the
process complete before killing it.  At least on my system (Athlon
2000, 7200rpm ide HD) whether rdiff-backup can construct an 11GB diff
is immaterial because it would take impractically long.

I never realized how slow rdiff-backup could be in cases like this.
Hopefully it will help for the rdiff block size to be set as a
function of file size.  Or it could be my python interface, I will
have to check.


-- 
Ben Escoto

pgpBKJTX27NOZ.pgp
Description: PGP signature