[rdiff-backup-users] Failing on 11GB file - other large files working

[Ty! Boyack]

We've been doing nightly backups for a couple of weeks now without any
problems, and then a user throws another kink at us.  I've attached a
trace below of the entire backup run, but the file it is failing on is
an 11GB file, of which about 5GB shows up on the backup mirror.

We regularly handle files larger than 2GB, so I thought we were done
with the large file problems, and we run with --no-compression as we
still have Python 2.2.1 and are limited to 2GB zip files if we don't
disable compression.  

I'm wondering if there may still be a 32 bit offset index that is used
when building the delta.  Granted that I know very little about the
inner workings of rdiff-backup (or librsync), so that's just a guess.

Any ideas on how to get large files like this (this file will have
very large block-level differences) to get backed up successfully?

Thanks,

-Ty!

(rdiff-backup 0.11.4 + patches from Ben, libsync 0.9.5.1, Python
2.2.1, Linux Redhat 8.0 w/ kernel 2.4.21)


------------------------------------------------------------------
Command:
rdiff-backup -v9 --no-compression /export/nrel-disk01-vga/hobbs.main
/export/nrel-disk01-vga/BACKUP/hobbs.main

Output:
Thu Jul 17 11:22:46 2003  Touching mirror marker
/export/nrel-disk01-vga/BACKUP/hobbs.main/rdiff-backup-data/current_mirror.2003-07-17T11:22:46-06:00.data
Thu Jul 17 11:22:46 2003  Touching
/export/nrel-disk01-vga/BACKUP/hobbs.main/rdiff-backup-data/current_mirror.2003-07-17T11:22:46-06:00.data
Thu Jul 17 11:22:53 2003  Getting signature of hobbs
backup/backup2.bkf
Thu Jul 17 11:22:53 2003  Getting delta of
/export/nrel-disk01-vga/hobbs.main/hobbs backup/backup2.bkf with
signature hobbs backup/backup2.bkf
Thu Jul 17 11:25:42 2003  Processing changed file .
Thu Jul 17 11:25:42 2003  Incrementing mirror file
/export/nrel-disk01-vga/BACKUP/hobbs.main
Thu Jul 17 11:25:42 2003  Touching
/export/nrel-disk01-vga/BACKUP/hobbs.main/rdiff-backup-data/increments.2003-07-17T10:16:22-06:00.dir
Thu Jul 17 11:25:42 2003  Copying attributes from () to
/export/nrel-disk01-vga/BACKUP/hobbs.main/rdiff-backup-data/increments.2003-07-17T10:16:22-06:00.dir
Thu Jul 17 11:25:42 2003  Setting time of
/export/nrel-disk01-vga/BACKUP/hobbs.main/rdiff-backup-data/increments.2003-07-17T10:16:22-06:00.dir
to 1058454062
Thu Jul 17 11:25:42 2003  Processing changed file hobbs backup
Thu Jul 17 11:25:42 2003  Incrementing mirror file
/export/nrel-disk01-vga/BACKUP/hobbs.main/hobbs backup
Thu Jul 17 11:25:42 2003  Touching
/export/nrel-disk01-vga/BACKUP/hobbs.main/rdiff-backup-data/increments/hobbs
backup.2003-07-17T10:16:22-06:00.dir
Thu Jul 17 11:25:42 2003  Copying attributes from ('hobbs backup',) to
/export/nrel-disk01-vga/BACKUP/hobbs.main/rdiff-backup-data/increments/hobbs
backup.2003-07-17T10:16:22-06:00.dir
Thu Jul 17 11:25:42 2003  Setting time of
/export/nrel-disk01-vga/BACKUP/hobbs.main/rdiff-backup-data/increments/hobbs
backup.2003-07-17T10:16:22-06:00.dir to 1050711261
Thu Jul 17 11:25:42 2003  Processing changed file hobbs
backup/backup2.bkf
Thu Jul 17 11:25:42 2003  Writing file object to
/export/nrel-disk01-vga/BACKUP/hobbs.main/hobbs
backup/rdiff-backup.tmp.0
python: ERROR: (rs_file_copy_cb) seek failed: Invalid argument
python: ERROR: (rs_job_complete) patch job failed: IO error
Thu Jul 17 11:30:29 2003  Exception 'librsync error 100 while in patch
cycle' raised of class 'rdiff_backup.librsync.librsyncError':
  File
"/usr/local/rdiff-backup-0.11.4/lib/python2.2/site-packages/rdiff_backup/robust.py",
line 32, in check_common_error
    try: return function(*args)
  File
"/usr/local/rdiff-backup-0.11.4/lib/python2.2/site-packages/rdiff_backup/Rdiff.py",
line 76, in patch_local
    if outrp: outrp.write_from_fileobj(patchfile)
  File
"/usr/local/rdiff-backup-0.11.4/lib/python2.2/site-packages/rdiff_backup/rpath.py",
line 828, in write_from_fileobj
    copyfileobj(fp, outfp)
  File
"/usr/local/rdiff-backup-0.11.4/lib/python2.2/site-packages/rdiff_backup/rpath.py",
line 58, in copyfileobj
    inbuf = inputfp.read(blocksize)
  File
"/usr/local/rdiff-backup-0.11.4/lib/python2.2/site-packages/rdiff_backup/librsync.py",
line 76, in read
    self._add_to_outbuf_once()
  File
"/usr/local/rdiff-backup-0.11.4/lib/python2.2/site-packages/rdiff_backup/librsync.py",
line 87, in _add_to_outbuf_once
    except _librsync.librsyncError, e: raise librsyncError(str(e))

Thu Jul 17 11:30:29 2003  UpdateError hobbs backup/backup2.bkf
librsync error 100 while in patch cycle
Thu Jul 17 11:30:30 2003  Deleting
/export/nrel-disk01-vga/BACKUP/hobbs.main/hobbs
backup/rdiff-backup.tmp.0
Thu Jul 17 11:37:12 2003  Copying attributes from ('hobbs backup',) to
/export/nrel-disk01-vga/BACKUP/hobbs.main/hobbs backup
Thu Jul 17 11:37:12 2003  Setting time of
/export/nrel-disk01-vga/BACKUP/hobbs.main/hobbs backup to 1050711261
Thu Jul 17 11:37:12 2003  Copying attributes from () to
/export/nrel-disk01-vga/BACKUP/hobbs.main
Thu Jul 17 11:37:12 2003  Setting time of
/export/nrel-disk01-vga/BACKUP/hobbs.main to 1058454062
Thu Jul 17 11:37:13 2003  Deleting
/export/nrel-disk01-vga/BACKUP/hobbs.main/rdiff-backup-data/current_mirror.2003-07-17T10:16:22-06:00.data
Thu Jul 17 11:37:13 2003  Cleaning up

-- 
-===========================-
  Ty! Boyack
  NREL Unix Network Manager
  address@hidden
  (970) 491-1186
-===========================-