Re: [rdiff-backup-users] IOError: [Errno 75] Value too large for defined data type

[Tore Ericsson]

Hi Bruno,

Since I am using FAT32 to backup from Ubuntu EXT3 (EXT4) with some success I 
will comment without much hard knowledge hoping that some more people will 
have more to say.

I suppose you want to use NTFS for backup in order to at the same time produce 
MSWin compatible files. If not, just reformat your USB into EXT3 or 4. (and my 
further comments are superfluos)

Since there seems to be less experience using rdiff-backup on MSWin and since 
FAT32 is a less complex and hopefully more secure filesystem than NTFS for 
Linux use I chose to backup to a FAT32 disk in order to simultaneously get 
MSWin-compatible files.

I do employ some precautions regarding file naming and some special care at 
rdiff-backup use. I have earlier posted questions touching these problems that 
may be similar to yours (on FAT32, though) but may be avoided, so far, by 
using 

        rdiff-backup --no-hard-links \
                --override-chars-to-quote '"*/:<>?\\|;' \
                $source $fat32usb

but I do not understand fully how it functions - se further my earlier post at 

http://lists.nongnu.org/archive/html/rdiff-backup-users/2011-04/msg00013.html

that yet nobody did respond to!

Hope somebody knows more on this topic!

Tore

-----------------------------
Från Bruno Basilio <address@hidden>
2011-05-10 17.21.16 [[rdiff-backup-users] IOError: [Errno 75] Value too large 
for defined data type]: 
> Hi,
> 
> I'm new at using rdiff-backup but already fascinated with the simple
> principle. Hope it will not stop to be maintained.
> 
> I'm running Ubuntu 10.04, installed the rdiff-backup 1.2.8 and tried
> to backup 80GB of pictures and movies using to USB external drives
> with NTFS, at around 30GB, it stopped with following error:
> 
> <<<
> $  rdiff-backup -v9 --print-statistics --no-hard-links
> /media/BBasilioHomeBackup/fotos/ /media/Elements/rdiff-backup/fotos/
> Mon May  9 22:23:34 2011  Using rdiff-backup version 1.2.8
> Mon May  9 22:23:34 2011  Found interrupted initial backup. Removing...
> Mon May  9 22:23:34 2011  Determining if directory contains files:
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments
> Mon May  9 22:23:34 2011  Determining if directory contains files:
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments/2009
> Mon May  9 22:23:34 2011  Determining if directory contains files:
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments/2009/2009_0
> 8_08 Mon May  9 22:23:34 2011  Determining if directory contains files:
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments/2009/2009_
> 08_08/2009_08_08 - Bruno
> Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments
> Mon May  9 22:23:34 2011  Removing directory
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments
> Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/backup.log
> Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/chars_to_quote
> Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/file_statistics.2011-0
> 5-09T22:20:28+01:00.data.gz Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/mirror_metadata.2011-0
> 5-09T22:20:28+01:00.snapshot.gz Mon May  9 22:23:34 2011  POSIX ACLs not
> supported by filesystem at /media/BBasilioHomeBackup/fotos
> Mon May  9 22:23:34 2011  Unable to import win32security module. Windows
> ACLs not supported by filesystem at /media/BBasilioHomeBackup/fotos
> Mon May  9 22:23:34 2011  escape_dos_devices not required by
> filesystem at /media/BBasilioHomeBackup/fotos
> Mon May  9 22:23:34 2011
> -----------------------------------------------------------------
> Detected abilities for source (read only) file system:
>   Access control lists                         Off
>   Extended attributes                          On
>   Windows access control lists                 Off
>   Case sensitivity                             On
>   Escape DOS devices                           Off
>   Escape trailing spaces                       Off
>   Mac OS X style resource forks                Off
>   Mac OS X Finder information                  Off
> -----------------------------------------------------------------
> Mon May  9 22:23:34 2011  Making directory
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0
> Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/5-_
> a.snapshot.gz
> Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/5-_
> a.snapshot.gz
> Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/uni
> ᄉ Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/uni
> ᄉ Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/:\"
> Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/:\"
> Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/A
> Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/A
> Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/foo
> Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/foo
> Mon May  9 22:23:34 2011  Making directory
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hl
> Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/har
> dlinked_file1 Mon May  9 22:23:34 2011  Hard linking
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hl/
> hardlinked_file2 to
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/ha
> rdlinked_file1 Mon May  9 22:23:34 2011  POSIX ACLs not supported by
> filesystem at
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0
> Mon May  9 22:23:34 2011  Unable to import win32security module. Windows
> ACLs not supported by filesystem at
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0
> Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/dir
> _inc_check Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/dir
> _inc_check Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/reg
> file Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/reg
> file Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hig
> h_perms_file Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hig
> h_perms_dir Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hig
> h_perms_file Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/hig
> h_perms_dir Mon May  9 22:23:34 2011  Touching
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/sym
> linked_file1 Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/sym
> linked_file2 Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0/sym
> linked_file1 Mon May  9 22:23:34 2011  escape_dos_devices not required by
> filesystem at
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0
> Mon May  9 22:23:34 2011  Deleting
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0
> Mon May  9 22:23:34 2011  Removing directory
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/rdiff-backup.tmp.0
> Mon May  9 22:23:34 2011
> -----------------------------------------------------------------
> Detected abilities for destination (read/write) file system:
>   Ownership changing                           Off
>   Hard linking                                 On
>   fsync() directories                          On
>   Directory inc permissions                    Off
>   High-bit permissions                         On
>   Symlink permissions                          Off
>   Extended filenames                           On
>   Windows reserved filenames                   Off
>   Access control lists                         Off
>   Extended attributes                          On
>   Windows access control lists                 Off
>   Case sensitivity                             On
>   Escape DOS devices                           Off
>   Escape trailing spaces                       Off
>   Mac OS X style resource forks                Off
>   Mac OS X Finder information                  Off
> -----------------------------------------------------------------
> Mon May  9 22:23:34 2011  Backup: must_escape_dos_devices = 0
> Mon May  9 22:23:34 2011  Starting mirror
> /media/BBasilioHomeBackup/fotos to /media/Elements/rdiff-backup/fotos
> Mon May  9 22:23:34 2011  Processing changed file .
> Mon May  9 22:24:00 2011  Processing changed file 2009
> Mon May  9 22:24:00 2011  Making directory path
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments
> Mon May  9 22:24:09 2011  Processing changed file 2009/2009_08_08
> Mon May  9 22:24:09 2011  Making directory path
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments/2009
> Mon May  9 22:24:09 2011  Processing changed file
> 2009/2009_08_08/2009_08_08 - Bruno
> Mon May  9 22:24:09 2011  Making directory path
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments/2009/2009_0
> 8_08 - Benfica
> Mon May  9 22:24:09 2011  Processing changed file 2009/2009_08_08 -
> Benfica/2009_08_08 - Bruno/P1030107.JPG
> Mon May  9 22:24:09 2011  Making directory path
> /media/Elements/rdiff-backup/fotos/rdiff-backup-data/increments/2009/2009_0
> 8_08 - Benfica/2009_08_08 - Bruno
> Mon May  9 22:24:09 2011  Regular copying ('2009', '2009_08_08 -
> Benfica', '2009_08_08 - Bruno', 'P1030107.JPG') to
> /media/Elements/rdiff-backup/fotos/2009/2009_08_08 -
> Benfica/2009_08_08 - Bruno/rdiff-backup.tmp.2
> Mon May  9 22:24:09 2011  Writing file object to
> /media/Elements/rdiff-backup/fotos/2009/2009_08_08 -
> Benfica/2009_08_08 - Bruno/rdiff-backup.tmp.2
> Mon May  9 22:24:09 2011  Exception '[Errno 75] Value too large for
> defined data type' raised of class '<type 'exceptions.IOError'>':
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/robust.py", line 32,
> in check_common_error
>     try: return function(*args)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 105, in
> copy if rpin.isreg(): return copy_reg_file(rpin, rpout, compress)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 133,
> in copy_reg_file
>     return rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line
> 1195, in write_from_fileobj
>     copyfileobj(fp, outfp)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 62,
> in copyfileobj
>     inbuf = inputfp.read(blocksize)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 1415, in
> read def read(self, length = -1): return self.file.read(length)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/hash.py", line 47, in
> read buf = self.fileobj.read(length)
> 
> Mon May  9 22:24:09 2011  Exception '[Errno 75] Value too large for
> defined data type' raised of class '<type 'exceptions.IOError'>':
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 304,
> in error_check_Main
>     try: Main(arglist)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 324, in
> Main take_action(rps)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 280,
> in take_action
>     elif action == "backup": Backup(rps[0], rps[1])
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 346, in
> Backup backup.Mirror(rpin, rpout)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 38, in
> Mirror DestS.patch(dest_rpath, source_diffiter)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 232, in
> patch ITR(diff.index, diff)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line
> 281, in __call__
>     last_branch.fast_process(*args)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line
> 529, in fast_process
>     if self.patch_to_temp(mirror_rp, diff_rorp, tf):
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line
> 553, in patch_to_temp
>     result = self.patch_snapshot_to_temp(diff_rorp, new)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line
> 582, in patch_snapshot_to_temp
>     (diff_rorp, new))
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/robust.py", line 32,
> in check_common_error
>     try: return function(*args)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 105, in
> copy if rpin.isreg(): return copy_reg_file(rpin, rpout, compress)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 133,
> in copy_reg_file
>     return rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line
> 1195, in write_from_fileobj
>     copyfileobj(fp, outfp)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 62,
> in copyfileobj
>     inbuf = inputfp.read(blocksize)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 1415, in
> read def read(self, length = -1): return self.file.read(length)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/hash.py", line 47, in
> read buf = self.fileobj.read(length)
> 
> Traceback (most recent call last):
>   File "/usr/bin/rdiff-backup", line 30, in <module>
>     rdiff_backup.Main.error_check_Main(sys.argv[1:])
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 304,
> in error_check_Main
>     try: Main(arglist)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 324, in
> Main take_action(rps)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 280,
> in take_action
>     elif action == "backup": Backup(rps[0], rps[1])
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/Main.py", line 346, in
> Backup backup.Mirror(rpin, rpout)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 38, in
> Mirror DestS.patch(dest_rpath, source_diffiter)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line 232, in
> patch ITR(diff.index, diff)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rorpiter.py", line
> 281, in __call__
>     last_branch.fast_process(*args)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line
> 529, in fast_process
>     if self.patch_to_temp(mirror_rp, diff_rorp, tf):
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line
> 553, in patch_to_temp
>     result = self.patch_snapshot_to_temp(diff_rorp, new)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/backup.py", line
> 582, in patch_snapshot_to_temp
>     (diff_rorp, new))
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/robust.py", line 32,
> in check_common_error
>     try: return function(*args)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 105, in
> copy if rpin.isreg(): return copy_reg_file(rpin, rpout, compress)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 133,
> in copy_reg_file
>     return rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line
> 1195, in write_from_fileobj
>     copyfileobj(fp, outfp)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 62,
> in copyfileobj
>     inbuf = inputfp.read(blocksize)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/rpath.py", line 1415, in
> read def read(self, length = -1): return self.file.read(length)
>   File "/usr/lib/pymodules/python2.6/rdiff_backup/hash.py", line 47, in
> read buf = self.fileobj.read(length)
> IOError: [Errno 75] Value too large for defined data type
> 
> 
> Search through the rdiff-backup-users Archives but couldn't find
> any cue to solve the problem.
> 
> Search the net for similar information and it seems to be related with
> python version before 2.2 and presently I'm using 2.6.5.
> Follow some version information.
> 
> <<<
> $ python --version
> Python 2.6.5
> $ rsync --version
> rsync  version 3.0.7  protocol version 30
> $ dpkg -l | grep rsync
> ii  librsync1                                        0.9.7-7
>                               rsync remote-delta algorithm library
> ii  rsync                                            3.0.7-1ubuntu1.1
>                               fast remote file copy program (like rcp)
> 
> 
> Thank you for your time.