Re: [rdiff-backup-users] compare: ignore ctime

[Carsten Lorenz]

Ben Escoto wrote:

>>>>>>"Carsten Lorenz" <address@hidden>
>>>>>>wrote the following on Thu, 17 Nov 2005 11:24:03 +0100
>>>>>>            
>>>>>>
>>We are now using rdiff-backup on our debian fileserver without any real
>>problems.
>>
>>Our daily backup of ca. 1.2TB starts at 1:00am and lasts for 43 hours.
>>As 43 hours are more than a day, this is our problem.
>>
>>rdiff-backup detects half of the files (395041 files with 567GB) as
>>changed, but rdiff-backups "Increment FileSize" is only 5GB.
>>The only thing that has changed on most of this files is their ctime. So
>>rdiff-backup works fine and correct.
>>    
>>
>So anyway, the problem isn't ctimes.  How can you figure out why
>rdiff-backup thinks a file has changed?  One way is to find a file you
>think hasn't changed, and compare its records in two consecutive
>mirror_metadata files.  (Note that there is no ctime information in
>the mirror_metadata file.)
>  
>
Some test later ...
There is something wrong with the acls.

With --no-acls it took only 1.5 hours to process 1.17 TB !
Of cause it depends how many files change, but our last full tape-backup
lasts more than 2.5 days.

I have found a test case where rdiff-backup detects every file as
changed although the files are unchanged.
If i add the option --no-acls all works fine!

Looks as if there is a bug in rdiff-backup or one of the library it uses :-(

Here are samples of the output:


With the options -v 8 -no-acls
>Executing ssh 192.168.0.1 rdiff-backup --server
>Registering connection 1
>-----------------------------------------------------------------
>Detected abilities for source (read only) file system:
>  Access control lists                         On
>  Extended attributes                          On
>  Mac OS X style resource forks                Off
>  Mac OS X Finder information                  Off
>-----------------------------------------------------------------
[...]
>-----------------------------------------------------------------
>Detected abilities for destination (read/write) file system:
>  Characters needing quoting                   ''
>  Ownership changing                           On
>  Hard linking                                 On
>  fsync() directories                          On
>  Directory inc permissions                    On
>  Access control lists                         On
>  Extended attributes                          On
>  Mac OS X style resource forks                Off
>  Mac OS X Finder information                  Off
>-----------------------------------------------------------------
>Writing mirror marker
/mnt/raid/backup/test/rdiff-backup-data/current_mirror.2005-11-28T17:32:47+01:00.data
>Starting increment operation
/mnt/files/clients/Archiv/NDE_Beaute/NDE-Creative Rumble/Booklet/Rumble
Booklet/alle eps to /mnt/raid/backup/test
>--------------[ Session statistics ]--------------
>StartTime 1133195567.00 (Mon Nov 28 17:32:47 2005)
>EndTime 1133195569.45 (Mon Nov 28 17:32:49 2005)
>ElapsedTime 2.45 (2.45 seconds)
>SourceFiles 185
>SourceFileSize 1406464859 (1.31 GB)
>MirrorFiles 185
>MirrorFileSize 1406464859 (1.31 GB)
>NewFiles 0
>NewFileSize 0 (0 bytes)
>DeletedFiles 0
>DeletedFileSize 0 (0 bytes)
>ChangedFiles 0
>ChangedSourceSize 0 (0 bytes)
>ChangedMirrorSize 0 (0 bytes)
>IncrementFiles 0
>IncrementFileSize 0 (0 bytes)
>TotalDestinationSizeChange 0 (0 bytes)
>Errors 0
>--------------------------------------------------
>Deleting
/mnt/raid/backup/test/rdiff-backup-data/current_mirror.2005-11-28T17:32:35+01:00.data
>Cleaning up


Only with the option -v 8
>Executing ssh 192.168.0.1 rdiff-backup --server
>Registering connection 1
>-----------------------------------------------------------------
>Detected abilities for source (read only) file system:
>  Access control lists                         On
>  Extended attributes                          On
>  Mac OS X style resource forks                Off
>  Mac OS X Finder information                  Off
>-----------------------------------------------------------------
[...]
>-----------------------------------------------------------------
>Detected abilities for destination (read/write) file system:
>  Characters needing quoting                   ''
>  Ownership changing                           On
>  Hard linking                                 On
>  fsync() directories                          On
>  Directory inc permissions                    On
>  Access control lists                         On
>  Extended attributes                          On
>  Mac OS X style resource forks                Off
>  Mac OS X Finder information                  Off
>-----------------------------------------------------------------
>Writing mirror marker
/mnt/raid/backup/test/rdiff-backup-data/current_mirror.2005-11-28T17:07:01+01:00.data
>Starting increment operation
/mnt/files/clients/Archiv/NDE_Beaute/NDE-Creative Rumble/Booklet/Rumble
Booklet/alle eps to /mnt/raid/backup/test
[...]
>Getting signature of the kiss.eps with blocksize 3968
[...]
>Getting delta of /mnt/files/clients/Archiv/NDE_Beaute/NDE-Creative
Rumble/Booklet/Rumble Booklet/alle eps/the kiss.eps with signature the
kiss.eps
[...]
>Processing changed file the kiss.eps
>Writing file object to /mnt/raid/backup/test/rdiff-backup.tmp.183
>Copying attributes from ('the kiss.eps',) to
/mnt/raid/backup/test/rdiff-backup.tmp.183
>Setting time of /mnt/raid/backup/test/rdiff-backup.tmp.183 to 1113838797
>Incrementing mirror file /mnt/raid/backup/test/the kiss.eps
>Writing delta /mnt/raid/backup/test/rdiff-backup.tmp.183 from
/mnt/raid/backup/test/the kiss.eps ->
/mnt/raid/backup/test/rdiff-backup-data/increments/the
kiss.eps.2005-11-28T16:54:53+01:00.diff.gz
>Getting signature of rdiff-backup.tmp.183 with blocksize 3968
>Writing file object to
/mnt/raid/backup/test/rdiff-backup-data/increments/the
kiss.eps.2005-11-28T16:54:53+01:00.diff.gz
>Copying inc attrs from ('the kiss.eps',) to
/mnt/raid/backup/test/rdiff-backup-data/increments/the
kiss.eps.2005-11-28T16:54:53+01:00.diff.gz
>Setting time of /mnt/raid/backup/test/rdiff-backup-data/increments/the
kiss.eps.2005-11-28T16:54:53+01:00.diff.gz to 1113838797
>Renaming /mnt/raid/backup/test/rdiff-backup.tmp.183 to
/mnt/raid/backup/test/the kiss.eps
[...]
>--------------[ Session statistics ]--------------
>StartTime 1133194021.00 (Mon Nov 28 17:07:01 2005)
>EndTime 1133194256.15 (Mon Nov 28 17:10:56 2005)
>ElapsedTime 235.15 (3 minutes 55.15 seconds)
>SourceFiles 185
>SourceFileSize 1406464859 (1.31 GB)
>MirrorFiles 185
>MirrorFileSize 1406464859 (1.31 GB)
>NewFiles 0
>NewFileSize 0 (0 bytes)
>DeletedFiles 0
>DeletedFileSize 0 (0 bytes)
>ChangedFiles 185
>ChangedSourceSize 1406464859 (1.31 GB)
>ChangedMirrorSize 1406464859 (1.31 GB)
>IncrementFiles 185
>IncrementFileSize 27238 (26.6 KB)
>TotalDestinationSizeChange 27238 (26.6 KB)
>Errors 0
>--------------------------------------------------
>Deleting
/mnt/raid/backup/test/rdiff-backup-data/current_mirror.2005-11-28T16:54:53+01:00.data
>Cleaning up


In the mirror_metadata:
>File the kiss.eps
>  Type reg
>  Size 15773010
>  ModTime 1113838797
>  Uid 1032
>  Uname mmohr
>  Gid 17000
>  Gname beaute
>  Permissions 436

In the file_statistics:
>the kiss.eps 1 15773010 15773010 126
(Only 11 bytes after unpacking the diff file.)

getfacl of source and destination gives:
># file: the\040kiss.eps
># owner: mmohr
># group: beaute
>user::rw-
>group::rwx                      #effective:rw-
>group:readonly:r-x              #effective:r--
>mask::rw-
>other::r--

stat of source gives:
>  File: `the kiss.eps'
>  Size: 15773010        Blocks: 30816      IO Block: 4096   regular file
>Device: 841h/2113d      Inode: 1109055492  Links: 1
>Access: (0664/-rw-rw-r--)  Uid: ( 1032/   mmohr)   Gid: (17000/  beaute)
>Access: 2005-11-28 17:29:19.987785284 +0100
>Modify: 2005-04-18 17:39:57.000000000 +0200
>Change: 2005-04-25 10:34:16.455895071 +0200

stat of destination:
>  File: `the kiss.eps'
>  Size: 15773010        Blocks: 30816      IO Block: 4096   regular file
>Device: fe00h/65024d    Inode: -2122278833  Links: 1
>Access: (0664/-rw-rw-r--)  Uid: ( 1032/   mmohr)   Gid: (17000/  beaute)
>Access: 2005-11-28 17:31:35.641293223 +0100
>Modify: 2005-04-18 17:39:57.000000000 +0200
>Change: 2005-11-28 17:31:35.934240451 +0100


The versions i use are:
pylibacl 0.2.1
rdiff-backup 1.0.1
kernel 2.6.12
filesystem is xfs

Carsten :-(