Re: [rdiff-backup-users] --check-destination-dir taking a very long time

[Walt Mankowski]

I found a file named

  rdiff-backup-data/current_mirror.2019-09-08T03:01:02-04:00.data

which contained

  4351

I moved it out of the way and reran the backup command. This time it
through an exception. The output is in the attached log file.

Walt

On Mon, Sep 09, 2019 at 08:17:04PM -0400, Walt Mankowski wrote:
> I ran
> 
>   $ sudo rdiff-backup -v9 --print-statistics --exclude-filelist 
> /usr/local/etc/rdiff_exclude / /backup/scruffy 2>&1 | tee rdiff-backup.txt
> 
> This time it exited right away. I've attached the log file, where the
> key message is
> 
>   Fatal Error: It appears that a previous rdiff-backup session with
>   process id 4351 is still running.
> 
> Process 4351 is /lib/systemd/systemd-resolved
> 
> Is it safe to rerun it with --force?
> 
> Walt
> 
> On Mon, Sep 09, 2019 at 08:04:46PM -0400, Patrik Dufresne wrote:
> > At this point, I would just kill all the rdiff-backup process. Then
> > manually start the backup again to the USB drive. Run it with -v9 and post
> > the logs here.
> > 
> > That should provide us enough guidance about what is going on. Either the
> > process will fail quickly (this is what I expect). If the process is taking
> > too long, try to give us the logs that you gather.
> > 
> > Since it's USB, could you check if the USB speed is alright ? If for
> > whatever reason the USB speed switched from USB 3.0 to USB 2.0. It might
> > take for ever to do a backup. You could double check this with "lsusb -t".
> > Expect 5000M for USB 3
> > 
> > ikus060@ikus060-laptop:~/workspace/HPUCA/hpuca-valuepack.git$ lsusb -t
> > /:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 10000M
> >     |__ Port 4: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
> > /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
> >     |__ Port 5: Dev 14, If 0, Class=Hub, Driver=hub/3p, 480M
> > 
> > A look to "dmesg" might also reveal some information about a change to the
> > usb channel.
> > 
> > --
> > Patrik Dufresne Service Logiciel inc.
> > http://www.patrikdufresne.com <http://patrikdufresne.com/>/
> > 514-971-6442
> > 130 rue Doris
> > St-Colomban, QC J5K 1T9
> > 
> > 
> > On Mon, Sep 9, 2019 at 7:47 PM Walt Mankowski <address@hidden> wrote:
> > 
> > > On Mon, Sep 09, 2019 at 07:38:52PM -0400, Patrik Dufresne wrote:
> > > > Hum, this is strange. It should not fail with a "no space left on
> > > device".
> > >
> > > Agreed! That's why I originally thought it must have been some sort of
> > > USB glitch.
> > >
> > > > Could you provide the log generate with -v9 ? Plz provide the full
> > > command
> > > > line you used.
> > >
> > > So kill the run with -v8?
> > >
> > > > What is the filesystem of your USB drive ?
> > >
> > > ext4
> > >
> > > > If you try to run the backup again do you have an error?
> > >
> > > In fact that happened last night. My normal nightly backup kicked in
> > > while a previous attempt at running --check-destination-dir was still
> > > running. The cronjob reported:
> > >
> > >   Previous backup seems to have failed, regressing destination now.
> > >   Fatal Error: Killed with signal 15
> > >
> > > The latter was when I killed it when I woke up and saw that both of
> > > them were running.
> > >
> > > Walt
> > >
> > > > On Mon, Sep 9, 2019, 7:33 PM Walt Mankowski, <address@hidden> wrote:
> > > >
> > > > > Good idea! But unfortunately it doesn't seem to be the problem:
> > > > >
> > > > > % df -hi /backup
> > > > > Filesystem     Inodes IUsed IFree IUse% Mounted on
> > > > > /dev/sde1        117M   19M   98M   17% /backup
> > > > >
> > > > >
> > > > > On Mon, Sep 09, 2019 at 07:23:14PM -0400, Patrik Dufresne wrote:
> > > > > > Hello Walt, could you double check the disk space. Especially the
> > > number
> > > > > of
> > > > > > inode ? It's probably the root cause of your issue.
> > > > > >
> > > > > > On Mon, Sep 9, 2019, 7:19 PM Walt Mankowski, <address@hidden>
> > > wrote:
> > > > > >
> > > > > > > I've been running rdiff-backup to an external USB drive for years
> > > > > > > without any problems. Over the weekend my backup failed with
> > > > > > >
> > > > > > >   Exception '[Errno 28] No space left on device
> > > > > > >
> > > > > > > This is odd, since there is 1.2 TB free on the drive. I didn't see
> > > any
> > > > > > > errors in syslog, and I was able to create a new file on the drive
> > > > > > > without any problem.
> > > > > > >
> > > > > > > Thinking it might have been a USB glitch I rebooted the machine 
> > > > > > > and
> > > > > > > now I'm running
> > > > > > >
> > > > > > >   rdiff-backup --check-destination-dir
> > > > > > >
> > > > > > > to recover the backup directory. It was taking a very long time,
> > > and I
> > > > > > > restarted it with the -v8 hoping I might get some clue as to what
> > > it
> > > > > > > was doing. Unfortunately after spitting out some routine-looking
> > > > > > > output in the first few seconds it's now been running in silence
> > > for
> > > > > > > nearly 12 hours.
> > > > > > >
> > > > > > > It's getting CPU time and I don't see any errors in syslog, so I'm
> > > > > > > assuming that it's doing something. But I don't have any idea what
> > > > > > > it's doing, if it's working correctly, or how much longer it's
> > > likely
> > > > > > > to take.
> > > > > > >
> > > > > > > Is it normal that a regression takes this long? /backup is
> > > currently
> > > > > > > at 527 GB.
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > Walt
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > rdiff-backup-users mailing list at address@hidden
> > > > > > > https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> > > > > > > Wiki URL:
> > > > > > >
> > > http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
> > > > >
> > > > > _______________________________________________
> > > > > rdiff-backup-users mailing list at address@hidden
> > > > > https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> > > > > Wiki URL:
> > > > > http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
> > >
> > > _______________________________________________
> > > rdiff-backup-users mailing list at address@hidden
> > > https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> > > Wiki URL:
> > > http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

> Mon Sep  9 20:09:56 2019  Using rdiff-backup version 1.2.8
> Mon Sep  9 20:09:57 2019  Unable to import win32security module. Windows ACLs
> not supported by filesystem at /
> Mon Sep  9 20:09:57 2019  escape_dos_devices not required by filesystem at /
> Mon Sep  9 20:09:57 2019  
> -----------------------------------------------------------------
> Detected abilities for source (read only) file system:
>   Access control lists                         On
>   Extended attributes                          On
>   Windows access control lists                 Off
>   Case sensitivity                             On
>   Escape DOS devices                           Off
>   Escape trailing spaces                       Off
>   Mac OS X style resource forks                Off
>   Mac OS X Finder information                  Off
> -----------------------------------------------------------------
> Mon Sep  9 20:09:57 2019  Making directory 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/5-_ a.snapshot.gz
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/5-_ a.snapshot.gz
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/uniᄉ
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/uniᄉ
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/:\"
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/:\"
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/A
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/A
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/foo
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/foo
> Mon Sep  9 20:09:57 2019  Making directory 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/hl
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/hardlinked_file1
> Mon Sep  9 20:09:57 2019  Hard linking 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/hl/hardlinked_file2 to 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/hardlinked_file1
> Mon Sep  9 20:09:57 2019  Unable to import win32security module. Windows ACLs
> not supported by filesystem at 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/dir_inc_check
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/dir_inc_check
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/regfile
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/regfile
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/high_perms_file
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/high_perms_dir
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/high_perms_file
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/high_perms_dir
> Mon Sep  9 20:09:57 2019  Touching 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/symlinked_file1
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/symlinked_file2
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0/symlinked_file1
> Mon Sep  9 20:09:57 2019  escape_dos_devices not required by filesystem at 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0
> Mon Sep  9 20:09:57 2019  Deleting 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0
> Mon Sep  9 20:09:57 2019  Removing directory 
> /backup/scruffy/rdiff-backup-data/rdiff-backup.tmp.0
> Mon Sep  9 20:09:57 2019  
> -----------------------------------------------------------------
> Detected abilities for destination (read/write) file system:
>   Ownership changing                         Mon Sep  9 20:09:57 2019  Fatal 
> Error: It appears that a previous rdiff-backup session with process
> id 4351 is still running.  If two different rdiff-backup processes write
> the same repository simultaneously, data corruption will probably
> result.  To proceed with regress anyway, rerun rdiff-backup with the
> --force option.
>   On
>   Hard linking                                 On
>   fsync() directories                          On
>   Directory inc permissions                    On
>   High-bit permissions                         On
>   Symlink permissions                          Off
>   Extended filenames                           On
>   Windows reserved filenames                   Off
>   Access control lists                         On
>   Extended attributes                          On
>   Windows access control lists                 Off
>   Case sensitivity                             On
>   Escape DOS devices                           Off
>   Escape trailing spaces                       Off
>   Mac OS X style resource forks                Off
>   Mac OS X Finder information                  Off
> -----------------------------------------------------------------
> Mon Sep  9 20:09:57 2019  Backup: must_escape_dos_devices = 0

> _______________________________________________
> rdiff-backup-users mailing list at address@hidden
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

rdiff-backup2.txt
Description: Text document