Re: using --remove-older-than takes a very long time?

[Derek Atkins]

Eric (et al),

Last night's backup session ran.  It started at 1am, finished a backup
of a server at 3:30 (which is to be expected; it has a lot of data), and
then proceeded to take the next NINE (9) HOURS to delete a single
increment.

I am happy to forward the data, but it's a relatively large email
(multiple megabytes) so I'm hesitant to send it to the list or to anyone
else without prior approval.

Anyone else willing to look at this?

I'll note that I have not completely ruled out filesystem slowness on
deletions.  The underlying file system is EncFS/Fuse over NFS over ZFS,
but raw Filesystem I/O tests show good throughput in general.

-derek

Derek Atkins <address@hidden> writes:

> "Eric L." <address@hidden> writes:
>
>> I don't think so but it probably depends on how many backups it has to
>> remove.
>
> So that first run it had to remove 7.  Normally it's just removing 1 (as
> it runs once per day).  However the difference in runtime between
> removing 7 and removing 1 was definitely not 7x the runtime.  But it
> definitely spends more time on the remove-old-backups than it does
> backing up the day's changes.
>
> [snip]
>> A profiling as such is possible at python level but I would start with
>> just calling the removal option with `-v9`, it should give you/us a
>> first hint as the date/times for each action are logged.
>
> I now have some multi-megabyte emails detailing the last few days of
> processing (modulo some VM system crashes that happened over the
> weekend).  There are dozens of operations per second (or more -- I
> haven't looked closely), but still thousands of operations as it runs
> for hours.  Frankly I'm not sure what I am looking at/for here.
>
> I'm happy to share a log email (privately) if someone wants to look at
> it with me?
>
> -derek

-- 
       Derek Atkins                 617-623-3745
       address@hidden             www.ihtfp.com
       Computer and Internet Security Consultant