On 2024-09-27 22:00, Duffer wrote:
> On 2024-09-27 05:29, Eric Lavarde wrote:
>> Hi,
>>
>> On 27/09/2024 07:09, Duffer wrote:
>>> On 2024-09-25 22:30, Duffer wrote:
>>>> Hi.
>>>>
>>>> I'm using rdiff-backup 2.2.6 for a large amount of data. All traffic
>>>> is encrypted via stunnel.
>>>>
>>>> I have two concerns:
>>>>
>>>> 1. Speed
>>>>
>>>> With SSH, the speed I've seen is 25-50 Mbps, probably averaging
>>>> 30-40 Mbps. iperf3 gets around 500 Mbps between my local system
>>>> (destination) and the remote system (source). The SSH speed seems
>>>> excessively slow to me - maybe 6-8% of the measured bandwidth.
>>>>
>>>> I switched to mapping the data source via NFS. Now I'm seeing full
>>>> speed for the transfers.
>>>>
>>>> Why is SSH so incredibly slow? Is that expected? I'm okay using NFS
>>>> but I liked the convenience of SSH.
>>
>> Well, I can only guess that it's the pickling and unpickling necessary
>> over the network. I'm working on improving performance but it takes
>> time. I first need to refactor and simplify the code. It could also be
>> worsened due to a weak CPU on one end or the other. Many small files
>> also increase a lot the overhead.
>
> Good to know. I think SSH has a lot of overhead, anyway. It's usually
> far from the fastest protocol. There's at least one project attempting
> to improve performance: https://www.psc.edu/hpn-ssh-home/
>
>>>> 2. Regression
>>>>
>>>> As noted, my backup was very slow so I stopped it (CTRL-C). The next
>>>> time I ran the backup, it went into "regress" mode, so I lost all
>>>> the progress. Is there no way to avoid this and resume somehow? I
>>>> couldn't find anything in the FAQ.
>>
>> No, but there is already an issue for this. Again, takes time...
>
> Thanks. Yes, it's definitely something you want to get right before
> deploying it.
>
>>> The regress didn't take long at all but it's now been running over 10
>>> hours and I've seen little progress. Disk usage hasn't increased at
>>> all. Is this expected? I thought after the regress it would start
>>> downloading, like it did before. I can't tell what it's doing but I
>>> seem to see it reading a lot from the disk. I used "-v 3" but maybe I
>>> should have opted for more output.
>>
>> -v3 is the default value, I generally use -v5 to see progress (you can
>> configure differently logfile and console output BTW).
>> Difficult to say what's happening from here, but did you check if
>> files are appearing in your backup repo, especially under
>> rdiff-backup-data?
>> If everything breaks, assuming Linux, you can try to guess what it's
>> doing using strace.
>>
>> KR, Eric
>
> I'm seeing continuous creation of files like rdiff-backup.tmp.217452 in
> the backup path. There are also lots of files under rdiff-backup-data,
> including two one file and one directory from today.
>
> It's been running 18 hours, so I hate to break it again but I really
> wish I could figure out what it's doing. One of the backup directories
> (where I see the tmp files) keeps growing and shrinking.
It's been running almost 24 hours now. I see lots of activity,
accessing
files in the actual data, .diff.gz files, and .tmp files. But the disk
usage still hasn't changed. Is it possible it's re-verifying every
single file? I wish I could figure out what it is doing. This seems
completely different from the previous run that I stopped.