[rdiff-backup-users] Tips/tricks for decreasing load?

[Jason L. Buberel]

I have been using rdiff-backup to perform remote backups of a production 
web-based application.

The backup server 'srv4' is in ca olocation facility in Mountain View, 
CA and the production server 'srv1' is located in Houston, TX. The 
backup script is executed from 'srv4' and first connects to the remote 
server in Houston and configures Postgres to go into 'hot backup' mode. 
The script then immediately executes:

/usr/bin/rdiff-backup srv1::/pgdata/8.1/ /data1/srv1/8.1

Upon completion, the backup script terminates the 'hot backup' runtime 
mode on the Postgres server running on 'srv1'.

The problem that I am running into is that during the time that 
rdiff-backup is running, the web application hosted on 'srv1' suffers a 
fairly severe performance degradation. The site, 
http://www.altosresearch.com/, is very dependent on the Postgres query 
performance. During the time that rdiff-backup is running, the machine 
will experience very high load averages due primarily to 'iowait';

14:00:03  up 13 days, 12:29,  1 user,  load average: 10.78, 10.94, 7.67
477 processes: 474 sleeping, 3 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
          total    6.0%    0.0%    3.4%   0.0%     0.4%   90.0%    0.0%
          cpu00    7.4%    0.0%    1.8%   0.0%     0.0%   90.6%    0.0%
          cpu01    3.7%    0.0%    5.6%   0.0%     0.0%   90.6%    0.0%
          cpu02    3.7%    0.0%    5.5%   0.0%     0.9%   89.8%    0.0%
          cpu03    9.2%    0.0%    0.9%   0.0%     0.9%   88.8%    0.0%
Mem:  2054024k av, 2034036k used,   19988k free,       0k shrd,   27512k 
buff
                  1555240k actv,  309016k in_d,   35480k in_c
Swap: 2048276k av,  403592k used, 1644684k free                  901548k 
cached

 PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
10717 java     16   0 1052M 779M  1300 S     2.0 38.8   1:26   2 java
32004 postgres  21   0 19004  18M 17176 D     2.0  0.9   0:00   0 postmaster
9483 root      15   0 53088  27M  1140 D     1.1  1.3  48:59   0 
rdiff-backup

Because this is a production system, I cannot afford to take the entire 
site offline while backups complete.

Are there any tactics, tricks, configuration options, etc. that I can 
use to lessen the impact of rdiff-backup while it is running? What if I 
reversed the direction of the rdiff-backup command - initiated the 
backup on 'srv1' to 'srv4' -  would that make a difference?

All suggestions welcome. If you think that a solution besides 
rdiff-backup may be more appropriate for my situation, please let me know.

Regards,
Jason