Re: [rdiff-backup-users] parallel backups

[Vadim Kouzmine]

On Sat, 2006-05-13 at 06:27 -0500, David Peterka wrote:
> Hello,
> 
>  
> 
> I just started using rdiff-backup to backup some boxes and playing
> around with cron, I’m wondering if anyone has scripted anything to
> back up stuff in parallel.

Hi David,

I use make for it and it works just great.
Simplified example of makefile:

---
#
# /srv/admin/Makefile.rdiff-backup
#

HOSTS = host1 host2 ...

all: $(HOSTS)

$(HOSTS):
  /usr/local/Python/bin/rdiff-backup \
    --preserve-numerical-ids \
    --remote-schema 'ssh -c blowfish %s /usr/local/Python/bin/rdiff-backup 
--server --restrict-read-only /' \
    --exclude-globbing-filelist /srv/admin/rdiff-backup.exclude \
    $@::/ $@/

.PHONY: all $(HOSTS)
---

I call it from cron like:
  cd archive_dir && make -j4 -s -f /srv/admin/Makefile.rdiff-backup all

You can choose how many backup jobs to run at once with -j parameter.
Check your network load and backup server capability.

Before rdiff-backup I used to ssh to hosts, run tar/bzip on them and
pipe output to archive server. Run 7 backup jobs in parallel.

I also redirect 2>&1 and filter output for some known warnings
(usually /var/log files changed during backup, etc), otherwise cron will
bug you too much.

Regards,
Vadim