[Duplicity-talk] AssertionError: snapshot: Using rdiffdir as an rsync re

duplicity-talk
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Duplicity-talk] AssertionError: snapshot: Using rdiffdir as an rsync re

From:	Mike Mestnik
Subject:	[Duplicity-talk] AssertionError: snapshot: Using rdiffdir as an rsync replacement.
Date:	Tue, 26 Aug 2008 11:17:37 -0500
User-agent:	Mutt/1.4.2.1i
Using rdiffdir we were able to generate signatures once per every
several replicating runs.  This gave an obvious advantage over rsync
and also allowed us to get a much smaller changeset, that rsync was
making difficult(sshing to localhost with no eyncription proved to be
the solution).

The backend storage is a netapp and we are using the snapshots to do
the archiving part of our backups.  Having the smaller changesets
generated by rdiffdir applied to the NFS mounted filesystem allowed
the netapp to make better use of it's avliable storage.

Unfortunatly none of the tools we have tried has been succsessfull at
replicating our dataset.  It's many folders of many small to large
files that have accumulated over a decade, many hundreds of GB of user
data.  rsync versions prior to 3 run out of memory, rsync 3 mostly
worked.

Could you identify what these errors mean?

AssertionError: snapshot

----- Forwarded message from Cron Daemon -----

> From: root (Cron Daemon)
> To: root
> Subject: [Alert-cron] Cron <root> /usr/local/bin/lockrun
>       --lockfile=/tmp/ad3.lockrun -- /usr/local/bin/lockrun
>       --lockfile=/tmp/master1.lockrun --wait --
>       /etc/scripts/rsync.pl ad3 >> /var/log/rsync-ad3.log
> Date: Tue, 26 Aug 2008 02:07:09 -0500 (CDT)
> 
> 1219734425 system( cd /sqldump; cd /sqldump/.snapshot/nightly.0; 
> /usr/local/bin/rdiffdir --exclude-other-filesystems 
> --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' delta 
> /rsync/61301/fetket/vol6/sqldump.sig . - | /usr/local/bin/rdiffdir 
> --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> --exclude-regexp='.*Maildir/..*' patch /rsync/61301/fetket/vol6/sqldump - >> 
> /home/rsync/ad3:sqldump; );
> cd: can't cd to /sqldump/.snapshot/nightly.0
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 184, in main
>     patch(file_args[0], get_fileobj(file_args[1], "rb"))
>   File "/usr/local/bin/rdiffdir", line 159, in patch
>     patchdir.Patch(path.Path(dirname), deltafp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 33, in Patch
>     patch_diff_tarfile(base_path, diff_tarfile)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 60, in patch_diff_tarfile
>     ITR(basis_path.index, basis_path, diff_ropath)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 297, 
> in __call__
>     last_branch.fast_process, args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 31, 
> in check_common_error
>     try: return function(*args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 257, in fast_process
>     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> AssertionError: snapshot
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 182, in main
>     else: write_delta(file_args[1], sig_infp, delta_outfp)
>   File "/usr/local/bin/rdiffdir", line 147, in write_delta
>     diffdir.write_block_iter(delta_iter, outfp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> 530, in write_block_iter
>     for block in block_iter: fp.write(block.data)
> IOError: [Errno 32] Broken pipe
> 
> _______________________________________________
> Alert-cron mailing list
> address@hidden
> http://weebles.visi.com/mailman/listinfo/alert-cron

----- End forwarded message -----
----- Forwarded message from Cron Daemon <root> -----

> From: root (Cron Daemon)
> To: root
> Subject: [Alert-cron] Cron <root> /usr/local/bin/lockrun
>       --lockfile=/tmp/apache.lockrun -- /usr/local/bin/lockrun
>       --lockfile=/tmp/master2.lockrun --wait --
>       /etc/scripts/rsync.pl apache >> /var/log/rsync-apache.log
> Date: Tue, 26 Aug 2008 02:25:42 -0500 (CDT)
> 
> 1219734490 system( cd /rsync/61327/jeeves/apache/www; cd 
> /rsync/61327/jeeves/apache/www/.snapshot/nightly.0; /usr/local/bin/rdiffdir 
> --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> --exclude-regexp='.*Maildir/..*' delta /rsync/61327/f760-2/vol0/www.sig . - | 
> /usr/local/bin/rdiffdir --exclude-other-filesystems 
> --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' patch 
> /rsync/61327/f760-2/vol0/www - >> /home/rsync/apache:www; );
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 184, in main
>     patch(file_args[0], get_fileobj(file_args[1], "rb"))
>   File "/usr/local/bin/rdiffdir", line 159, in patch
>     patchdir.Patch(path.Path(dirname), deltafp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 33, in Patch
>     patch_diff_tarfile(base_path, diff_tarfile)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 60, in patch_diff_tarfile
>     ITR(basis_path.index, basis_path, diff_ropath)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 297, 
> in __call__
>     last_branch.fast_process, args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 31, 
> in check_common_error
>     try: return function(*args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 257, in fast_process
>     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> AssertionError: snapshot
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 182, in main
>     else: write_delta(file_args[1], sig_infp, delta_outfp)
>   File "/usr/local/bin/rdiffdir", line 147, in write_delta
>     diffdir.write_block_iter(delta_iter, outfp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> 530, in write_block_iter
>     for block in block_iter: fp.write(block.data)
> IOError: [Errno 32] Broken pipe
> 1219734507 system( cd /rsync/61327/jeeves/apache/ftp; cd 
> /rsync/61327/jeeves/apache/ftp/.snapshot/nightly.0; /usr/local/bin/rdiffdir 
> --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> --exclude-regexp='.*Maildir/..*' delta /rsync/61327/f760-2/vol0/ftp.sig . - | 
> /usr/local/bin/rdiffdir --exclude-other-filesystems 
> --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' patch 
> /rsync/61327/f760-2/vol0/ftp - >> /home/rsync/apache:ftp; );
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 184, in main
>     patch(file_args[0], get_fileobj(file_args[1], "rb"))
>   File "/usr/local/bin/rdiffdir", line 159, in patch
>     patchdir.Patch(path.Path(dirname), deltafp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 33, in Patch
>     patch_diff_tarfile(base_path, diff_tarfile)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 60, in patch_diff_tarfile
>     ITR(basis_path.index, basis_path, diff_ropath)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 297, 
> in __call__
>     last_branch.fast_process, args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 31, 
> in check_common_error
>     try: return function(*args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 257, in fast_process
>     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> AssertionError: snapshot
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 182, in main
>     else: write_delta(file_args[1], sig_infp, delta_outfp)
>   File "/usr/local/bin/rdiffdir", line 147, in write_delta
>     diffdir.write_block_iter(delta_iter, outfp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> 530, in write_block_iter
>     for block in block_iter: fp.write(block.data)
> IOError: [Errno 32] Broken pipe
> 1219734536 system( cd /rsync/61327/jeeves/apache/clockwork; cd 
> /rsync/61327/jeeves/apache/clockwork/.snapshot/nightly.0; 
> /usr/local/bin/rdiffdir --exclude-other-filesystems 
> --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' delta 
> /rsync/61327/f760-2/vol0/clockwork.sig . - | /usr/local/bin/rdiffdir 
> --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> --exclude-regexp='.*Maildir/..*' patch /rsync/61327/f760-2/vol0/clockwork - 
> >> /home/rsync/apache:clockwork; );
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 184, in main
>     patch(file_args[0], get_fileobj(file_args[1], "rb"))
>   File "/usr/local/bin/rdiffdir", line 159, in patch
>     patchdir.Patch(path.Path(dirname), deltafp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 33, in Patch
>     patch_diff_tarfile(base_path, diff_tarfile)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 63, in patch_diff_tarfile
>     ITR(diff_ropath.index, basis_path, diff_ropath)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 297, 
> in __call__
>     last_branch.fast_process, args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 31, 
> in check_common_error
>     try: return function(*args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 248, in fast_process
>     diff_ropath.copy(self.base_path.new_index(index))
>   File "/usr/local/lib/python2.4/site-packages/duplicity/path.py", line 353, 
> in copy
>     if self.isreg(): other.writefileobj(self.open("rb"))
>   File "/usr/local/lib/python2.4/site-packages/duplicity/path.py", line 492, 
> in writefileobj
>     fout = self.open("wb")
>   File "/usr/local/lib/python2.4/site-packages/duplicity/path.py", line 444, 
> in open
>     else: result = open(self.name, mode)
> IOError: [Errno 28] No space left on device: 
> '/rsync/61327/f760-2/vol0/clockwork/bigip-apache/conf/access.conf-dist'
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 182, in main
>     else: write_delta(file_args[1], sig_infp, delta_outfp)
>   File "/usr/local/bin/rdiffdir", line 147, in write_delta
>     diffdir.write_block_iter(delta_iter, outfp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> 530, in write_block_iter
>     for block in block_iter: fp.write(block.data)
> IOError: [Errno 32] Broken pipe
> 1219734645 system( cd /rsync/61327/jeeves/apache/cvs; cd 
> /rsync/61327/jeeves/apache/cvs/.snapshot/nightly.0; /usr/local/bin/rdiffdir 
> --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> --exclude-regexp='.*Maildir/..*' delta /rsync/61327/f760-2/vol0/cvs.sig . - | 
> /usr/local/bin/rdiffdir --exclude-other-filesystems 
> --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' patch 
> /rsync/61327/f760-2/vol0/cvs - >> /home/rsync/apache:cvs; );
> 1219734718 system( cd /rsync/61327/jeeves/apache/tftpboot; cd 
> /rsync/61327/jeeves/apache/tftpboot/.snapshot/nightly.0; 
> /usr/local/bin/rdiffdir --exclude-other-filesystems 
> --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' delta 
> /rsync/61327/f760-2/vol0/tftpboot.sig . - | /usr/local/bin/rdiffdir 
> --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> --exclude-regexp='.*Maildir/..*' patch /rsync/61327/f760-2/vol0/tftpboot - >> 
> /home/rsync/apache:tftpboot; );
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 184, in main
>     patch(file_args[0], get_fileobj(file_args[1], "rb"))
>   File "/usr/local/bin/rdiffdir", line 159, in patch
>     patchdir.Patch(path.Path(dirname), deltafp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 33, in Patch
>     patch_diff_tarfile(base_path, diff_tarfile)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 60, in patch_diff_tarfile
>     ITR(basis_path.index, basis_path, diff_ropath)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 297, 
> in __call__
>     last_branch.fast_process, args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 31, 
> in check_common_error
>     try: return function(*args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 257, in fast_process
>     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> AssertionError: snapshot
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 182, in main
>     else: write_delta(file_args[1], sig_infp, delta_outfp)
>   File "/usr/local/bin/rdiffdir", line 147, in write_delta
>     diffdir.write_block_iter(delta_iter, outfp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> 530, in write_block_iter
>     for block in block_iter: fp.write(block.data)
> IOError: [Errno 32] Broken pipe
> 1219734721 system( cd /rsync/61327/jeeves/apache/staff; cd 
> /rsync/61327/jeeves/apache/staff/.snapshot/nightly.0; /usr/local/bin/rdiffdir 
> --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> --exclude-regexp='.*Maildir/..*' delta /rsync/61327/f760-2/vol0/staff.sig . - 
> | /usr/local/bin/rdiffdir --exclude-other-filesystems 
> --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' patch 
> /rsync/61327/f760-2/vol0/staff - >> /home/rsync/apache:staff; );
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 184, in main
>     patch(file_args[0], get_fileobj(file_args[1], "rb"))
>   File "/usr/local/bin/rdiffdir", line 159, in patch
>     patchdir.Patch(path.Path(dirname), deltafp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 33, in Patch
>     patch_diff_tarfile(base_path, diff_tarfile)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 60, in patch_diff_tarfile
>     ITR(basis_path.index, basis_path, diff_ropath)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/lazy.py", line 297, 
> in __call__
>     last_branch.fast_process, args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/robust.py", line 31, 
> in check_common_error
>     try: return function(*args)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/patchdir.py", line 
> 257, in fast_process
>     assert diff_ropath.difftype == "diff", diff_ropath.difftype
> AssertionError: snapshot
> Traceback (most recent call last):
>   File "/usr/local/bin/rdiffdir", line 193, in ?
>     if __name__ == "__main__": main()
>   File "/usr/local/bin/rdiffdir", line 182, in main
>     else: write_delta(file_args[1], sig_infp, delta_outfp)
>   File "/usr/local/bin/rdiffdir", line 147, in write_delta
>     diffdir.write_block_iter(delta_iter, outfp)
>   File "/usr/local/lib/python2.4/site-packages/duplicity/diffdir.py", line 
> 530, in write_block_iter
>     for block in block_iter: fp.write(block.data)
> IOError: [Errno 32] Broken pipe
> 1219734723 system( cd /rsync/61327/jeeves/apache/bitstream; cd 
> /rsync/61327/jeeves/apache/bitstream/.snapshot/nightly.0; 
> /usr/local/bin/rdiffdir --exclude-other-filesystems 
> --exclude-regexp='.*.snapshot' --exclude-regexp='.*Maildir/..*' delta 
> /rsync/61327/f760-2/vol0/bitstream.sig . - | /usr/local/bin/rdiffdir 
> --exclude-other-filesystems --exclude-regexp='.*.snapshot' 
> --exclude-regexp='.*Maildir/..*' patch /rsync/61327/f760-2/vol0/bitstream - 
> >> /home/rsync/apache:bitstream; );
> 
> _______________________________________________
> Alert-cron mailing list
> address@hidden
> http://weebles.visi.com/mailman/listinfo/alert-cron

----- End forwarded message -----

-- 
/****************************************************************
 *   Mike Mestnik: Junior Admin          612-395-8932           *
 *      address@hidden                  VISI                 *
 ****************************************************************/
 Alt address: address@hidden
[Prev in Thread]
Current Thread
[Next in Thread]
[Duplicity-talk] AssertionError: snapshot: Using rdiffdir as an rsync replacement., Mike Mestnik <=
- [Duplicity-talk] Re: AssertionError: snapshot: Using rdiffdir as an rsync replacement., Mike Mestnik, 2008/08/29
Prev by Date: Re: gpg: [don't know]: invalid packet (ctb=14) - WAS: [Duplicity-talk] Rollup Functionality and Parity?
Next by Date: Re: gpg: [don't know]: invalid packet (ctb=14) - WAS: [Duplicity-talk] Rollup Functionality and Parity?
Previous by thread: [Duplicity-talk] unused assert
Next by thread: [Duplicity-talk] Re: AssertionError: snapshot: Using rdiffdir as an rsync replacement.
Index(es):
- Date
- Thread