[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Rdiff-backup-commits] rdiff-backup FAQ-body.html
|
From: |
Andrew Ferguson |
|
Subject: |
[Rdiff-backup-commits] rdiff-backup FAQ-body.html |
|
Date: |
Sat, 04 Aug 2007 16:44:19 +0000 |
CVSROOT: /sources/rdiff-backup
Module name: rdiff-backup
Changes by: Andrew Ferguson <owsla> 07/08/04 16:44:19
Modified files:
. : FAQ-body.html
Log message:
FAQ update in advance of new stable release
CVSWeb URLs:
http://cvs.savannah.gnu.org/viewcvs/rdiff-backup/FAQ-body.html?cvsroot=rdiff-backup&r1=1.18&r2=1.19
Patches:
Index: FAQ-body.html
===================================================================
RCS file: /sources/rdiff-backup/rdiff-backup/FAQ-body.html,v
retrieving revision 1.18
retrieving revision 1.19
diff -u -b -r1.18 -r1.19
--- FAQ-body.html 12 Aug 2005 05:38:20 -0000 1.18
+++ FAQ-body.html 4 Aug 2007 16:44:19 -0000 1.19
@@ -1,52 +1,36 @@
<!-- #bbpragma doctype="-//W3C//DTD XHTML 1.0 Strict//EN" root_element="body"
-->
-
-<h3>Table of contents</h3>
-
+<h3><a name="ToC3">Table of contents</a></h3>
<ol><li><a href="#verbosity">What do the different verbosity levels
mean?</a></li>
-
<li><a href="#windows">Does rdiff-backup run under Windows?</a></li>
-
<li><a href="#OSX">Does rdiff-backup run under Mac OS X?</a></li>
-
+<li><a href="#cifs">Can I backup files to a CIFS or smbfs mount?</a></li>
+<li><a href="#case_insensitive">Help! Why do all my filenames now look like
;077y;070ile ?!</a></li>
<li><a href="#remove_dir">My backup set contains some files that I just
realized I don't want/need backed up. How do I remove them from the backup
volume to save space?</a></li>
-
<li><a href="#solaris">Does rdiff-backup work under Solaris?</a></li>
-
<li><a href="#speed">How fast is rdiff-backup? Can it be run on large
data sets?</a></li>
-
<li><a href="#statistics">What do the various fields mean in the
session statistics and directory statistics files?</a></li>
-
<li><a href="#bwlimit">Is there some way to limit rdiff-backup's
bandwidth usage, as in rsync's --bwlimit option?</a></li>
-
<li><a href="#leak">How much memory should rdiff-backup use? Is there a
memory leak?</a></li>
-
<li><a href="#dir_not_empty">I use NFS and keep getting some error that
includes "OSError: [Errno 39] Directory not empty"</a></li>
-
<li><a href="#regress_failure">For some reason rdiff-backup failed
while backing up. Now every time it runs it says "regressing
destination" and then fails again. What should I do?</a></li>
-
<li><a href="#free_space">Where does rdiff-backup need free space and
how much is required? What is the problem if rdiff-backup says
"<code>ValueError: Incorrect length of data produced</code>"?</a></li>
-
+<li><a href="#librsync_bug">What does "internal error: job made no progress"
mean?</a></li>
+<li><a href="#path">Why does rdiff-backup say it's not in my $PATH? It is when
I login!</a></li>
</ol>
-
-
-<h3>Questions and Answers</h3>
-
+<h3><a name="ToC4">Questions and Answers</a></h3>
<ol>
-
<li><strong><a name="verbosity">What do the different verbosity levels
mean?</a></strong>
-
<p>There is no formal specification, but here is a rough description
(settings are always cumulative, so 5 displays everything 4 does):</p>
-
-<table cellspacing="10">
+<table cellspacing="10" summary="">
<tr><td>0</td><td>No information given</td></tr>
<tr><td>1</td><td>Fatal Errors displayed</td></tr>
<tr><td>2</td><td>Warnings</td></tr>
@@ -59,64 +43,32 @@
<tr><td>9</td><td>Details on which objects are moving across the
connection</td></tr>
</table>
</li>
-
<li><strong><a name="windows">Does rdiff-backup run under Windows?</a></strong>
-
-<p>Yes, apparently it is possible. First, follow Jason Piterak's
-instructions:</p>
-
-<pre>Subject: Cygwin rdiff-backup
-From: Jason Piterak <address@hidden>
-Date: Mon, 4 Feb 2002 16:54:24 -0500 (13:54 PST)
-To: address@hidden
-
-Hello all,
- On a lark, I thought I would attempt to get rdiff-backup to work under
-Windows98 under Cygwin. We have a number of NT/Win2K servers in the field
-that I'd love to be backing up via rdiff-backup, and this was the start of
-getting that working.
-
-SUMMARY:
- o You can get all the pieces for rdiff-backup working under Cygwin.
- o The backup process works up to the point of writing any files with
-timestamps.
- ... This is because the ':' character is reserved for Alternate Data
-Stream (ADS) file designations under NTFS.
-
-HOW TO GET IT WORKING (to a point, anyway):
- o Install Cygwin
- o Download the Python 2.2 update through the Cygwin installer and install.
- o Download the librsync libraries from the usual place, but before
-compiling...
- o Cygwin does not use/provide glibc. Because of this, you have to repoint
-some header files in the Makefile:
-
- -- Make sure that you have /usr/include/inttypes.h
- redirected to /usr/include/sys/types.h. Do this by:
-
- create a file /usr/include/inttypes.h with the contents:
- <protect>#include <sys/types.h></protect>
- o Put rdiff-backup in your PATH, as you normally would.
-
-</pre>
-
-<p>XXX The above information is old, point to newer porting efforts?</p>
-
+<p>Yes, although it is not a heavily tested configuration. Using the latest
releases, such as
+1.1.12 and beyond, rdiff-backup runs quite well under Cygwin. Dave Kempe has
also made some .exe versions <a
href="http://solutionsfirst.com.au/~dave/backup/";>available for
download</a>.</p>
+<p>The setup under Cygwin is the same as under other Unix-like operating
systems. From the Cygwin installer you will need Python 2.2 or higher (under
Interpreters), autoconf, automake, binutils, gcc, make, and patchutils (all
under Devel). Then you will need to compile and install librsync, which can be
downloaded <a
href="https://sourceforge.net/project/showfiles.php?group_id=56125";>from
Sourceforge</a>. Finally, you can compile and install rdiff-backup using the
usual instructions.</p>
<p>Although some Windows filesystems lack features like FIFOs, case
-sensitive filenames, or files with colons (":") in them, this should
+sensitive filenames, or files with colons (":") in them, all of these
situations should
be autodetected and compensated for by rdiff-backup.</p>
+<p>If you would like more detailed instructions for compiling and installing
rdiff-backup on Cygwin, you can read this blog entry: <a
href="http://katastrophos.net/andre/blog/?p=19";>http://katastrophos.net/andre/blog/?p=19</a>.
Note: The patch that the blog suggests that you download is no longer
necessary starting with version 1.1.8 of rdiff-backup.</p>
</li>
-
<li><strong><a name="OSX">Does rdiff-backup run under Mac OS X?</a></strong>
-
<p>Yes, quite a few people seem to be using rdiff-backup under Mac OS
-X. rdiff-backup can also backup resource forks to a traditional unix
-filesystem, which is can be a handy feature for Mac users. The easiest
-option is probably to use Fink <a
+X. rdiff-backup can also backup resource forks and other Mac OS X metadata
+to a traditional unix filesystem, which is can be a handy feature for Mac
+users. When rdiff-backup is used to do the restore, all of the metadata is
+recovered from rdiff-backup's storage.</p>
+<p>The easiest option is probably to use Fink <a
href="http://fink.sourceforge.net/";>http://fink.sourceforge.net/</a>,
-which can install rdiff-backup automatically for you. If you want to
-build rdiff-backup yourself, see this message from Gerd Knops:</p>
-
+which can install rdiff-backup automatically for you. With Fink, you
+should install the <code>librsync</code>, <code>librsync-shlibs</code>,
+<code>python25</code>, <code>python25-shlibs</code>, <code>xattr-py25</code>,
+and <code>rdiff-backup</code> packages. Another option is DarwinPorts
+<a href="http://www.macports.org/";>http://www.macports.org/</a>, for which
+you should install the <code>py-xattr</code> and <code>rdiff-backup</code>
packages.</p>
+<p>If you want to build rdiff-backup yourself, you should be able to build
+librsync and rdiff-backup using the standard Unix instructions.
+You can also see this message from Gerd Knops:</p>
<pre>From: Gerd Knops <address@hidden>
Date: Thu, 3 Oct 2002 03:56:47 -0500 (01:56 PDT)
@@ -130,27 +82,60 @@
env CFLAGS=-no-cpp-precomp ./configure
make
make install</pre>
-
+<p>An important note if you use the Apple-provided version of Python: Apple's
version
+of Python will install rdiff-backup in something like
+<code>/System/Library/Frameworks/Python.framework/Versions/Current/bin/rdiff-backup</code>
and <code>rdiff-backup</code>
+will not be in your <code>$PATH</code>. You can copy rdiff-backup out of this
folder and into
+someplace reasonable like <code>/usr/bin</code> or another directory in your
<code>$PATH</code> to use it. For
+a full explanation of why this happens see this post to the mailing list:
+<a
href="http://lists.nongnu.org/archive/html/rdiff-backup-users/2007-06/msg00107.html";>http://lists.nongnu.org/archive/html/rdiff-backup-users/2007-06/msg00107.html</a>.</p>
+</li>
+<li><strong><a name="cifs">Can I backup files to a CIFS or smbfs
mount?</a></strong>
+<p>You can certainly try! Using a CIFS or smbfs mount as the mirror directory
has been troublesome for
+some users because of the wide variety of Samba configurations. If possible,
the best solution is always
+to use rdiff-backup over SSH in the default configuration. Under both Linux
and Mac OS X, smbfs seems to
+be working quite well. However, it has a 2 GB file limit and is deprecated on
Linux. CIFS users sometimes
+experience one of these common errors:</p>
+<ul>
+ <li>rdiff-backup fails to run, printing an exception about
"<code>assert not upper_a.lstat()</code>" failing.
+ This can be resolved by unmounting the share, running the following
command as root:<br>
+ <code>$ echo 0 > /proc/fs/cifs/LookupCacheEnabled</code><br>
+ and then remounting the CIFS share.<br><br>
+ <li>If filenames in the mirror directory have some characters
transformed to a '?' instead of remaining
+ the expected Unicode character, you will need to adjust the
<code>iocharset=</code> mount option. This
+ happens because the server is using a codepage with only partial
Unicode support and is not translating
+ characters correctly. See the mount.cifs man page for more information.
Using smbfs can also improve this
+ situation since it has both an <code>iocharset=</code> and a
<code>codepage=</code> option.
+</ul>
+<p>If you're still having trouble backing up to a CIFS or smbfs mount, try
searching the
+<a href="http://lists.gnu.org/archive/html/rdiff-backup-users/";>mailing-list
archives</a> and then sending
+further questions to the list.</p>
+</li>
+<li><strong><a name="case_insensitive">Help! Why do all my filenames now look
like ;077y;070ile ?!</a></strong>
+<p>When backing up from a case-sensitive filesystem to a case-insensitive
filesystem (such as Mac's HFS+ or
+Windows's FAT32 or NTFS), rdiff-backup escapes uppercase characters in
filenames to make sure that no files
+are accidentally overwritten. When a filesystem is case-preserving but
case-insensitive, it means that it
+remembers that a file is named "Foo" but doesn't distinguish between "Foo",
"foo", "foO", "fOo", etc. However,
+Filesystems such as Linux's ext3 do treat these names as separate files.</p>
+<p>Imagine you have a Linux directory with two files, "bar" and "BAR", and you
copy them to a Mac system. You will
+wind up with only one file (!) since HFS+ doesn't distinguish between the
names, and the second file copied will
+overwrite the first. Therefore, when rdiff-backup copies files from
case-sensitive to case-insensitive filesystems, it escapes the uppercase
characters (eg, "M" is replaced with ";077", and "F" with ";070") so that no
filename
+conflicts occur. Upon restore (from the Mac backup server to the Linux
system), the filenames are unquoted and
+you will get "MyFile" back.</p>
</li>
-
<li><strong><a name="remove_dir">My backup set contains some files that I just
realized I
don't want/need backed up. How do I remove them from the backup
volume to save space?</a></strong>
-
<p>The only official way to remove files from an rdiff-backup
repository is by letting them expire using the --remove-older-than
option. Deleting increments from the rdiff-backup-data directory will
prevent you from recovering those files, but shouldn't prevent the
rest of the repository from being restored.</p>
-
</li>
-
<li><strong><a name="solaris">Does rdiff-backup work under
Solaris?</a></strong>
-
<p>There may be a problem with rdiff-backup and Solaris' libthread.
Adding "ulimit -n unlimited" may fix the problem though. Here is a
post by Kevin Spicer on the subject:</p>
-
<pre>Subject: RE: Crash report....still not^H^H^H working
From: "Spicer, Kevin" <address@hidden>
Date: Sat, 11 May 2002 23:36:42 +0100
@@ -165,7 +150,6 @@
to the start of my cron job and created a wrapper script on the remote
machine which looked like this...
- #!/bin/sh
ulimit -n unlimited
rdiff-backup --server
exit
@@ -178,10 +162,8 @@
at least I think I've now got round it).
</pre>
</li>
-
<li><strong><a name="speed">How fast is rdiff-backup? Can it be run on large
data sets?</a></strong>
-
<p>rdiff-backup can be limited by the CPU, disk IO, or available
bandwidth, and the length of a session can be affected by the amount
of data, how much the data changed, and how many files are present.
@@ -191,18 +173,14 @@
the total number of files. Initial mirrorings will usually be
bandwidth or disk bound, and will take much longer than subsequent
updates.</p>
-
<p>To give one arbitrary data point, when I back up my personal HD
locally (about 36GB, 530000 files, maybe 500 MB turnover, Athlon 2000,
7200 IDE disks, version 0.12.2) rdiff-backup takes about 15 minutes
and is usually CPU bound.</p>
</li>
-
<li><strong><a name="statistics">What do the various fields mean in the
session statistics and directory statistics files?</a></strong>
-
<p>Let's examine an example session statistics file:</p>
-
<pre>StartTime 1028200920.44 (Thu Aug 1 04:22:00 2002)
EndTime 1028203082.77 (Thu Aug 1 04:58:02 2002)
ElapsedTime 2162.33 (36 minutes 2.33 seconds)
@@ -221,11 +199,9 @@
IncrementFileSize 13799799 (13.2 MB)
TotalDestinationSizeChange 28034365 (26.7 MB)
Errors 0</pre>
-
<p>StartTime and EndTime are measured in seconds since the epoch.
ElapsedTime is just EndTime - StartTime, the length of the
rdiff-backup session.</p>
-
<p>SourceFiles are the number of files found in the source directory,
and SourceFileSize is the total size of those files. MirrorFiles are
the number of files found in the mirror directory (not including the
@@ -233,26 +209,21 @@
those files. All sizes are in bytes. If the source directory hasn't
changed since the last backup, MirrorFiles == SourceFiles and
SourceFileSize == MirrorFileSize.</p>
-
<p>NewFiles and NewFileSize are the total number and size of the files
found in the source directory but not in the mirror directory. They
are new as of the last backup.</p>
-
<p>DeletedFiles and DeletedFileSize are the total number and size of
the files found in the mirror directory but not the source directory.
They have been deleted since the last backup.</p>
-
<p>ChangedFiles are the number of files that exist both on the mirror
and on the source directories and have changed since the previous
backup. ChangedSourceSize is their total size on the source
directory, and ChangedMirrorSize is their total size on the mirror
directory.</p>
-
<p>IncrementFiles is the number of increment files written to the
rdiff-backup-data directory, and IncrementFileSize is their total
size. Generally one increment file will be written for every new,
deleted, and changed file.</p>
-
<p>TotalDestinationSizeChange is the number of bytes the destination
directory as a whole (mirror portion and rdiff-backup-data directory)
has grown during the given rdiff-backup session. This is usually
@@ -260,16 +231,13 @@
ChangedSourceSize - ChangedMirrorSize, but it also includes the space
taken up by the hardlink_data file to record hard links.</p>
</li>
-
<li><strong><a name="bwlimit">Is there some way to limit rdiff-backup's
bandwidth usage, as in rsync's --bwlimit option?</a></strong>
-
<p>There is no internal rdiff-backup option to do this. However,
external utilities such as <a
href="http://www.cons.org/cracauer/cstream.html";>cstream</a> can be
used to monitor bandwidth explicitly. address@hidden
writes:</p>
-
<pre>rdiff-backup --remote-schema
'cstream -v 1 -t 10000 | ssh %s '\''rdiff-backup --server'\'' | cstream -t
20000'
'address@hidden::/mnt/backup' localbakdir
@@ -283,90 +251,69 @@
to output stats for both directions as it will confuse whatever script
you have parsing the output. I guess it wouldn't hurt for manual runs
however.</pre>
-
<p>To only limit bandwidth in one directory, simply remove one of the
cstream commands. Two cstream caveats may be worth mentioning:</p>
-
<ol> <li>Because cstream is limiting the uncompressed data heading
into or out of ssh, if ssh compression is turned on, cstream may be
overly restrictive.</li>
-
<li>cstream may be "bursty", limiting average bandwidth but allowing
rdiff-backup to exceed it for significant periods.</li>
</ol>
-
<p>Another option is to limit bandwidth at a lower (and perhaps more
appropriate) level. Adam Lazur mentions <a
href="http://lartc.org/wondershaper/";>The Wonder Shaper</a>.</p>
</li>
-
<li><strong><a name="leak">How much memory should rdiff-backup use? Is there a
memory leak?</a></strong>
-
<p>The amount of memory rdiff-backup uses should not depend much on
the size of directories being processed. Keeping track of hard links
may use up memory, so if you have, say, hundreds of thousands of files
hard linked together, rdiff-backup may need tens of MB.</p>
-
<p>If rdiff-backup seems to be leaking memory, it is probably because
it is using an early version of librsync. <strong>librsync 0.9.5
leaks lots of memory.</strong> Later versions should not leak and are
available from the <a
href="https://sourceforge.net/projects/librsync/";>librsync homepage</a>.</p>
</li>
-
<li><strong><a name="dir_not_empty">I use NFS and keep getting some error that
includes "OSError: [Errno 39] Directory not empty"</a></strong>
-
<p>Several users have reported seeing errors that contain lines like
this:</p>
-
<pre>File "/usr/lib/python2.2/site-packages/rdiff_backup/rpath.py",
line 661, in rmdir
OSError: [Errno 39] Directory not empty:
'/nfs/backup/redfish/win/Program Files/Common Files/GMT/Banners/11132'
Exception exceptions.TypeError: "'NoneType' object is not callable"
in <bound method GzipFile.__del__ of</pre>
-
<p>All of these users were backing up onto NFS (Network File System).
I think this is probably a bug in NFS, although tell me if you know
how to make rdiff-backup more NFS-friendly. To avoid this problem,
run rdiff-backup locally on both ends instead of over NFS. This
should be faster anyway.</p>
</li>
-
<li><strong><a name="regress_failure">For some reason rdiff-backup failed
while backing up. Now every time it runs it says "regressing
destination" and then fails again. What should I do?</a></strong>
-
<p>Firstly, this shouldn't happen. If it does, it indicates a
corrupted destination directory, a bug in rdiff-backup, or some other
serious recurring problem.</p>
-
<p>However, here is a workaround that you might want to use, even
though it probably won't solve the underlying problem: In the
destination's rdiff-backup-data directory, there should be two
"current_mirror" files, for instance:</p>
-
<pre>current_mirror.2003-09-07T16:43:00-07:00.data
current_mirror.2003-09-08T04:22:01-07:00.data</pre>
-
<p>Delete the one with the earlier date. Also move the mirror_metadata
file with the later date out of the way, because it probably didn't
get written correctly because that session was aborted:</p>
-
-<pre>mv mirror_metadata.2003-09-08T04:22:01-07:00.snapshot.gz \
- aborted-metadata.2003-09-08T04:22:01-07:00.snapshot.gz</pre>
-
+<pre>mv mirror_metadata.2003-09-08T04:22:01-07:00.snapshot.gz
aborted-metadata.2003-09-08T04:22:01-07:00.snapshot.gz</pre>
<p>The next time rdiff-backup runs it won't try regressing the
destination. Metadata will be read from the file system, which may
result in some extra files being backed up, but there shouldn't be any
data loss.</p>
</li>
-
<li><strong><a name="free_space">Where does rdiff-backup need free
space and how much is required? What is the problem when rdiff-backup
says "<code>ValueError: Incorrect length of data
produced</code>"?</a></strong>
-
<p>When backing up, rdiff-backup needs free space in the mirror
directory. The amount of free space required is usually a bit more
than the size of the file getting backed up, but can be as much as
@@ -374,21 +321,37 @@
<code>rdiff-backup foo bar</code> and the largest file,
<code>foo/largefile</code>, was 1GB. Then rdiff-backup would need
1+GB of free space in the <code>bar</code> directory.</p>
-
-<p>When restoring, rdiff-backup needs free space in the default temp
+<p>When restoring or regressing, rdiff-backup needs free space in the default
temp
directory. Under unix systems this is usually the <code>/tmp</code>
-directory---see the entry for <code>tempfile.tempdir</code> in the <a
+directory. The temp directory that rdiff-backup uses can be set using the
+<code>--tempdir</code> and <code>--remote-tempdir</code> options available
+in versions 1.1.13 and newer. See the entry for <code>tempfile.tempdir</code>
in the <a
href="http://www.python.org/doc/2.4.1/lib/module-tempfile.html";>Python
tempfile docs</a> for more
information on the default temp directory. The amount of free space
required can vary, but it usually about the size of the largest file
being restored.</p>
-
<p>Usually free space errors are intelligible, like <code>IOError:
[Errno 28] No space left on device</code> or similar. However, do to
a gzip quirk they may look like <code>ValueError: Incorrect length of data
produced</code>.</p>
-
</li>
-
-
+<li><strong><a name="librsync_bug">What does "internal error: job made
+no progress" mean?</a></strong>
+<p>This error happens due to a bug in <code>librsync</code> that prevents
+it from handling files greater than 4 GB in some situations, such as
+when transferring between a 32-bit host and a 64-bit host.
+<a
href="https://sourceforge.net/tracker/index.php?func=detail&aid=1439412&group_id=56125&atid=479441";>A
patch is available</a>
+from the librsync project page on Sourceforge. The
+<a href="https://sourceforge.net/cvs/?group_id=56125";>CVS version</a> of
librsync
+also contains the patch. More information is also available in
+<a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=355178";>Debian bug
report #355178</a>.</p>
+</li>
+<li><strong><a name="path">Why does rdiff-backup say it's not in my $PATH? It
is when I login!</a></strong>
+<p>If you get an error like <code>sh: line1: rdiff-backup: command not
found</code>, but rdiff-backup
+<i>is</i> in your <code>$PATH</code> when you login to the remote host, it is
happening because the
+value of bash's <code>$PATH</code> is set differently when you login to an
interactive shell than when you run a command remotely via SSH. For more
+information, read the <a href="http://linux.die.net/man/1/bash";>bash
manpage</a> and look at your
+<code>.bashrc</code> and <code>.bash_profile</code> files.</p>
+<p>In particular, this can happen if rdiff-backup was installed via Fink on a
remote Mac OS X system. <code>/sw/bin</code> is magically added to your
<code>$PATH</code> by the script <code>/sw/bin/init.sh</code> when you login
with an interative shell. Fink did this behind the scenes when you set it up.
Simply add <code>/sw/bin</code> to your path manually, or copy rdiff-backup to
a directory that is in your <code>$PATH</code>.</p>
+</li>
</ol>
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Rdiff-backup-commits] rdiff-backup FAQ-body.html,
Andrew Ferguson <=