Re: BTRFS file clone support for cp

[Giuseppe Scrivano]

Jim Meyering <address@hidden> writes:

> I am now convinced that cp's new behavior belongs on
> a separate option, --reflink (i.e., it should not be the default).
> Giuseppe, do you feel like adding that option and adjusting your
> test accordingly?

I attached two separate patches, --reflink option and file-clone test.
Last versions of btrfs have a bug (I asked on #btrfs and they confirmed
it), btrfs doesn't use correctly all the free space available.  In fact
I get ENOSPC while in reality only 54% is used.  Probably it is better
to postpone the second patch inclusion after the bug is fixed.

Another note, I changed this line in the NEWS file:
-  "when both the source and destination are on the same btrfs partition."

considering that BTRFS supports multiple devices I am not convinced that
it is always true, I guess source and destination could be on different
partitions, though I couldn't find a clear answer on the btrfs wiki to
this question.


Any comment?

Thanks,
Giuseppe



>From d110badaf7583acf957477bc7eda2e212b404343 Mon Sep 17 00:00:00 2001
From: Giuseppe Scrivano <address@hidden>
Date: Sat, 1 Aug 2009 19:36:48 +0200
Subject: [PATCH 1/2] cp: accept the --reflink option

* NEWS: Mention it.
* doc/coreutils.texi: Likewise.
* src/copy.h (struct cp_options): New member reflink.
* src/copy.c (usage): Likewise.
(copy_reg): If reflink is true try to clone the file.
(main): Check for --reflink.
(cp_option_init): By default set reflink to false.
---
 NEWS               |    4 ++--
 doc/coreutils.texi |    9 +++++++++
 src/copy.c         |    5 +++--
 src/copy.h         |    3 +++
 src/cp.c           |   10 +++++++++-
 5 files changed, 26 insertions(+), 5 deletions(-)

diff --git a/NEWS b/NEWS
index 80c60e2..8d6d7a6 100644
--- a/NEWS
+++ b/NEWS
@@ -35,8 +35,8 @@ GNU coreutils NEWS                                    -*- 
outline -*-
 
   chroot now accepts the options --userspec and --groups.
 
-  cp, install, mv: take advantage of btrfs' O(1) copy-on-write feature
-  when both the source and destination are on the same btrfs partition.
+  cp accepts a new option, --reflink: attempt a copy-on-write (COW)
+  when the file system supports it.
 
   sort accepts a new option, --human-numeric-sort (-h): sort numbers
   while honoring human readable suffixes like KiB and MB etc.
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index acec76e..03c9eb7 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -7543,6 +7543,15 @@ Also, it is not portable to use @option{-R} to copy 
symbolic links
 unless you also specify @option{-P}, as @acronym{POSIX} allows
 implementations that dereference symbolic links by default.
 
address@hidden --reflink
address@hidden --reflink
+Attempt a O(1) copy-on-write (COW) when the underlying file system
+supports this operation instead of really copying the file.
+The source and destination files share the same disk data blocks until
+they are equal.  Changes done in a file are not visible in the other
+one because shared blocks will be duplicated before these changes are
+stored.
+
 @item --remove-destination
 @opindex --remove-destination
 Remove each existing destination file before attempting to open it
diff --git a/src/copy.c b/src/copy.c
index bbed336..02d36f3 100644
--- a/src/copy.c
+++ b/src/copy.c
@@ -610,10 +610,11 @@ copy_reg (char const *src_name, char const *dst_name,
       goto close_src_and_dst_desc;
     }
 
-  /* If --sparse=auto is in effect, attempt a btrfs clone operation.
+  /* Attempt a clone operation.  It is possible only when --sparse=auto
+     is in effect.
      If the operation is not supported or it fails then copy the file
      in the usual way.  */
-  bool copied = (x->sparse_mode == SPARSE_AUTO
+  bool copied = (x->reflink && x->sparse_mode == SPARSE_AUTO
                  && clone_file (dest_desc, source_desc) == 0);
 
   if (!copied)
diff --git a/src/copy.h b/src/copy.h
index 8e0b408..ddf4f4e 100644
--- a/src/copy.h
+++ b/src/copy.h
@@ -219,6 +219,9 @@ struct cp_options
      such a symlink) and returns false.  */
   bool open_dangling_dest_symlink;
 
+  /* If true, attempt to clone the file instead of copying it.  */
+  bool reflink;
+
   /* This is a set of destination name/inode/dev triples.  Each such triple
      represents a file we have created corresponding to a source file name
      that was specified on the command line.  Use it to avoid clobbering
diff --git a/src/cp.c b/src/cp.c
index 8785076..63c07d4 100644
--- a/src/cp.c
+++ b/src/cp.c
@@ -78,7 +78,8 @@ enum
   PRESERVE_ATTRIBUTES_OPTION,
   SPARSE_OPTION,
   STRIP_TRAILING_SLASHES_OPTION,
-  UNLINK_DEST_BEFORE_OPENING
+  UNLINK_DEST_BEFORE_OPENING,
+  REFLINK_OPTION
 };
 
 /* True if the kernel is SELinux enabled.  */
@@ -121,6 +122,7 @@ static struct option const long_opts[] =
   {"recursive", no_argument, NULL, 'R'},
   {"remove-destination", no_argument, NULL, UNLINK_DEST_BEFORE_OPENING},
   {"sparse", required_argument, NULL, SPARSE_OPTION},
+  {"reflink", no_argument, NULL, REFLINK_OPTION},
   {"strip-trailing-slashes", no_argument, NULL, STRIP_TRAILING_SLASHES_OPTION},
   {"suffix", required_argument, NULL, 'S'},
   {"symbolic-link", no_argument, NULL, 's'},
@@ -194,6 +196,7 @@ Mandatory arguments to long options are mandatory for short 
options too.\n\
                                  attempting to open it (contrast with 
--force)\n\
 "), stdout);
       fputs (_("\
+      --reflink                clone the file if it is possible\n\
       --sparse=WHEN            control creation of sparse files\n\
       --strip-trailing-slashes  remove any trailing slashes from each SOURCE\n\
                                  argument\n\
@@ -752,6 +755,7 @@ cp_option_init (struct cp_options *x)
   x->interactive = I_UNSPECIFIED;
   x->move_mode = false;
   x->one_file_system = false;
+  x->reflink = false;
 
   x->preserve_ownership = false;
   x->preserve_links = false;
@@ -916,6 +920,10 @@ main (int argc, char **argv)
                                     sparse_type_string, sparse_type);
          break;
 
+       case REFLINK_OPTION:
+         x.reflink = true;
+         break;
+
        case 'a':               /* Like -dR --preserve=all with reduced failure 
diagnostics. */
          x.dereference = DEREF_NEVER;
          x.preserve_links = true;
-- 
1.6.3.3


>From 5271957a8ff23df273e975033dabaf7b93e2606f Mon Sep 17 00:00:00 2001
From: Giuseppe Scrivano <address@hidden>
Date: Sat, 1 Aug 2009 19:40:09 +0200
Subject: [PATCH 2/2] tests: add new test for copy-on-write (COW) cp support.

* tests/Makefile.am: Consider the new test.
* tests/cp/file-clone: New file.
---
 tests/Makefile.am   |    1 +
 tests/cp/file-clone |   65 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 0 deletions(-)
 create mode 100644 tests/cp/file-clone

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 59737a0..9841aa3 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -20,6 +20,7 @@ EXTRA_DIST =          \
 
 root_tests =                                   \
   chown/basic                                  \
+  cp/file-clone                                \
   cp/cp-a-selinux                              \
   cp/preserve-gid                              \
   cp/special-bits                              \
diff --git a/tests/cp/file-clone b/tests/cp/file-clone
new file mode 100644
index 0000000..21cb61a
--- /dev/null
+++ b/tests/cp/file-clone
@@ -0,0 +1,65 @@
+#!/bin/sh
+# Make sure file-clone on a btrfs file system works properly.
+
+# Copyright (C) 2009 Free Software Foundation, Inc.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+if test "$VERBOSE" = yes; then
+  set -x
+  cp --version
+fi
+
+. $srcdir/test-lib.sh
+
+require_root_
+require_sparse_support_
+
+cleanup_(){ umount btrfs; }
+
+fail=0
+
+mkfs.btrfs --version || skip_test_ "btrfs userland tools not installed"
+
+# 256MB seems to be the minimum size for a btrfs with default parameters.
+truncate --size=256M btrfs.img  || framework_failure
+
+mkfs.btrfs btrfs.img  || framework_failure
+mkdir btrfs || framework_failure
+mount -t btrfs -o loop btrfs.img btrfs || framework_failure
+
+echo hello > btrfs/a.test || framework_failure
+cp --reflink btrfs/a.test btrfs/b.test || fail=1
+compare btrfs/a.test btrfs/b.test || fail=1
+
+# Be sure that files are different if one is modified.
+echo hello >> btrfs/a.test
+compare btrfs/a.test btrfs/b.test && fail=1
+
+dd bs=1M count=200 if=/dev/zero of=btrfs/alloc.test || framework_failure
+
+# If the file is cloned, only additional space for metadata is required.
+# Two 200MB files can be present even if the total file system space is 256MB.
+cp --reflink btrfs/alloc.test btrfs/clone.test || fail=1
+rm btrfs/clone.test
+
+# A normal copy should raise ENOSPC.
+cp btrfs/alloc.test btrfs/clone.test && fail=1
+
+# When --sparse={always,never} is used, the file is copied without any cloning.
+# Use --sparse=never to be sure the file is copied without holes and it is not
+# possible since there is not enough free space.
+cp --reflink --sparse=never btrfs/alloc.test btrfs/clone.test && fail=1
+
+Exit $fail
-- 
1.6.3.3