Re: [Quilt-dev] Another shell re-write of backup-files.

[Kaz Kylheku]

On Fri, 18 Mar 2011 09:50:18 -0700, Kaz Kylheku <address@hidden>
wrote:
> On Fri, 18 Mar 2011 10:18:27 +0100, Jean Delvare <address@hidden>
> wrote:
>> Hi Kaz,
>>
>> On Friday 18 March 2011 02:26:27 am Kaz Kylheku wrote:
>>> Hey everyone,
>>>
>>> Recently I became interested in a quilt that consists only of shell
>>> scripts.
>>
>> You're not the only one. I'm happy to see momentum grow in this
>> direction.
>>
> 
> I did some further hacking on this last night, sliding the program
> under quilt and getting it to work, including adding new files,
> and pushes and pops with file creating/deleting patches.

Plus a few more fixes. It's behaving well for me now. I'm ready
for stress-testing with big patch stacks.

There was a bug when the list of nonexistent files is empty: xargs was
calling touch with no arguments, resulting in an error.

Secondly, it turns out that quilt does invoke the "noop" case, so
it is handled properly: no usage is printed and returns success
(whether or not -L is specified).

I fixed the touch issue like this: quite simply, restore does
not use hard links at all. If -t is specified, it does a
copy without preserving timestamps. Otherwise a "cp -a".

This leaves the issue that "quilt pop -a" cannot be optimized
as nicely. (It's obvious that the "pop -a" operation, and
more generally, the "pop down to this patch" operation,
can be optimized by restoring file using hard links only
at each patch level, followed by a final pass which breaks
the hard links for all affected files that still exist after
the pop).

Index: quilt/backup-files
===================================================================
--- quilt.orig/backup-files
+++ quilt/backup-files
@@ -16,8 +16,9 @@
 #   pass-through mode).
 # - Use tree-to-tree recursive cp to restore a backup (taking
everything
 #   in the backup, ignoring the file list).
-# - To touch files on restore, if requested, we can do a find + xargs
+
-#   touch over the backup instead, prior to restoring it. [WRONG]
+# - We don't use hard linking when restoring files, because then we
have
+#   to break the hard links anyway (quilt issues a noop -L operation).
+# - To touch restored files, we can do a copy without preserving
timestamps.
 # - Added files (i.e. backups of nonexistent files) are represented as
a
 #   specially named file containing an explicit list, and not as
 #   zero-length files. This eases the implementation, and lets us back
@@ -209,7 +210,9 @@ B )
 
        # escaping issues here: we are assuming opt_prefix
        # doesn't contain # or things interpreted by sed
-  sed -e "s#.*#$opt_prefix&#" < $noex_list | xargs touch
+       if [ -s $noex_list ] ; then
+               sed -e "s#.*#$opt_prefix&#" < $noex_list | xargs touch
+       fi
        ;;
 R )
        if [ -z $all_backup_files ] ; then
@@ -217,14 +220,10 @@ R )
                exit 1
        fi
 
-       if [ $opt_nolink ] ; then
-               cp -a "$opt_prefix"/. .
+       if [ $opt_touch ] ; then
+               cp -dR --preserve=mode,ownership "$opt_prefix"/. .
        else
-               if [ $opt_touch ] ; then
-                       find "$opt_prefix" -type f | xargs touch
-               fi
-
-               cp -rlf "$opt_prefix"/. .
+               cp -a "$opt_prefix"/. .
        fi
 
        find "$opt_prefix" -type f -size 0c \
@@ -238,9 +237,11 @@ R )
 D )
        rm -rf "$opt_prefix"
        ;;
+'' )
+  # noop
+       ;;
 * )
-       # either no operation was specified, or multiple operations
-       # were been specified, like -b and -r.
+       # multiple operations were specified together, like -b and -r.
        usage
        exit 1
        ;;



Full script:

#!/bin/bash

#
# Shell script replacement for quilt's backup-files utility
# Tested to some extent under quilt.
# Mar 18, 2011
# Kaz Kylheku <address@hidden>
#

#
# Main concepts:
#
# - Goal is to avoid invoking a process for each file name
# - We use the CPIO utility for creating hard-linked backups; CPIO
#   pass-through mode can take list of names and create hard links (in
#   pass-through mode).
# - Use tree-to-tree recursive cp to restore a backup (taking
everything
#   in the backup, ignoring the file list).
# - We don't use hard linking when restoring files, because then we
have
#   to break the hard links anyway (quilt issues a noop -L operation).
# - To touch restored files, we can do a copy without preserving
timestamps.
# - Added files (i.e. backups of nonexistent files) are represented as
a
#   specially named file containing an explicit list, and not as
#   zero-length files. This eases the implementation, and lets us back
#   up/restore zero-length files!
#

set -eu # bail on any errors, and unbound variable uses

opt_prefix=
opt_suffix=
opt_file=
opt_backup=
opt_restore=
opt_remove_backup=
opt_keep_backups=
opt_silent=
opt_touch=
opt_nolink=

all_backup_files=

usage()
{
cat <<!
Usage: $0 [-B prefix] [-f {file|-}] [-sktL] [-b|-r|-x] {file|-} ...

        Create hard linked backup copies of a list of files
        read from a file (or standard input), or from the
        argument list.

        The argument list "-" means "every regular file in
        the backup directory specified by the -B prefix.

        -b      Create backup
        -r      Restore the backup
        -x      Remove backup files and empty parent directories
        -k      When doing a restore, keep the backup files
        -B      Path name prefix for backup files. Should have trailing slash.
                Without a trailing slash, the behavior is different from the
                C implementation.
        -z      Unsupported, obsolete option
        -s      Silent operation; only print error messages
        -f      Read the filenames to process from file (- = standard input)
        -t      Touch original files after restore (update their mtimes)

        -L      Ensure that when finished, the source file has a link count of 1
!
}

if ! options=`getopt -o B:f:brxkstLh -- "$@"` ; then
        usage
        exit 1
fi

eval set -- "$options"

while true
do
        case "$1" in
        -B)
                opt_prefix="$2"
                shift 2 ;;
        -f)
                opt_file="$2"
                shift 2 ;;
        -b)
                opt_backup=B
                shift ;;
        -r)
                opt_restore=R
                shift ;;
        -x)
                opt_remove_backup=D
                shift ;;
        -k)
                opt_keep_backups=y
                shift ;;
        -s)
                opt_silent=y
                shift ;;
        -t)
                opt_touch=y
                shift ;;
        -L)
                opt_nolink=y
                shift ;;
        -h)
                usage
                exit 0 ;;
        --)
                shift
                break ;;
        esac
done

if [ $# -eq 0 -a -z "$opt_file" ] ; then
        echo "Error: specify input file names as arguments or via -f option"
        echo
        usage
        exit 1
fi

if [ $# -ge 1 -a -n "$opt_file" ] ; then
        echo "Error: conflict: both -f and file name argument given"
        echo
        usage
        exit 1
fi

if [ $# -gt 1 -a "$1" == "-" ] ; then
        echo "Error: if - is specified, then no other arguments can be added"
        echo
        usage
        exit 1
fi


if [ -z "$opt_prefix" ] ; then
        echo "Error: specify backup/restore directory with -B"
        echo
        usage
        exit 1
fi

# temp file that holds raw list of names
temp_list=$(mktemp "${TMP_DIR:-/tmp}/backup-files-tl-XXXXXX")

# temp file that holds names of files that exist
file_list=$(mktemp "${TMP_DIR:-/tmp}/backup-files-fl-XXXXXX")

# temp file that holds names of nonexistent files
noex_list=$(mktemp "${TMP_DIR:-/tmp}/backup-files-ne-XXXXXX")

cleanup()
{
        rm -f $temp_list $file_list $noex_list
}

trap cleanup exit

#
# capture the file list into the $temp_list file
#

if [ -n "$opt_file" -a "$opt_file" == "-" ] ; then
        cat > $temp_list
elif [ -n "$opt_file" ] ; then
        cat "$opt_file" > $temp_list
elif [ "$1" == "-" ] ; then
        all_backup_files=y
else
        # IFS trick. The string literal here contains a newline
        ( IFS="
"
          echo "$*" > $temp_list )
fi

#
# separate name list into existing and nonexisting
#

> $file_list
> $noex_list

while read name ; do
        if [ -e "$name" ] ; then
                echo "$name" >> $file_list
        else
                echo "$name" >> $noex_list
        fi
done < $temp_list

case $opt_backup$opt_restore$opt_remove_backup in
B )
        if [ $all_backup_files ] ; then
                echo "Error: \"-\" argument makes no sense for backup action"
                echo
                usage
                exit 1
        fi

        if [ $opt_nolink ] ; then
                cpio --quiet -pd "$opt_prefix" < $file_list
        else
                cpio --quiet -pdl "$opt_prefix" < $file_list
        fi

        umask 0111 # so touched files are 0666

        # escaping issues here: we are assuming opt_prefix
        # doesn't contain # or things interpreted by sed
        if [ -s $noex_list ] ; then
                sed -e "s#.*#$opt_prefix&#" < $noex_list | xargs touch
        fi
        ;;
R )
        if [ -z $all_backup_files ] ; then
                echo "Error: restoring individual files is not supported"
                exit 1
        fi

        if [ $opt_touch ] ; then
                cp -dR --preserve=mode,ownership "$opt_prefix"/. .
        else
                cp -a "$opt_prefix"/. .
        fi

        find "$opt_prefix" -type f -size 0c \
          | cut -c "$((1+${#opt_prefix}))-" \
          | xargs rm -f

        if [ -z "$opt_keep_backups" ] ; then
                rm -rf "$opt_prefix"
        fi
        ;;
D )
        rm -rf "$opt_prefix"
        ;;
'' )
  # noop
        ;;
* )
        # multiple operations were specified together, like -b and -r.
        usage
        exit 1
        ;;
esac