working with locally modified or augmented gnulib repositories

[Bruno Haible]

Hi all,

So far, it was easy for users of gnulib-tool to use the modules unmodified,
but difficult to experiment with modifications of gnulib modules, and even
harder to keep modified versions of gnulib modules for a long time.

Multi-level source control versioning systems (SCMs) like mercurial,
monotone, git/cogito, bzr would help a lot doing so during the development
process, but
  1) they are not yet widely supported, e.g. in savannah,
  2) they don't address the distribution issue. When a package foobar
     needs a modified version of, say, stdint_.h, it either has to
     put a comment into foobar/autogen.sh saying "Attention! This doesn't
     work with a pristine gnulib, you need this and that patch after
     checking out gnulib", or it has to use the --avoid=stdint option
     and provide the modified stdint module in a different directory
     and use libtool convenience libraries.

To solve this problem, I'm adding a --local-dir option to gnulib. It allows
the package to override or augment gnulib. This means:
  - You can store files that are to override gnulib files.
    For example, I can add comments to my copy of xalloc.h although Paul
    and Jim don't run after comments in .h files.
  - You can likewise override a module description.
    For example, I use a completely different 'closeout' implementation
    than Jim.
  - You can store context diffs to be applied to gnulib files.
    For example, I patch the variable declarations in error.h, adding a
    DLL_VARIABLE marker, that is needed for portability to mingw and cygwin
    under certain circumstances.
  - You can add modules of your own, that are not (yet) in gnulib.
  - You can also add unstructured amounts of code to the library, by grouping
    the non-gnulib files of the library in a single kitchen-sink "module".
And in a release tarball, you can distribute the contents of this --local-dir
directory that - barring incompatible changes in gnulib - will be combinable
with newer versions of gnulib.

Please make wise use of this option. It allows you to easily hold back
modifications you make to gnulib macros (where it may be better to share
them). On the other hand, it allows you to create your own pool of modules
in a "staging area" for gnulib.

Bruno


2006-07-29  Bruno Haible  <address@hidden>

        Make it possible for gnulib-tool to work with locally modified or
        augmented gnulib repositories.
        * gnulib-tool (func_usage): Document --local-dir option.
        (local_gnulib_dir): New variable.
        Handle --local-dir option.
        (func_lookup_file): New function.
        (func_all_modules, func_verify_module): Look also in $local_gnulib_dir.
        (func_get_description, func_get_filelist, func_get_description,
        func_get_filelist, func_get_dependencies, func_get_autoconf_snippet,
        func_get_automake_snippet, func_get_include_directive,
        func_get_license, func_get_maintainer): Use func_lookup_file.
        (func_import, func_create_testdir): Use func_lookup_file.

*** gnulib-tool 29 Jul 2006 13:18:04 -0000      1.126
--- gnulib-tool 29 Jul 2006 13:21:19 -0000
***************
*** 111,116 ****
--- 111,118 ----
                              For --import, this specifies where your
                              configure.ac can be found.  Defaults to current
                              directory.
+       --local-dir=DIRECTORY  Specify a local override directory where to look
+                             up files before looking in gnulib's directory.
  
  Options for --import:
        --lib=LIBRARY         Specify the library name.  Defaults to 'libgnu'.
***************
*** 261,266 ****
--- 263,270 ----
  # - autoconf_minversion  minimum supported autoconf version
  # - do_changelog    false if --no-changelog was given, : otherwise
  # - doit            : if actions shall be executed, false if only to be 
printed
+ # - local_gnulib_dir  from --local-dir
+ # - symbolic        true if --symbolic was given, blank otherwise
  {
    mode=
    destdir=
***************
*** 279,284 ****
--- 283,289 ----
    autoconf_minversion=
    do_changelog=:
    doit=:
+   local_gnulib_dir=
    symbolic=
  
    supplied_opts="$@"
***************
*** 416,421 ****
--- 421,436 ----
        --dry-run )
          doit=false
          shift ;;
+       --local-dir )
+         shift
+         if test $# = 0; then
+           func_fatal_error "missing argument for --local-dir"
+         fi
+         local_gnulib_dir=$1
+         shift ;;
+       --local-dir=* )
+         local_gnulib_dir=`echo "X$1" | sed -e 's/^X--local-dir=//'`
+         shift ;;
        -s | --symbolic | --symboli | --symbol | --symbo | --symb | --symlink | 
--symlin | --symli | --syml | --sym | --sy )
          symbolic=true
          shift ;;
***************
*** 518,540 ****
  done
  gnulib_dir=`echo "$self_abspathname" | sed -e 's,/[^/]*$,,'`
  
  # func_all_modules
  func_all_modules ()
  {
    # Filter out metainformation files like README, which are not modules.
    # Filter out unit test modules; they can be retrieved through
    # --extract-tests-module if desired.
!   (cd "$gnulib_dir/modules" && ls -1) \
        | sed -e '/^CVS$/d' -e '/^ChangeLog$/d' -e '/^README$/d' -e 
'/^TEMPLATE$/d' -e '/^TEMPLATE-TESTS$/d' -e '/~$/d' \
        | sed -e '/-tests$/d' \
!       | LC_ALL=C sort
  }
  
  # func_verify_module
  # verifies a module name
  func_verify_module ()
  {
!   if test ! -f "$gnulib_dir/modules/$module" \
       || test "CVS" = "$module" \
       || test "ChangeLog" = "$module" \
       || test "README" = "$module" \
--- 533,598 ----
  done
  gnulib_dir=`echo "$self_abspathname" | sed -e 's,/[^/]*$,,'`
  
+ func_tmpdir
+ trap 'rm -rf "$tmp"' 0 1 2 3 15
+ 
+ # func_lookup_file file
+ # looks up a file in $local_gnulib_dir or $gnulib_dir, or combines it through
+ # 'patch'.
+ # Output:
+ # - lookedup_file   name of the merged (combined) file
+ # - lookedup_tmp    true if it is located in the tmp directory, blank 
otherwise
+ func_lookup_file ()
+ {
+   lkfile="$1"
+   if test -n "$local_gnulib_dir" && test -f "$local_gnulib_dir/$lkfile"; then
+     lookedup_file="$local_gnulib_dir/$lkfile"
+     lookedup_tmp=
+   else
+     if test -f "$gnulib_dir/$lkfile"; then
+       if test -n "$local_gnulib_dir" && test -f 
"$local_gnulib_dir/$lkfile.diff"; then
+         lkbase=`echo "$lkfile" | sed -e 's,^.*/,,'`
+         rm -f "$tmp/$lkbase"
+         cp "$gnulib_dir/$lkfile" "$tmp/$lkbase"
+         patch -s "$tmp/$lkbase" < "$local_gnulib_dir/$lkfile.diff" \
+           || func_fatal_error "patch file $local_gnulib_dir/$lkfile.diff 
didn't apply cleanly"
+         lookedup_file="$tmp/$lkbase"
+         lookedup_tmp=true
+       else
+         lookedup_file="$gnulib_dir/$lkfile"
+         lookedup_tmp=
+       fi
+     else
+       func_fatal_error "file $gnulib_dir/$lkfile not found"
+     fi
+   fi
+ }
+ 
  # func_all_modules
  func_all_modules ()
  {
    # Filter out metainformation files like README, which are not modules.
    # Filter out unit test modules; they can be retrieved through
    # --extract-tests-module if desired.
!   {
!     (cd "$gnulib_dir/modules" && ls -1)
!     if test -n "$local_gnulib_dir" && test -d "$local_gnulib_dir/modules"; 
then
!       (cd "$local_gnulib_dir/modules" && ls -1 | sed -e 's,\.diff$,,')
!     fi
!   } \
        | sed -e '/^CVS$/d' -e '/^ChangeLog$/d' -e '/^README$/d' -e 
'/^TEMPLATE$/d' -e '/^TEMPLATE-TESTS$/d' -e '/~$/d' \
        | sed -e '/-tests$/d' \
!       | LC_ALL=C sort \
!       | LC_ALL=C uniq
  }
  
  # func_verify_module
  # verifies a module name
  func_verify_module ()
  {
!   if ! { test -f "$gnulib_dir/modules/$module" \
!          || { test -n "$local_gnulib_dir" && test -d 
"$local_gnulib_dir/modules" \
!               && test -f "$local_gnulib_dir/modules/$module"; }; } \
       || test "CVS" = "$module" \
       || test "ChangeLog" = "$module" \
       || test "README" = "$module" \
***************
*** 585,597 ****
  # func_get_description module
  func_get_description ()
  {
!   sed -n -e "/^Description$sed_extract_prog" < "$gnulib_dir/modules/$1"
  }
  
  # func_get_filelist module
  func_get_filelist ()
  {
!   sed -n -e "/^Files$sed_extract_prog" < "$gnulib_dir/modules/$1"
    case "$autoconf_minversion" in
      2.59)
        #echo m4/onceonly.m4
--- 643,657 ----
  # func_get_description module
  func_get_description ()
  {
!   func_lookup_file "modules/$1"
!   sed -n -e "/^Description$sed_extract_prog" < "$lookedup_file"
  }
  
  # func_get_filelist module
  func_get_filelist ()
  {
!   func_lookup_file "modules/$1"
!   sed -n -e "/^Files$sed_extract_prog" < "$lookedup_file"
    case "$autoconf_minversion" in
      2.59)
        #echo m4/onceonly.m4
***************
*** 606,643 ****
    # ${module}-tests always implicitly depends on ${module}.
    echo "$1" | sed -n -e 's/-tests//p'
    # Then the explicit dependencies listed in the module description.
!   sed -n -e "/^Depends-on$sed_extract_prog" < "$gnulib_dir/modules/$1"
  }
  
  # func_get_autoconf_snippet module
  func_get_autoconf_snippet ()
  {
!   sed -n -e "/^configure\.ac$sed_extract_prog" < "$gnulib_dir/modules/$1"
  }
  
  # func_get_automake_snippet module
  func_get_automake_snippet ()
  {
!   sed -n -e "/^Makefile\.am$sed_extract_prog" < "$gnulib_dir/modules/$1"
  }
  
  # func_get_include_directive module
  func_get_include_directive ()
  {
!   sed -n -e "/^Include$sed_extract_prog" < "$gnulib_dir/modules/$1" | \
    sed -e 's/^\(["<]\)/#include \1/'
  }
  
  # func_get_license module
  func_get_license ()
  {
!   sed -n -e "/^License$sed_extract_prog" < "$gnulib_dir/modules/$1"
  }
  
  # func_get_maintainer module
  func_get_maintainer ()
  {
!   sed -n -e "/^Maintainer$sed_extract_prog" < "$gnulib_dir/modules/$1"
  }
  
  # func_get_tests_module module
--- 666,709 ----
    # ${module}-tests always implicitly depends on ${module}.
    echo "$1" | sed -n -e 's/-tests//p'
    # Then the explicit dependencies listed in the module description.
!   func_lookup_file "modules/$1"
!   sed -n -e "/^Depends-on$sed_extract_prog" < "$lookedup_file"
  }
  
  # func_get_autoconf_snippet module
  func_get_autoconf_snippet ()
  {
!   func_lookup_file "modules/$1"
!   sed -n -e "/^configure\.ac$sed_extract_prog" < "$lookedup_file"
  }
  
  # func_get_automake_snippet module
  func_get_automake_snippet ()
  {
!   func_lookup_file "modules/$1"
!   sed -n -e "/^Makefile\.am$sed_extract_prog" < "$lookedup_file"
  }
  
  # func_get_include_directive module
  func_get_include_directive ()
  {
!   func_lookup_file "modules/$1"
!   sed -n -e "/^Include$sed_extract_prog" < "$lookedup_file" | \
    sed -e 's/^\(["<]\)/#include \1/'
  }
  
  # func_get_license module
  func_get_license ()
  {
!   func_lookup_file "modules/$1"
!   sed -n -e "/^License$sed_extract_prog" < "$lookedup_file"
  }
  
  # func_get_maintainer module
  func_get_maintainer ()
  {
!   func_lookup_file "modules/$1"
!   sed -n -e "/^Maintainer$sed_extract_prog" < "$lookedup_file"
  }
  
  # func_get_tests_module module
***************
*** 1152,1159 ****
      fi
    fi
  
-   func_tmpdir
-   trap 'rm -rf "$tmp"' 0 1 2 3 15
    # func_dest_tmpfilename file
    # determines the name of a temporary file (file is relative to destdir).
    # Sets variable:
--- 1218,1223 ----
***************
*** 1213,1226 ****
    func_add_or_update ()
    {
      func_dest_tmpfilename "$g"
!     cp "$gnulib_dir/$f" "$tmpfile" || func_fatal_error "failed"
      if test -n "$lgpl"; then
        # Update license.
        case "$f" in
          lib/*)
            sed -e 's/GNU General/GNU Lesser General/g' \
                -e 's/version 2\([ ,]\)/version 2.1\1/g' \
!             < "$gnulib_dir/$f" > "$tmpfile" || func_fatal_error "failed"
            ;;
        esac
      fi
--- 1277,1291 ----
    func_add_or_update ()
    {
      func_dest_tmpfilename "$g"
!     func_lookup_file "$f"
!     cp "$lookedup_file" "$tmpfile" || func_fatal_error "failed"
      if test -n "$lgpl"; then
        # Update license.
        case "$f" in
          lib/*)
            sed -e 's/GNU General/GNU Lesser General/g' \
                -e 's/version 2\([ ,]\)/version 2.1\1/g' \
!             < "$lookedup_file" > "$tmpfile" || func_fatal_error "failed"
            ;;
        esac
      fi
***************
*** 1237,1244 ****
              echo "Replacing file $g (non-gnulib code backuped in ${g}~) !!"
            fi
            mv -f "$destdir/$g" "$destdir/${g}~" || func_fatal_error "failed"
!           if test -n "$symbolic" && cmp "$gnulib_dir/$f" "$tmpfile" > 
/dev/null; then
!             func_ln_if_changed "$gnulib_dir/$f" "$destdir/$g"
            else
              mv -f "$tmpfile" "$destdir/${g}" || func_fatal_error "failed"
            fi
--- 1302,1310 ----
              echo "Replacing file $g (non-gnulib code backuped in ${g}~) !!"
            fi
            mv -f "$destdir/$g" "$destdir/${g}~" || func_fatal_error "failed"
!           if test -n "$symbolic" && test -z "$lookedup_tmp" \
!              && cmp "$lookedup_file" "$tmpfile" > /dev/null; then
!             func_ln_if_changed "$lookedup_file" "$destdir/$g"
            else
              mv -f "$tmpfile" "$destdir/${g}" || func_fatal_error "failed"
            fi
***************
*** 1256,1263 ****
        # frequently that developers don't put autogenerated files into CVS.
        if $doit; then
          echo "Copying file $g"
!         if test -n "$symbolic" && cmp "$gnulib_dir/$f" "$tmpfile" > 
/dev/null; then
!           func_ln_if_changed "$gnulib_dir/$f" "$destdir/$g"
          else
            mv -f "$tmpfile" "$destdir/${g}" || func_fatal_error "failed"
          fi
--- 1322,1330 ----
        # frequently that developers don't put autogenerated files into CVS.
        if $doit; then
          echo "Copying file $g"
!         if test -n "$symbolic" && test -z "$lookedup_tmp" \
!            && cmp "$lookedup_file" "$tmpfile" > /dev/null; then
!           func_ln_if_changed "$lookedup_file" "$destdir/$g"
          else
            mv -f "$tmpfile" "$destdir/${g}" || func_fatal_error "failed"
          fi
***************
*** 1524,1538 ****
      fi
    fi
  
-   rm -rf "$tmp"
-   # Undo the effect of the previous 'trap' command. Some shellology:
-   # We cannot use "trap - 0 1 2 3 15", because Solaris sh would attempt to
-   # execute the command "-". "trap '' ..." is fine only for signal 0 (= normal
-   # exit); for the others we need to call 'exit' explicitly. The value of $? 
is
-   # 128 + signal number and is set before the trap-registered command is run.
-   trap '' 0
-   trap 'exit $?' 1 2 3 15
- 
    echo "Finished."
    echo
    echo "You may need to add #include directives for the following .h files."
--- 1591,1596 ----
***************
*** 1612,1626 ****
  
    # Copy files or make symbolic links.
    for f in $files; do
      case "$f" in
        build-aux/*) g=`echo "$f" | sed -e "s,^build-aux/,$auxdir/,"` ;;
        *) g="$f" ;;
      esac
!     ln "$gnulib_dir/$f" "$testdir/$g" 2>/dev/null ||
!     if test -z "$symbolic"; then
!       cp -p "$gnulib_dir/$f" "$testdir/$g"
      else
!       ln -s "$gnulib_dir/$f" "$testdir/$g"
      fi
    done
  
--- 1670,1689 ----
  
    # Copy files or make symbolic links.
    for f in $files; do
+     func_lookup_file "$f"
      case "$f" in
        build-aux/*) g=`echo "$f" | sed -e "s,^build-aux/,$auxdir/,"` ;;
        *) g="$f" ;;
      esac
!     if test -n "$lookedup_tmp"; then
!       cp -p "$lookedup_file" "$testdir/$g"
      else
!       ln "$lookedup_file" "$testdir/$g" 2>/dev/null ||
!       if test -z "$symbolic"; then
!         cp -p "$lookedup_file" "$testdir/$g"
!       else
!         ln -s "$lookedup_file" "$testdir/$g"
!       fi
      fi
    done
  
***************
*** 2176,2179 ****
--- 2239,2251 ----
      func_fatal_error "unknown operation mode --$mode" ;;
  esac
  
+ rm -rf "$tmp"
+ # Undo the effect of the previous 'trap' command. Some shellology:
+ # We cannot use "trap - 0 1 2 3 15", because Solaris sh would attempt to
+ # execute the command "-". "trap '' ..." is fine only for signal 0 (= normal
+ # exit); for the others we need to call 'exit' explicitly. The value of $? is
+ # 128 + signal number and is set before the trap-registered command is run.
+ trap '' 0
+ trap 'exit $?' 1 2 3 15
+ 
  exit 0