Re: Creating subdirs of distdir

[Ralf Wildenhues]

Hi Stepan,

* Stepan Kasal wrote on Tue, May 23, 2006 at 03:29:01PM CEST:
> On Sun, May 14, 2006 at 07:32:57PM +0200, Ralf Wildenhues wrote:
> > Hmm.  You could "sort -u" the list of directories to create at "make
> > dist" time.
> 
> nice idea.  Attached please find a patch which implements it.

Thanks for doing the hard work!  This is cool stuff.

I think we can kill the per-file forks before the mkdir_p, too.  Both
combined kills about half the time of `distdir' for me (measured in
minutes for some packages; but automake/tests alone is impressive
enough); of course your part of the change makes for the bulk of the
improvement.

Killing the per-file forks in the actual copying part of distdir seems
rather difficult to me (without changing the semantics).  Not completely
sure though.  If possible, it could make `distdir' really fly.

A note to the strip sed expressions:  Based on this observation:
http://article.gmane.org/gmane.comp.sysutils.autoconf.general/7986
I did not add `|' to the list of characters to escape, although it is
later used as delimiter.  We need to either change the delimiter (and
add that to the list of characters to escape, iff safe to do so), or
just keep it this way and pray nobody uses `|' in a file name.  I think
it's pretty safe to assume this.  Furthermore, sed will usually complain
when it sees an extra delimiter, so this is better than a misinterpreted
alternation operator that could lead to silent data loss.

The testsuite passes with this on GNU/Linux.

Cheers,
Ralf

2006-05-23  Stepan Kasal  <address@hidden>
            Ralf Wildenhues <address@hidden>

        * lib/am/distdir.am: Do not call $(mkdir_p) for each
        distributed file, collect them and create them in one run,
        and strip $(srcdir) and $(top_srcdir) all at once.
        Fix some comment typos.

Index: lib/am/distdir.am
===================================================================
RCS file: /cvs/automake/automake/lib/am/distdir.am,v
retrieving revision 1.64
diff -u -r1.64 distdir.am
--- lib/am/distdir.am   19 Mar 2006 05:04:28 -0000      1.64
+++ lib/am/distdir.am   23 May 2006 19:00:20 -0000
@@ -58,9 +58,8 @@
 ?DISTDIRS?     $(mkdir_p) %DISTDIRS%
 ##
 ##
-       @srcdirstrip=`echo "$(srcdir)" | sed 's|.|.|g'`; \
-       topsrcdirstrip=`echo "$(top_srcdir)" | sed 's|.|.|g'`; \
-       list='$(DISTFILES)'; for file in $$list; do \
+       @srcdirstrip=`echo "$(srcdir)" | sed 's/[.[^$$\\*]/\\\\&/g'`; \
+       topsrcdirstrip=`echo "$(top_srcdir)" | sed 's/[.[^$$\\*]/\\\\&/g'`; \
 ##
 ## Yet another hack to support SUN make.
 ##
@@ -75,7 +74,7 @@
 ## So let's strip this leading $(srcdir)/ when it exists.  (As far we
 ## know, only SUN make and OSF1/Tru64 make add it.)  Searching whether
 ## the file is to be found in the source or build directory will be
-## done latter.
+## done later.
 ##
 ## In case we are _not_ using SUN or OSF1/Tru64 make, how can we be sure
 ## we are not stripping a legitimate filename that starts with the
@@ -99,13 +98,25 @@
 ## above the current directory.  Fortunately, apart from auxdir files which
 ## can be located in .. or ../.., this situation hardly occurs in practice.
 ##
-         case $$file in \
-           $(srcdir)/*) file=`echo "$$file" | sed "s|^$$srcdirstrip/||"`;; \
 ## Also rewrite $(top_srcdir) (which sometimes appears in DISTFILES, and can
 ## be absolute) by $(top_builddir) (which is always relative).  $(srcdir) will
-## be prepended latter.
-           $(top_srcdir)/*) file=`echo "$$file" | sed 
"s|^$$topsrcdirstrip/|$(top_builddir)/|"`;; \
-         esac; \
+## be prepended later.
+       list='$(DISTFILES)'; \
+         dist_files=`for file in $$list; do echo $$file; done | \
+         sed "t 
c;:c;s|^$$srcdirstrip/||;t;s|^$$topsrcdirstrip/|$(top_builddir)/|"`; \
+##
+## Make the subdirectories for the files.
+## (The DISTDIRS list can be incomplete, because files in subdirectories can
+## be specified for `dist' conditionally.)
+##
+       case $$dist_files in */*) \
+         ( cd "$(distdir)"; \
+           $(mkdir_p) `echo "$$dist_files" | sed -n 's,/[^/]*$$,,p' | sort 
-u`; \
+         ) ;; \
+       esac; \
+##
+##
+       for file in $$dist_files; do \
 ##
 ## Always look for the file in the build directory first.  That way
 ## for something like yacc output we will correctly pick up the latest
@@ -114,26 +125,14 @@
 ##
          if test -f $$file || test -d $$file; then d=.; else d=$(srcdir); fi; \
 ##
-## Make the subdirectory for the file.  This is going to make `dist'
-## really crawl, but it seems like the only way to do it, given that
-## files in subdirectories can be specified for `dist' conditionally.
-##
-         dir=`echo "$$file" | sed -e 's,/[^/]*$$,,'`; \
-         if test "$$dir" != "$$file" && test "$$dir" != "."; then \
-           dir="/$$dir"; \
-           $(mkdir_p) "$(distdir)$$dir"; \
-         else \
-           dir=''; \
-         fi; \
-##
 ## Use cp, not ln.  There are situations in which "ln" can fail.  For
 ## instance a file to distribute could actually be a cross-filesystem
 ## symlink -- this can easily happen if "gettextize" was run on the
 ## distribution.
 ##
          if test -d $$d/$$file; then \
-## Don't mention $$file in destination argument, since this fails if
-## destination directory already exists.  Also, use `-R' and not `-r'.
+## Don't mention $$file in the destination argument, since this fails if
+## the destination directory already exists.  Also, use `-R' and not `-r'.
 ## `-r' is almost always incorrect.
 ##
 ## If a directory exists both in `.' and $(srcdir), then
@@ -142,6 +141,7 @@
 ## source files _and_ generated files.  It is also important when the
 ## directory exists only in $(srcdir), because some vendor Make (such
 ## as Tru64) will magically create an empty directory in `.'
+           dir=`echo "/$$file" | sed -e 's,/[^/]*$$,,'`; \
            if test -d $(srcdir)/$$file && test $$d != $(srcdir); then \
              cp -pR $(srcdir)/$$file $(distdir)$$dir || exit 1; \
            fi; \