Re: New doc section about command line length limits.

[Ralf Wildenhues]

* Ralf Wildenhues wrote on Sun, Sep 07, 2008 at 01:16:46PM CEST:
> Review and comments of whatever sort appreciated.
> I haven't pushed this yet.

Pushed as below, slightly amended to cater for
- libtool multi-file install,
- nobase is not equivalent to several vars for LTLIBRARIES, as
- order is not unimportant for library dependencies,
- 'make dist' may only work on better hosts.

Cheers,
Ralf

    New doc section about command line length limits.
    
    * doc/automake.texi (Length limitations): New node.
    (Alternative): `nobase_' is not always equivalent to several
    base variables.
    (Install): Mention multi-file install.
    * NEWS: Expand on the issue, list some more user-visible
    consequences.

diff --git a/NEWS b/NEWS
index 9c4dc91..dddf0a2 100644
--- a/NEWS
+++ b/NEWS
@@ -50,8 +50,26 @@ New in 1.10a:
   - install-sh supports -C, which does not update the installed file
     (and its time stamps) if the contents did not change.
 
-  - `make install' now installs multiple files in one `install' invocation
-    for files with DATA, HEADERS, PYTHON, LIBRARIES and TEXINFOS primaries.
+  - The targets `install' and `uninstall' are more efficient now, in that
+    for example multiple files from one Automake variable such as
+    `bin_SCRIPTS' are copied in one `install' (or `libtool --mode=install')
+    invocation if they do not have to be renamed.
+
+    Both install and uninstall may sometimes enter (`cd' into) the target
+    installation directory now, when no build-local scripts are used.
+
+    For builtin rules, `make install' now fails reliably if installation
+    of a file failed.
+
+    These changes may need some adjustments from users:  For example,
+    some `install' programs refuse to install multiple copies of the
+    same file in one invocation, so you may need to remove duplicate
+    entries from file lists.
+
+    Also, within one set of files, say, nobase_data_DATA, the order of
+    installation may be changed, or even unstable among different hosts,
+    due to the use of associative arrays in awk.  The increased use of
+    awk matches a similar move in Autoconf to provide for better scaling.
 
   - The "deleted header file problem" for *.m4 files is avoided by
     stub rules.  This allows `make' to trigger a rerun of `aclocal'
diff --git a/doc/automake.texi b/doc/automake.texi
index 263d1b6..c2e0be5 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -161,6 +161,7 @@ General ideas
 * Strictness::                  Standards conformance checking
 * Uniform::                     The Uniform Naming Scheme
 * Canonicalization::            How derived variables are named
+* Length limitations::          Staying below the command line length limit
 * User Variables::              Variables reserved for the user
 * Auxiliary Programs::          Programs automake might require
 
@@ -1688,6 +1689,7 @@ understand how Automake works.
 * Strictness::                  Standards conformance checking
 * Uniform::                     The Uniform Naming Scheme
 * Canonicalization::            How derived variables are named
+* Length limitations::          Staying below the command line length limit
 * User Variables::              Variables reserved for the user
 * Auxiliary Programs::          Programs automake might require
 @end menu
@@ -1978,6 +1980,65 @@ These prefixes are explained later (@pxref{Program and 
Library Variables})
 (@pxref{Man pages}).
 
 
address@hidden Length limitations
address@hidden Staying below the command line length limit
+
address@hidden command line length limit
address@hidden ARG_MAX
+
+Traditionally, most unix-like systems have a length limitation for the
+command line arguments and environment contents when creating new
+processes (see for example
address@hidden://www.in-ulm.de/@/~mascheck/@/various/@/argmax/} for an
+overview on this issue),
+which of course also applies to commands spawned by @command{make}.
+POSIX requires this limit to be at least 4096 bytes, and most modern
+systems have quite high limits, if at all.
+
+In order to create portable Makefiles that do not trip over these
+limits, it is necessary to keep the length of file lists bounded.
+Unfortunately, it is not possible to do so fully transparently within
+Automake, so your help may be needed.  Typically, you can split long
+file lists manually and use different installation directory names for
+each list.  For example,
+
address@hidden
+data_DATA = file1 @dots{} address@hidden address@hidden @dots{} address@hidden
address@hidden example
+
address@hidden
+may also be written as
+
address@hidden
+data_DATA = file1 @dots{} address@hidden
+data2dir = $(datadir)
+data2_DATA = address@hidden @dots{} address@hidden
address@hidden example
+
address@hidden
+and will cause Automake to treat the two lists separately during
address@hidden install}.  See @ref{Install} for choosing directory names
+that will keep the ordering of the two parts of installation
+(@pxref{Two-Part Install}).  Note that @code{make dist} may still
+only work on a host with a higher length limit in this example.
+
+Automake itself employs a couple of strategies to avoid long command
+lines.  For example, when @address@hidden@}/} is prepended to file
+names, as can happen with above @code{$(data_DATA)} lists, it limits
+the amount of arguments passed to external commands.
+
+Unfortunately, some system's @command{make} commands may prepend
address@hidden prefixes like @address@hidden@}/} to file names from the
+source tree automatically (@pxref{Automatic Rule Rewriting, , Automatic
+Rule Rewriting, autoconf, The Autoconf Manual}).  In this case, the user
+may have to switch to use GNU Make, or refrain from using VPATH builds,
+in order to stay below the length limit.
+
+For libraries and programs built from many sources, convenience archives
+may be used as intermediates in order to limit the object list length
+(@pxref{Libtool Convenience Libraries}).
+
+
 @node Canonicalization
 @section How derived variables are named
 
@@ -4241,7 +4302,7 @@ nobase_dist_pkgdata_DATA = images/vortex.pgm 
sounds/whirl.ogg
 @end example
 
 Finally, note that a variable using the @samp{nobase_} prefix can
-always be replaced by several variables, one for each destination
+often be replaced by several variables, one for each destination
 directory (@pxref{Uniform}).  For instance, the last example could be
 rewritten as follows:
 
@@ -4256,6 +4317,10 @@ dist_sounds_DATA = sounds/whirl.ogg
 This latter syntax makes it possible to change one destination
 directory without changing the layout of the source tree.
 
+Currently, @samp{nobase_*_LTLIBRARIES} are the only exception to this
+rule, in that there is no particular installation order guarantee for
+an otherwise equivalent set of variables without @samp{nobase_} prefix.
+
 @node Subpackages
 @section Nesting Packages
 @cindex Nesting packages
@@ -7744,6 +7809,18 @@ nobase_include_HEADERS = stdio.h sys/types.h
 Will install @file{stdio.h} in @samp{$(includedir)} and @file{types.h}
 in @samp{$(includedir)/sys}.
 
+For most file types, Automake will install multiple files at once, while
+avoiding command line length issues (@pxref{Length limitations}).  Since
+some @command{install} programs will not install the same file twice in
+one invocation, you may need to ensure that file lists are unique within
+one variable such as @samp{nobase_include_HEADERS} above.
+
+You should not rely on the order in which files listed in one variable
+are installed.  Likewise, to cater for parallel make, you should not
+rely on any particular file installation order even among different
+file types (library dependencies are an exception here).
+
+
 @section The two parts of install
 
 Automake generates separate @code{install-data} and @code{install-exec}