FYI: get rid of $automake_needs_to_reprocess_all_files (Was: Re: automake 1.8 performance problems)

[Alexandre Duret-Lutz]

>>> "adl" == Alexandre Duret-Lutz <address@hidden> writes:

>>> "Harlan" == Harlan Stenn <address@hidden> writes:
 Harlan> OK, I think I have something.
 Harlan> If AC_CONFIG_AUX_DIR is populated, automake runs in 9-15 minutes of
 Harlan> wall-clock time.

 Harlan> If it is not populated (except for ltmain.sh) then it
 Harlan> takes 45 minutes to run.
[...]
 adl> There is a slowdown to expect when the auxdir is populated.
 adl> This is because the Makefile that cause depcomp (for instance)
 adl> to be installed is usually not the same as the Makefile that
 adl> will distribute it (the latter is the auxdir/Makefile or the
 adl> top-level Makefile).  When automake detects such a situation, it
 adl> simply schedule a second pass to process again all the Makefiles
 adl> (now the aux files are installed they will be automatically
 adl> distributed by the right Makefile).  The --verbose option
 adl> will print "processing Makefiles another time to fix them up"
 adl> when this happens.

 adl> This behavior would explain why automake sometimes takes twice
 adl> more time; not as much as what you observe.  I should also adds
 adl> that Automake has a sanity check that ensure this "reprocessing"
 adl> does not occur more than once.

 adl> It seems to me it might be possible to suppress this
 adl> reprocessing my letting Automake select a good processing order
 adl> for the Makefiles: process the Makefile that will ship the
 adl> auxdir files after all the other Makefiles.

I'm installing the following patch (which does just that) on HEAD.

2004-04-12  Alexandre Duret-Lutz  <address@hidden>

        * automake.in ($automake_needs_to_reprocess_all_files): Remove.
        ($automake_will_process_aux_dir): New variable.
        (scan_autoconf_traces): Reorder @input_files so that the Makefile
        that distributes aux files is processed last.  This way we do not
        have to process all files twice using
        $automake_will_process_aux_dir.
        (require_file_internal): Suggest a full run of automake when
        appropriate.
        (MAIN): Remove the loop on $automake_needs_to_reprocess_all_files.
        * tests/distcom7.test: New file.
        * tests/Makefile.am (TESTS): Add distcom7.test.

Index: automake.in
===================================================================
RCS file: /cvs/automake/automake/automake.in,v
retrieving revision 1.1551
diff -u -p -r1.1551 automake.in
--- automake.in 12 Apr 2004 19:19:16 -0000      1.1551
+++ automake.in 12 Apr 2004 20:50:39 -0000
@@ -421,11 +421,9 @@ my %required_targets =
    'install-man' => 1,
   );
 
-# This is set to 1 when Automake needs to be run again.
-# (For instance, this happens when an auxiliary file such as
-# depcomp is added after the toplevel Makefile.in -- which
-# should distribute depcomp -- has been generated.)
-my $automake_needs_to_reprocess_all_files = 0;
+# Set to 1 if this run will create the Makefile.in that distribute
+# the files in config_aux_dir.
+my $automake_will_process_aux_dir = 0;
 
 # The name of the Makefile currently being processed.
 my $am_file = 'BUG';
@@ -4845,6 +4843,37 @@ sub scan_autoconf_files ()
 
   # Preserve dist_common for later.
   $configure_dist_common = variable_value ('DIST_COMMON') || '';
+
+  # Reorder @input_files so that the Makefile that distributes aux
+  # files is processed last.  This is important because each directory
+  # can require auxiliary scripts and we should wait until they have
+  # been installed before distributing them.
+
+  # The Makefile.in that distribute the aux files is the one in
+  # $config_aux_dir or the top-level Makefile.
+  my $auxdirdist = is_make_dir ($config_aux_dir) ? $config_aux_dir : '.';
+  my @new_input_files = ();
+  while (@input_files)
+    {
+      my $in = pop @input_files;
+      my @ins = split (/:/, $output_files{$in});
+      if (dirname ($ins[0]) eq $auxdirdist)
+       {
+         push @new_input_files, $in;
+         $automake_will_process_aux_dir = 1;
+       }
+      else
+       {
+         unshift @new_input_files, $in;
+       }
+    }
+  @input_files = @new_input_files;
+
+  # If neither the auxdir/Makefile nor the ./Makefile are generated
+  # by Automake, we won't distribute the aux files anyway.  Assume
+  # the user know what (s)he does, and pretend we will distribute
+  # them to disable the error in require_file_internal.
+  $automake_will_process_aux_dir = 1 if ! is_make_dir ($auxdirdist);
 }
 
 ################################################################
@@ -6801,19 +6830,16 @@ sub require_file_internal ($$$@)
                  if (! maybe_push_required_file (dirname ($fullfile),
                                                  $file, $fullfile))
                    {
-                     if (! $found_it)
+                     if (! $found_it && ! $automake_will_process_aux_dir)
                        {
                          # We have added the file but could not push it
-                         # into DIST_COMMON (probably because this is
+                         # into DIST_COMMON, probably because this is
                          # an auxiliary file and we are not processing
-                         # the top level Makefile). This is unfortunate,
-                         # since it means we are using a file which is not
-                         # distributed!
-
-                         # Get Automake to be run again: on the second
-                         # run the file will be found, and pushed into
-                         # the toplevel DIST_COMMON automatically.
-                         $automake_needs_to_reprocess_all_files = 1;
+                         # the top level Makefile.  Furthermore Automake
+                         # hasn't been asked to create the Makefile.in
+                         # that distribute the aux dir files.
+                         error ($where, 'Please make a full run of automake'
+                                . " so $fullfile gets distributed.");
                        }
                    }
                }
@@ -7343,40 +7369,25 @@ scan_autoconf_files;
 fatal "no `Makefile.am' found or specified\n"
   if ! @input_files;
 
-my $automake_has_run = 0;
-
-do
-{
-  if ($automake_has_run)
-    {
-      verb 'processing Makefiles another time to fix them up.';
-      prog_error 'running more than two times should never be needed.'
-       if $automake_has_run >= 2;
-    }
-  $automake_needs_to_reprocess_all_files = 0;
-
-  # Now do all the work on each file.
-  foreach my $file (@input_files)
-    {
-      ($am_file = $file) =~ s/\.in$//;
-      if (! -f ($am_file . '.am'))
-       {
-         error "`$am_file.am' does not exist";
-       }
-      else
-       {
-         # Any warning setting now local to this Makefile.am.
-         dup_channel_setup;
+# Now do all the work on each file.
+foreach my $file (@input_files)
+  {
+    ($am_file = $file) =~ s/\.in$//;
+    if (! -f ($am_file . '.am'))
+      {
+       error "`$am_file.am' does not exist";
+      }
+    else
+      {
+       # Any warning setting now local to this Makefile.am.
+       dup_channel_setup;
 
-         generate_makefile ($am_file . '.am', $file);
+       generate_makefile ($am_file . '.am', $file);
 
-         # Back out any warning setting.
-         drop_channel_setup;
-       }
-    }
-  ++$automake_has_run;
-}
-while ($automake_needs_to_reprocess_all_files);
+       # Back out any warning setting.
+       drop_channel_setup;
+      }
+  }
 
 exit $exit_code;
 
Index: tests/Makefile.am
===================================================================
RCS file: /cvs/automake/automake/tests/Makefile.am,v
retrieving revision 1.553
diff -u -p -r1.553 Makefile.am
--- tests/Makefile.am   9 Apr 2004 22:21:57 -0000       1.553
+++ tests/Makefile.am   12 Apr 2004 20:50:40 -0000
@@ -186,6 +186,7 @@ distcom3.test \
 distcom4.test \
 distcom5.test \
 distcom6.test \
+distcom7.test \
 distdir.test \
 distname.test \
 dollar.test \
Index: tests/distcom7.test
===================================================================
RCS file: tests/distcom7.test
diff -N tests/distcom7.test
--- /dev/null   1 Jan 1970 00:00:00 -0000
+++ tests/distcom7.test 12 Apr 2004 20:50:44 -0000
@@ -0,0 +1,45 @@
+#! /bin/sh
+# Copyright (C) 2004  Free Software Foundation, Inc.
+#
+# This file is part of GNU Automake.
+#
+# GNU Automake is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# GNU Automake is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Automake; see the file COPYING.  If not, write to
+# the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+# Test to make sure that Automake complains when an auxfile (here depcomp)
+# is installed, but the Makefile tht distributes it is not processed.
+
+. ./defs || exit 1
+
+cat > configure.in << 'END'
+AC_INIT(subdir/foo.c)
+AM_INIT_AUTOMAKE(nonesuch, nonesuch)
+AC_PROG_CC
+AC_OUTPUT(Makefile subdir/Makefile)
+END
+
+cat > Makefile.am << 'END'
+SUBDIRS = subdir
+END
+
+rm -f depcomp
+mkdir subdir
+
+echo bin_PROGRAMS = foo > subdir/Makefile.am
+: > subdir/foo.c
+
+$ACLOCAL
+AUTOMAKE_fails --add-missing subdir/Makefile
+grep 'full run' stderr
-- 
Alexandre Duret-Lutz