[automake-commit] branch master updated: automake: again revise file mtime resolution support.

[Karl Berry]

This is an automated email from the git hooks/post-receive script.

karl pushed a commit to branch master
in repository automake.

View the commit online:
https://git.savannah.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=cfdb83d4220cfb7b8dc9e5b0a10eff0c628872ce

The following commit(s) were added to refs/heads/master by this push:
     new cfdb83d42 automake: again revise file mtime resolution support.
cfdb83d42 is described below

commit cfdb83d4220cfb7b8dc9e5b0a10eff0c628872ce
Author: Zack Weinberg <zack@owlfolio.org>
AuthorDate: Sun Dec 10 09:50:51 2023 -0800

    automake: again revise file mtime resolution support.
    
    This patch is from https://bugs.gnu.org/67670.
    
    In order for the Automake testsuite to be able to use sub-second
    delays to control whether certain files are considered newer
    than others, five(!) separate pieces of software all need to
    cooperate: automake itself, autoconf's internal `autom4te'
    utility, the Perl interpreter and its libraries, the sleep(1)
    shell utility, and finally the filesystem hosting the build
    directory. The existing tests for this are a combination of
    inadequate and incorrect. This patch, in conjunction with a
    patch just committed to Autoconf trunk,
    
https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=39d96e6fff7ceae63b823872602caf4d255a38c8
    
    should make everything much more robust, as follows:
    
    - _AM_FILESYSTEM_TIMESTAMP_RESOLUTION is completely rewritten.
     It no longer looks for autom4te at all, because this macro is
     invoked unconditionally from AM_INIT, so *every* project that
     uses Automake would get this test that's only relevant to
     Automake's own testsuite. Also, it tries sleeping for as little
     as one millisecond (smaller delays consistently get rounded up
     to 1ms on my computer and I expect that's universal), it should
     accurately detect FAT's two-second resolution now, and it should
     not be tripped up anymore by running at precisely the moment
     that will make a 0.1s sleep cross a 1s boundary (this may sound
     unlikely but it used to cause a couple of test failures *every
     time* I ran the automake testsuite on a network filesystem that
     only supported 1s resolution).
    
    - In support of the above, the test for working ls -t moved from
     AM_SANITY_CHECK to _AM_FILESYSTEM_TIMESTAMP_RESOLUTION. This
     allowed me to simplify the test for $srcdir/configure being
     older than a freshly created file.
    
    - If automake is capable of reading high-resolution file
     modification timestamps from the operating system, it prints
     `Features: subsecond-mtime' as the second line of --version
     output. (We can't just assume this works for sufficiently new
     automake, because it depends on whether the Perl interpreter
     provides this capability, and that's not a simple question of
     which version of Perl you have, either.)
    
    - The Autoconf patch mentioned above adds the same annotation to the
     output of autom4te --version.
    
    - Finally, t/ax/test-defs.in looks for the
     `Features: subsecond-mtime' string from both automake and
     autom4te and resets the sleep time to one second if it's not
     there. There might be a better place to put this, somewhere
     it'll execute every time the *overall testsuite* is invoked
     rather than once for each test, but I couldn't find one.
    
    Tested on x86-64-linux with development automake and development
    autoconf.
    
    Previous discussion:
    - https://lists.gnu.org/archive/html/automake/2023-03/msg00000.html
    - https://lists.gnu.org/archive/html/automake/2023-04/msg00002.html
    - https://lists.gnu.org/archive/html/automake/2023-12/msg00005.html
    - https://debbugs.gnu.org/cgi/bugreport.cgi?bug=64756
    
    * m4/sanity.m4 (_AM_FILESYSTEM_TIMESTAMP_RESOLUTION): Rewrite for
    greater reliability.  Don't probe autom4te at all here.
    Check for working ls -t here.
    (AM_SANITY_CHECK): Do not cache the result.
    Do not check for working `ls -t' here.
    Disentangle control flow in the loop probing the relative ages of
      build and source directory.
    
    * lib/Automake/FileUtils.pm: Sync from autoconf.
    * bin/automake.in (version): Include `Features: subsecond-mtime'
      in the output if $Automake::FileUtils::subsecond_mtime is true.
    * configure.ac: Rename the substitution variable MODIFICATION_DELAY
      to MTIME_RESOLUTION.
    * t/ax/test-defs.in: Require both $AUTOMAKE and $AUTOM4TE to report
      support for high-resolution timestamps before setting $sleep to
      delay for less than one second.
    * NEWS: Update info.
---
 NEWS                      |   7 +-
 bin/automake.in           |  17 +++--
 configure.ac              |   9 ++-
 lib/Automake/FileUtils.pm |  69 ++++++++++++++-----
 m4/sanity.m4              | 169 +++++++++++++++++++++++++---------------------
 t/ax/test-defs.in         |  26 +++++--
 6 files changed, 183 insertions(+), 114 deletions(-)

diff --git a/NEWS b/NEWS
index 5150f585a..634e7160f 100644
--- a/NEWS
+++ b/NEWS
@@ -47,8 +47,11 @@ New in 1.17:
     timestamp granularity dynamically, greatly speeding up make check,
     etc. However, this requires an autom4te from Autoconf 2.72 (as yet
     unreleased) or later (or random test failures and other timing
-    problems may ensue); for older autom4te, we fall back to one-second
-    granularity as before. (bug#64756)
+    problems may ensue), as well as a Perl, sleep utility, and
+    filesystem that supports sub-second resolution; otherwise, we fall
+    back to one-second granularity as before. When everything is
+    supported, a new line `Features: subsecond-mtime' is printed by
+    automake --version (and autom4mte --version). (bugs #64756, #67670)
 
   - The default value of $ARFLAGS is now "cr" instead of "cru", to better
     support deterministic builds. (bug#20082)
diff --git a/bin/automake.in b/bin/automake.in
index c0ffc1986..820d0d541 100644
--- a/bin/automake.in
+++ b/bin/automake.in
@@ -8271,16 +8271,19 @@ General help using GNU software: 
<https://www.gnu.org/gethelp/>.
 
 sub version ()
 {
-  print <<EOF;
-automake (GNU $PACKAGE) $VERSION
-Copyright (C) $RELEASE_YEAR Free Software Foundation, Inc.
-License GPLv2+: GNU GPL version 2 or later 
<https://gnu.org/licenses/gpl-2.0.html>
+  print "automake (GNU $PACKAGE) $VERSION\n";
+  print "Features: subsecond-mtime\n"
+      if $Automake::FileUtils::subsecond_mtime;
+  print "\nCopyright (C) $RELEASE_YEAR Free Software Foundation, Inc.";
+  print '
+License GPLv2+: GNU GPL version 2 or later
+  <https://gnu.org/licenses/gpl-2.0.html>
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.
 
-Written by Tom Tromey <tromey\@redhat.com>
-       and Alexandre Duret-Lutz <adl\@gnu.org>.
-EOF
+Written by Tom Tromey <tromey@redhat.com>
+       and Alexandre Duret-Lutz <adl@gnu.org>.
+';
   # --version always returns 0 per GNU standards.
   exit 0;
 }
diff --git a/configure.ac b/configure.ac
index 5cda80a18..33aec7788 100644
--- a/configure.ac
+++ b/configure.ac
@@ -180,9 +180,6 @@ result=no
 test "x$am_cv_prog_ln" = xln && result=yes
 AC_MSG_RESULT([$result])
 
-MODIFICATION_DELAY=$am_cv_filesystem_timestamp_resolution
-AC_SUBST([MODIFICATION_DELAY])
-
 ## ------------------------------------------- ##
 ##  Test for things needed by the test suite.  ##
 ## ------------------------------------------- ##
@@ -190,6 +187,12 @@ AC_SUBST([MODIFICATION_DELAY])
 AC_PROG_EGREP
 AC_PROG_FGREP
 
+# The test suite needs to know the resolution of timestamps
+# supported by the filesystem hosting the build.  The value
+# will be acceptable to 'sleep' on this system.
+MTIME_RESOLUTION=$am_cv_filesystem_timestamp_resolution
+AC_SUBST([MTIME_RESOLUTION])
+
 dnl FIXME: could we extract this in a simpler way through autoconf
 dnl FIXME: idioms or internals?
 AC_DEFUN(
diff --git a/lib/Automake/FileUtils.pm b/lib/Automake/FileUtils.pm
index 8d0b36802..553d5c6dc 100644
--- a/lib/Automake/FileUtils.pm
+++ b/lib/Automake/FileUtils.pm
@@ -38,24 +38,45 @@ use 5.006;
 use strict;
 use warnings FATAL => 'all';
 
-use Exporter;
-use IO::File;
+BEGIN
+{
+  require Exporter;
+  our @ISA = qw (Exporter);
+  our @EXPORT = qw (&contents
+                   &find_file &mtime
+                   &update_file
+                   &xsystem &xsystem_hint &xqx
+                   &dir_has_case_matching_file &reset_dir_cache
+                   &set_dir_cache_file);
+}
 
-# use sub-second resolution timestamps if available,
-# carry on with one-second resolution timestamps if that is all we have
-BEGIN { eval { require Time::HiRes; import Time::HiRes qw(stat) } }
+# Use sub-second resolution file timestamps if available, carry on
+# with one-second resolution timestamps if Time::HiRes is not available.
+#
+# Unfortunately, even if Time::HiRes is available, we don't get
+# timestamps to the full precision recorded by the operating system,
+# because Time::HiRes converts timestamps to floating-point, and the
+# rounding error is hundreds of nanoseconds for circa-2023 timestamps
+# in IEEE double precision.  But this is the best we can do without
+# dropping down to C.
+#
+# $subsecond_mtime is not exported, but is intended for external
+# consumption, as $Automake::FileUtils::subsecond_mtime.
+BEGIN
+{
+  our $subsecond_mtime = 0;
+  eval
+    {
+      require Time::HiRes;
+      import Time::HiRes qw(stat);
+      $subsecond_mtime = 1;
+    }
+}
 
+use IO::File;
 use Automake::Channels;
 use Automake::ChannelDefs;
 
-our @ISA = qw (Exporter);
-our @EXPORT = qw (&contents
-                 &find_file &mtime
-                 &update_file
-                 &xsystem &xsystem_hint &xqx
-                 &dir_has_case_matching_file &reset_dir_cache
-                 &set_dir_cache_file);
-
 =over 4
 
 =item C<find_file ($file_name, @include)>
@@ -122,11 +143,6 @@ sub mtime ($)
     $atime,$mtime,$ctime,$blksize,$blocks) = stat ($file)
     or fatal "cannot stat $file: $!";
 
-  # Unfortunately Time::HiRes converts timestamps to floating-point, and the
-  # rounding error can be hundreds of nanoseconds for circa-2023 timestamps.
-  # Perhaps some day Perl will support accurate file timestamps.
-  # For now, do the best we can without going outside Perl.
-
   return $mtime;
 }
 
@@ -394,3 +410,20 @@ sub set_dir_cache_file ($$)
 =cut
 
 1; # for require
+
+### Setup "GNU" style for perl-mode and cperl-mode.
+## Local Variables:
+## perl-indent-level: 2
+## perl-continued-statement-offset: 2
+## perl-continued-brace-offset: 0
+## perl-brace-offset: 0
+## perl-brace-imaginary-offset: 0
+## perl-label-offset: -2
+## cperl-indent-level: 2
+## cperl-brace-offset: 0
+## cperl-continued-brace-offset: 0
+## cperl-label-offset: -2
+## cperl-extra-newline-before-brace: t
+## cperl-merge-trailing-else: nil
+## cperl-continued-statement-offset: 2
+## End:
diff --git a/m4/sanity.m4 b/m4/sanity.m4
index 8887a0c98..93c4854b9 100644
--- a/m4/sanity.m4
+++ b/m4/sanity.m4
@@ -15,66 +15,87 @@ AS_IF([sleep 0.001 2>/dev/null], 
[am_cv_sleep_fractional_seconds=true], [am_cv_s
 
 # _AM_FILESYSTEM_TIMESTAMP_RESOLUTION
 # -----------------------------------
-# Determine the filesystem timestamp resolution.  Modern systems are
-# nanosecond capable, but historical systems could have millisecond,
-# second, or even 2-second resolution.
+# Determine the filesystem's resolution for file modification
+# timestamps.  The coarsest we know of is FAT, with a resolution
+# of only two seconds, even with the most recent "exFAT" extensions.
+# The finest (e.g. ext4 with large inodes, XFS, ZFS) is one
+# nanosecond, matching clock_gettime.  However, it is probably not
+# possible to delay execution of a shell script for less than one
+# millisecond, due to process creation overhead and scheduling
+# granularity, so we don't check for anything finer than that.
 AC_DEFUN([_AM_FILESYSTEM_TIMESTAMP_RESOLUTION], [dnl
 AC_REQUIRE([_AM_SLEEP_FRACTIONAL_SECONDS])
-#
-# Check if Autom4te uses Time::HiRes. If not, we cannot use fractional sleep,
-# because this sanity test and automated tests will be unreliable due to
-# Autom4te's caching of results and comparing timestamps.
-# More info: long thread around
-#     https://lists.gnu.org/archive/html/automake/2023-04/msg00002.html
-# and https://debbugs.gnu.org/cgi/bugreport.cgi?bug=64756.
-# By the way, we cannot use Perl to see if %INC{q[Time/HiRes.pm]} is
-#   defined, because Time::HiRes might get pulled in from other system
-#   modules even when not used directly. (An idea suggested in that thread.)
-AC_PATH_PROG([AUTOM4TE], [autom4te])
-if test x"$autom4te_perllibdir" = x; then
-  autom4te_perllibdir=`sed -n \
-   '/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' <$AUTOM4TE`
+AC_CACHE_CHECK([the filesystem timestamp resolution], 
am_cv_filesystem_timestamp_resolution, [dnl
+# Default to the worst case.
+am_cv_filesystem_timestamp_resolution=2
+
+# Only try to go finer than 1s if sleep can do it.
+am_try_resolutions=1
+if $am_cv_sleep_fractional_seconds; then
+  am_try_resolutions="0.001 0.01 0.1 $am_try_resolutions"
 fi
-if grep HiRes "$autom4te_perllibdir"/Autom4te/FileUtils.pm >/dev/null; then
-  :
-else
-  am_cv_sleep_fractional_seconds=false
+
+# In order to catch current-generation FAT out, we must *modify* files
+# that already exist; the *creation* timestamp is finer.  Use names
+# that make ls -t sort them differently when they have equal
+# timestamps than when they have distinct timestamps, keeping
+# in mind that ls -t prints the *newest* file first.
+rm -f conftest.ts?
+: > conftest.ts1
+: > conftest.ts2
+: > conftest.ts3
+
+# Make sure ls -t actually works.  Do 'set' in a subshell so we don't
+# clobber the current shell's arguments.
+if (
+     set X `[ls -t conftest.ts[12]]` &&
+     {
+       test "$[*]" != "X conftest.ts1 conftest.ts2" ||
+       test "$[*]" != "X conftest.ts2 conftest.ts1";
+     }
+); then :; else
+  # If neither matched, then we have a broken ls.  This can happen
+  # if, for instance, CONFIG_SHELL is bash and it inherits a
+  # broken ls alias from the environment.  This has actually
+  # happened.  Such a system could not be considered "sane".
+  _AS_ECHO_UNQUOTED(
+    ["Bad output from ls -t: \"`[ls -t conftest.ts[12]]`\""],
+    [AS_MESSAGE_LOG_FD])
+  AC_MSG_FAILURE([ls -t produces unexpected output.
+Make sure there is not a broken alias in your environment.])
 fi
 
-AC_CACHE_CHECK([the filesystem timestamp resolution], 
am_cv_filesystem_timestamp_resolution, [dnl
-# Use names that lexically sort older-first when the timestamps are equal.
-rm -f conftest.file.a conftest.file.b
-: > conftest.file.a
-AS_IF([$am_cv_sleep_fractional_seconds], [dnl
-  am_try_sleep=0.1 am_try_loops=20
-], [dnl
-  am_try_sleep=1   am_try_loops=2
-])
-am_try=0
-while :; do
-  AS_VAR_ARITH([am_try], [$am_try + 1])
-  echo "timestamp $am_try" > conftest.file.b
-  set X `ls -t conftest.file.a conftest.file.b`
-  if test "$[2]" = conftest.file.b || test $am_try -eq $am_try_loops; then
+for am_try_res in $am_try_resolutions; do
+  # Any one fine-grained sleep might happen to cross the boundary
+  # between two values of a coarser actual resolution, but if we do
+  # two fine-grained sleeps in a row, at least one of them will fall
+  # entirely within a coarse interval.
+  echo alpha > conftest.ts1
+  sleep $am_try_res
+  echo beta > conftest.ts2
+  sleep $am_try_res
+  echo gamma > conftest.ts3
+
+  # We assume that 'ls -t' will make use of high-resolution
+  # timestamps if the operating system supports them at all.
+  if (set X `ls -t conftest.ts?` &&
+      test "$[]2" = conftest.ts3 &&
+      test "$[]3" = conftest.ts2 &&
+      test "$[]4" = conftest.ts1); then
+    am_cv_filesystem_timestamp_resolution=$am_try_res
     break
   fi
-  sleep $am_try_sleep
 done
-rm -f conftest.file.a conftest.file.b
-am_cv_filesystem_timestamp_resolution=$am_try
-AS_IF([$am_cv_sleep_fractional_seconds], [dnl
-  AS_VAR_ARITH([am_cv_filesystem_timestamp_resolution], [$am_try / 10])
-  AS_VAR_ARITH([am_fraction], [$am_try % 10])
-  AS_VAR_APPEND([am_cv_filesystem_timestamp_resolution], [.$am_fraction])
-])
+rm -f conftest.ts?
 ])])
 
 # AM_SANITY_CHECK
 # ---------------
 AC_DEFUN([AM_SANITY_CHECK],
 [AC_REQUIRE([_AM_FILESYSTEM_TIMESTAMP_RESOLUTION])
-rm -f conftest.file
-AC_CACHE_CHECK([whether build environment is sane], am_cv_build_env_is_sane, 
[dnl
+# This check should not be cached, as it may vary across builds of
+# different projects.
+AC_MSG_CHECKING([whether build environment is sane])
 # Reject unsafe characters in $srcdir or the absolute working directory
 # name.  Accept space and tab only in the latter.
 am_lf='
@@ -93,41 +114,33 @@ esac
 # symlink; some systems play weird games with the mod time of symlinks
 # (eg FreeBSD returns the mod time of the symlink's containing
 # directory).
-if (
-   am_has_slept=no
-   for am_try in 1 2; do
-     echo "timestamp, slept: $am_has_slept" > conftest.file
-     set X `ls -Lt "$srcdir/configure" conftest.file 2> /dev/null`
-     if test "$[*]" = "X"; then
-       # -L didn't work.
-       set X `ls -t "$srcdir/configure" conftest.file`
-     fi
-     if test "$[*]" != "X $srcdir/configure conftest.file" \
-       && test "$[*]" != "X conftest.file $srcdir/configure"; then
+am_build_env_is_sane=no
+am_has_slept=no
+rm -f conftest.file
+for am_try in 1 2; do
+  echo "timestamp, slept: $am_has_slept" > conftest.file
+  if (
+    set X `ls -Lt "$srcdir/configure" conftest.file 2> /dev/null`
+    if test "$[*]" = "X"; then
+      # -L didn't work.
+      set X `ls -t "$srcdir/configure" conftest.file`
+    fi
+    test "$[2]" = conftest.file
+  ); then
+    am_build_env_is_sane=yes
+    break
+  fi
+  # Just in case.
+  sleep $am_cv_filesystem_timestamp_resolution
+  am_has_slept=yes
+done
 
-       # If neither matched, then we have a broken ls.  This can happen
-       # if, for instance, CONFIG_SHELL is bash and it inherits a
-       # broken ls alias from the environment.  This has actually
-       # happened.  Such a system could not be considered "sane".
-       AC_MSG_ERROR([ls -t appears to fail.  Make sure there is not a broken
-  alias in your environment])
-     fi
-     if test "$[2]" = conftest.file || test $am_try -eq 2; then
-       break
-     fi
-     # Just in case.
-     sleep $am_cv_filesystem_timestamp_resolution
-     am_has_slept=yes
-   done
-   test "$[2]" = conftest.file
-   )
-then
-  am_cv_build_env_is_sane=yes
-else
-   AC_MSG_ERROR([newly created file is older than distributed files!
+AC_MSG_RESULT([$am_build_env_is_sane])
+if test $am_build_env_is_sane = no; then
+  AC_MSG_ERROR([newly created file is older than distributed files!
 Check your system clock])
 fi
-])
+
 # If we didn't sleep, we still need to ensure time stamps of config.status and
 # generated files are strictly newer.
 am_sleep_pid=
diff --git a/t/ax/test-defs.in b/t/ax/test-defs.in
index e09a387cd..696403ee0 100644
--- a/t/ax/test-defs.in
+++ b/t/ax/test-defs.in
@@ -177,12 +177,26 @@ 
GNU_GCJFLAGS=${AM_TESTSUITE_GNU_GCJFLAGS-${GNU_GCJFLAGS-'@GNU_GCJFLAGS@'}}
 # this variable.
 TEX=${AM_TESTSUITE_TEX-'@TEX@'}
 
-# The amount we should wait after modifying files depends on the platform.
-# For instance, Windows '95, '98 and ME have 2-second granularity
-# and can be up to 3 seconds in the future w.r.t. the system clock.
-# The creative quoting is to avoid spuriously triggering a failure in
-# the maintainer checks,
-sleep='sleep ''@MODIFICATION_DELAY@'
+# The time we should wait after modifying files depends on the platform, and
+# also the capabilities of the 'automake' and 'autom4te' programs, which in
+# turn depend on the capabilities of the Perl interpreter.  configure has
+# detected the platform's resolution for file modification times; we still
+# need to check automake and autom4te.
+MTIME_RESOLUTION='@MTIME_RESOLUTION@'
+case $MTIME_RESOLUTION in
+  1 | 2) ;; # Can assume these values are usable as is.
+  *)
+    if $AUTOMAKE --version 2>&1 |
+         grep 'Features:.*subsecond-mtime' > /dev/null 2>&1 &&
+       $AUTOM4TE --version 2>&1 |
+         grep 'Features:.*subsecond-mtime' > /dev/null 2>&1; then
+      :
+    else
+      MTIME_RESOLUTION=1
+    fi
+  ;;
+esac
+sleep="sleep $MTIME_RESOLUTION"
 
 # An old timestamp that can be given to a file, in "touch -t" format.
 # The time stamp should be portable to all file systems of interest.