Re: libtool 2.2.6 does not install libraries correctly on HP-UX

[Ralf Wildenhues]

[ moving from bug-libtool; this is
  <http://thread.gmane.org/gmane.comp.gnu.libtool.bugs/6539> ]

Hello Bruno, all,

* Bruno Haible wrote on Mon, Sep 29, 2008 at 12:15:42PM CEST:
> 
> > Or is it that postinstall_cmds come too late, because the tools run by
> > the libtool script itself need this library already?
> 
> I think so, yes. Once you have coreutils, sed, and whatever installed and
> they all depend on libintl, you are immediately hosed if the replacement
> of the shared library is not done atomically.
> 
> The "chmod 555" afterwards is not wrong. But it would be good if you could
> also apply a heuristic: Look for the arguments of the --mode=install command.
> If you find consecutive arguments '-m' and '644', replace the second one
> with '755'.

Hmm, yes.  This turned out a bit more tricky than it looked at first.

First, replacing '-m XXX' with '-m 555' in the arguments is rather
straightforward, as is adding '-m 555' if not already present.

But then, the question is whether this really solves the issue:
The build system may not invoke an external program linked against
the library that is being installed.  Without the patch below,
libtool may directly invoke the following tools between the
installation of said shared library and its mode adjustment:
  ln      (for the soname symlinks)
  strip
  sed     (through func_show_eval, func_quote_for_expand for example)
  echo, printf, sh (through $ECHO)
  chmod

most of which may actually be linked against libintl, for example
('install' itself is not run again before postinst_cmds).

However, the program used for installation could also reasonably be
either cp, install-sh, or another install script (some ancestor versions
of install-sh are popular).  cp can only happen if the user explicitly
chose it.  install-sh may be chosen if no good install is found.

install-sh may itself invoke these programs:
  cp chmod chgrp chown mkdir rm strip mv cmp   rmdir ls sed

of which all but the last three are overridable through environment
variables ($CPPROG and the like).  The current version of the script
does the right thing though, by first installing a file to a temporary
place (on the destination mount point), setting its permission, then
moving it in place atomically.

The patch below fixes this issue for when GNU coreutils install
or install-sh are used, and then even allow parallel install
('make -j install') to work correctly.

Remains discussing other install scripts that AC_PROG_INSTALL may
accept.  I've found a couple that ship with imake.  One of them is
problematic (cp, then chown, chgrp, chmod used).  Autoconf >= 2.62
AC_PROG_INSTALL rejects this because it removes the files from the
uninstalled location.  When using older Autoconf, you have to go back
to using the environment variable overrides as above.

The other script passes on -m to the SYSV install (and is thus not
applicable on HP-UX).

I briefly looked at shtool (it's not searched for by AC_PROG_INSTALL,
but could arguably be chosen explicitly by the user).  It copies to a
temporary install location, fixes permissions, but then finally does
        rm -f $dst && mv $dsttmp $dst

thus 'mv' could fail if it needed the library.  Not quite sure if it's
worthwhile fixing shtool for this, but I'll notify them about it.

AFAIK only HP-UX has this problem that shared libraries fail to be
mapped with lack of permissions.  If you know of other systems, then
we'd appreciate a report.

Others, OK to install?  The testsuite addition does not expose the race
per se, only the mode setting.  I don't know how to do that portably.

Thanks,
Ralf

    Atomic shared library install permissions on HP-UX.
    
    The HP-UX 11 runtime linker fails to mmap shared libraries
    without execute permissions.  Fixing them with postinstall_cmds
    may be too late, as ln, sed, and chmod may link against the
    library being installed.
    
    * libltdl/m4/libtool.m4 (install_override_mode): New LT_DECL.
    (_LT_SYS_DYNAMIC_LINKER) [hpux9*, hpux10*, hpux11*]: Set it.
    * libltdl/config/ltmain.m4sh (func_mode_install): Build a second
    command line, $install_shared_prog, for the installation of the
    shared library.  Override resp. set the permission mode if
    install_override_mode is nonempty.
    * doc/libtool.texi (libtool script contents): Document it.
    * tests/install.at (Install tests): New test.
    * Makefile.am (TESTSUITE_AT): Adjust.
    (TESTS_ENVIRONMENT): Pass INSTALL.
    Report by Bruno Haible.

diff --git a/Makefile.am b/Makefile.am
index ddcff08..c842a40 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -460,6 +460,7 @@ TESTSUITE_AT        = tests/testsuite.at \
                  tests/archive-in-archive.at \
                  tests/execute-mode.at \
                  tests/localization.at \
+                 tests/install.at \
                  tests/destdir.at \
                  tests/old-m4-iface.at \
                  tests/am-subdir.at \
@@ -486,7 +487,7 @@ EXTRA_DIST     += $(srcdir)/$(TESTSUITE) $(TESTSUITE_AT) 
$(srcdir)/tests/package
 TESTS_ENVIRONMENT = MAKE="$(MAKE)" CC="$(CC)" CFLAGS="$(CFLAGS)" \
        CPP="$(CPP)" CPPFLAGS="$(CPPFLAGS)" LD="$(LD)" LDFLAGS="$(LDFLAGS)" \
        LIBS="$(LIBS)" LN_S="$(LN_S)" NM="$(NM)" RANLIB="$(RANLIB)" \
-       STRIP="$(STRIP)" \
+       STRIP="$(STRIP)" INSTALL="$(INSTALL)" \
        OBJEXT="$(OBJEXT)" EXEEXT="$(EXEEXT)" \
        SHELL="$(SHELL)" CONFIG_SHELL="$(SHELL)" \
        CXX="$(CXX)" CXXFLAGS="$(CXXFLAGS)" CXXCPP="$(CXXCPP)" \
diff --git a/doc/libtool.texi b/doc/libtool.texi
index 59f3872..1ee12aa 100644
--- a/doc/libtool.texi
+++ b/doc/libtool.texi
@@ -5819,6 +5819,15 @@ runtime path list, requiring libtool to relink the 
output when installing.
 Set to @samp{yes} or @samp{no}.  Default is @samp{no}.
 @end defvar
 
address@hidden install_override_mode
+Permission mode override for installation of shared libraries.  If the
+runtime linker fails to load libraries with wrong permissions, then it
+may fail to execute programs that are needed during installation,
+because these need the library that has just been installed.  In this
+case, it is necessary to pass the mode to @command{install} with
address@hidden @var{install_override_mode}}.
address@hidden defvar
+
 @defvar libext
 The standard old archive suffix (normally @samp{a}).
 @end defvar
diff --git a/libltdl/config/ltmain.m4sh b/libltdl/config/ltmain.m4sh
index 431c4cd..43f3798 100644
--- a/libltdl/config/ltmain.m4sh
+++ b/libltdl/config/ltmain.m4sh
@@ -1458,6 +1458,7 @@ func_mode_install ()
     # Aesthetically quote it.
     func_quote_for_eval "$arg"
     install_prog="$install_prog$func_quote_for_eval_result"
+    install_shared_prog=$install_prog
 
     # We need to accept at least all the BSD install flags.
     dest=
@@ -1467,8 +1468,10 @@ func_mode_install ()
     install_type=
     isdir=no
     stripme=
+    no_mode=:
     for arg
     do
+      arg2=
       if test -n "$dest"; then
        files="$files $dest"
        dest=$arg
@@ -1495,6 +1498,10 @@ func_mode_install ()
       *)
        # If the previous option needed an argument, then skip it.
        if test -n "$prev"; then
+         if test "x$prev" = x-m && test -n "$install_override_mode"; then
+           arg2=$install_override_mode
+           no_mode=false
+         fi
          prev=
        else
          dest=$arg
@@ -1506,6 +1513,10 @@ func_mode_install ()
       # Aesthetically quote the argument.
       func_quote_for_eval "$arg"
       install_prog="$install_prog $func_quote_for_eval_result"
+      if test -n "$arg2"; then
+       func_quote_for_eval "$arg2"
+      fi
+      install_shared_prog="$install_shared_prog $func_quote_for_eval_result"
     done
 
     test -z "$install_prog" && \
@@ -1514,6 +1525,11 @@ func_mode_install ()
     test -n "$prev" && \
       func_fatal_help "the \`$prev' option requires an argument"
 
+    if test -n "$install_override_mode" && $no_mode; then
+      func_quote_for_eval "$install_override_mode"
+      install_shared_prog="$install_shared_prog -m $func_quote_for_eval_result"
+    fi
+
     if test -z "$files"; then
       if test -z "$dest"; then
        func_fatal_help "no file or destination specified"
@@ -1634,7 +1650,7 @@ func_mode_install ()
          test -n "$relink_command" && srcname="$realname"T
 
          # Install the shared library and build the symlinks.
-         func_show_eval "$install_prog $dir/$srcname $destdir/$realname" \
+         func_show_eval "$install_shared_prog $dir/$srcname 
$destdir/$realname" \
              'exit $?'
          tstripme="$stripme"
          case $host_os in
diff --git a/libltdl/m4/libtool.m4 b/libltdl/m4/libtool.m4
index 904bb97..bddf80c 100644
--- a/libltdl/m4/libtool.m4
+++ b/libltdl/m4/libtool.m4
@@ -2387,8 +2387,10 @@ hpux9* | hpux10* | hpux11*)
     soname_spec='${libname}${release}${shared_ext}$major'
     ;;
   esac
-  # HP-UX runs *really* slowly unless shared libraries are mode 555.
+  # HP-UX runs *really* slowly unless shared libraries are mode 555, ...
   postinstall_cmds='chmod 555 $lib'
+  # or fails outright, so override atomically:
+  install_override_mode=555
   ;;
 
 interix[[3-9]]*)
@@ -2706,6 +2708,8 @@ _LT_DECL([], [library_names_spec], [1],
     The last name is the one that the linker finds with -lNAME]])
 _LT_DECL([], [soname_spec], [1],
     [[The coded name of the library, if different from the real name]])
+_LT_DECL([], [install_override_mode], [1],
+    [Permission mode override for installation of shared libraries])
 _LT_DECL([], [postinstall_cmds], [2],
     [Command to use after installation of a shared archive])
 _LT_DECL([], [postuninstall_cmds], [2],
diff --git a/tests/install.at b/tests/install.at
new file mode 100644
index 0000000..dd92100
--- /dev/null
+++ b/tests/install.at
@@ -0,0 +1,49 @@
+# install.at -- install mode tests             -*- Autotest -*-
+#
+#   Copyright (C) 2008 Free Software Foundation, Inc.
+#   Written by Ralf Wildenhues, 2008
+#
+#   This file is part of GNU Libtool.
+#
+# GNU Libtool is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# GNU Libtool is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Libtool; see the file COPYING.  If not, a copy
+# can be downloaded from  http://www.gnu.org/licenses/gpl.html,
+# or obtained by writing to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+####
+
+AT_SETUP([Install tests])
+AT_KEYWORDS([libtool])
+
+# Ensure that install_override_mode overrides the mode of the shared
+# library (and only the shared library).
+
+sed 's,^\(install_override_mode=\).*,&755,' < $LIBTOOL > ./libtool
+LIBTOOL=./libtool
+chmod a+rx $LIBTOOL
+
+inst=`pwd`/inst
+mkdir inst
+
+echo 'int a () { return 0; }' > a.c
+$LIBTOOL --mode=compile $CC $CPPFLAGS $CFLAGS -c a.c
+$LIBTOOL --mode=link $CC $CFLAGS $LDFLAGS -o liba.la a.lo -rpath $inst
+AT_CHECK([$LIBTOOL --mode=install $INSTALL liba.la $inst],
+        [], [stdout], [ignore])
+AT_CHECK([grep ' -m 755' stdout], [], [ignore])
+AT_CHECK([$LIBTOOL --mode=install $INSTALL -m 644 liba.la $inst],
+        [], [stdout], [ignore])
+AT_CHECK([grep ' -m 755' stdout], [], [ignore])
+AT_CHECK([grep ' -m 644.*liba.la' stdout], [], [ignore])
+
+AT_CLEANUP