Re: [MLB] Fix for arg list too long (another [final?] attempt)

[Robert Boehne]

Alexandre Oliva wrote:
> 
> On Feb 19, 2001, Robert Boehne <address@hidden> wrote:
> 
> > +while test `$0 --fallback-echo "X$testring" >/dev/null 2>&1` == `echo 
> > "X$testring" >/dev/null 2>&1` &&
> 
> $0 may not be executable.  Whenever you run a shell-script, you should
> use something like `${CONFIG_SHELL-/bin/sh} shell-script-name'.
> 
> > +        expr "X$cmds" : ".*"
> > +        if test $? -ne 0 || \
> > +           test `expr "X$cmds" : ".*"` -gt $max_cmd_len ; then
> 
> How about something like:
> 
>            if len=`expr "X$cmds" : ".*" &&
>               test $len -le $max_cmd_len; then
>              :
>            else
>              ...
> 
Ah yes, this looks good, and it works too.
Attached is another revised patch that incorporates the
suggestion above, as well as a method of removing the
reloadble object files.

I still have an unusual problem on one platform, the Alpha.
It seems that the number it reports is too large for it
to actually handle.  It looks like one of two things,
the length of arguments is different for ld and expr,
or the test in ltconfig.in is not quite right.
I think the problem may be that $CXX is getting a command from
libtool that isn't too long, but that cxx is passing a
command line to ld that IS too long.  Anyone agree?

Testing libtool on TKBool library on OSF1 anubis V4.0 1229 alpha
Find the actual command line length with wc -c
The length returned by wc -c is      19222
The length returned by expr "X$cmds" : ".*" is 19236  and the exit
status was 0
but the maximum command line length is 24577
using normal linking
<long command line snipped>
m          -lc   -msym -soname libTKBool.so `test -n "0.0.0:0.0" && echo
-set_version 0.0.0:0.0` -update_registry .libs/so_locations -o
.libs/libTKBool.so.0.0.0
cxx: Error: can't find or exec: /usr/lib/cmplrs/cxx/V6.3-002/ld
  : Arg list too long
cxx: Error: can't find or exec: /usr/lib/cmplrs/cxx/V6.3-002/ld
  : Arg list too long
make[3]: *** [libTKBool.la] Error 1
make[3]: Leaving directory `/tmp_mnt/icarus/build-axp40/src/TKBool'

I do intend to get all these quirks fixed, and to make sure that
goofy characters will still work, but I want to get the main
framework out there so that more eyes will see it.

Robert

-- 
Robert Boehne             Software Engineer
Ricardo Software   Chicago Technical Center
TEL: (630)789-0003 x. 238
FAX: (630)789-0127
email:  address@hidden

2001-02-20  Robert Boehne  <address@hidden>
        
        * ltconfig.in: Add a test to find the approximate limit
        to the length of command line arguments.  The number
        calculated here should always be lower than the actual
        limit.

        * ltmain.in: Test the length of the command line to be
        executed and use an incremtnal linking scheme if the
        command is too long to be interpreted without error.


Index: ltconfig.in
===================================================================
RCS file: /cvsroot/libtool/libtool/Attic/ltconfig.in,v
retrieving revision 1.246.2.44
diff -u -r1.246.2.44 ltconfig.in
--- ltconfig.in 2001/02/15 21:55:40     1.246.2.44
+++ ltconfig.in 2001/02/21 00:53:30
@@ -216,6 +216,7 @@
 ac_ext=c
 libext=a
 cache_file=
+max_cmd_len=
 
 ## Dependencies to place before and after the object being linked:
 predep_objects=
@@ -774,6 +775,21 @@
 no_builtin_flag="$ac_cv_prog_cc_no_builtin"
 can_build_shared="$ac_cv_prog_cc_can_build_shared"
 
+# find the maximum length of command line arguments
+echo $ac_n "finding the maximum length of command line arguments... $ac_c" 1>&6
+echo "$progname:@LINENO@:finding the maximum length of command line arguments" 
1>&5
+i=0
+testring="ABCDEF"
+while test `$CONFIG_SHELL $0 --fallback-echo "X$testring" >/dev/null 2>&1` == 
`echo "X$testring" >/dev/null 2>&1` &&
+        new_result=`expr "X$testring" : ".*" 2>&1` &&
+        max_cmd_len=$new_result &&
+        test $i != 32 # 1 MB should be enough
+do
+        i=`expr $i + 1`
+        testring=$testring$testring
+done
+echo "$ac_t""$max_cmd_len" 1>&6
+
 # Check to see if options -o and -c are simultaneously supported by compiler
 echo $ac_n "checking if $compiler supports -c -o file.$objext... $ac_c" 1>&6
 $rm -r conftest 2>/dev/null
@@ -2221,6 +2237,9 @@
 # Additional compiler flags for building library objects.
 pic_flag=$pic_flag
 pic_mode=$pic_mode
+
+# What is the maximum length of a command?
+max_cmd_len=$max_cmd_len
 
 # Does compiler simultaneously support -c and -o options?
 compiler_c_o=$compiler_c_o
Index: ltmain.in
===================================================================
RCS file: /cvsroot/libtool/libtool/ltmain.in,v
retrieving revision 1.200.2.44
diff -u -r1.200.2.44 ltmain.in
--- ltmain.in   2001/01/24 20:08:34     1.200.2.44
+++ ltmain.in   2001/02/21 00:53:30
@@ -950,7 +950,18 @@
          prev=
          continue
          ;;
-       rpath | xrpath)
+# Enable libtool to be executed with the -link-file 'filename'  argument
+       link-file)
+          if test -f $arg ; then
+            libobjs=`cat $arg`
+          else
+            $echo "$0 Error: Cannot find file $arg for -link-file argument" 
1>&2
+            exit 1
+          fi
+          prev=
+          continue
+          ;;
+               rpath | xrpath)
          # We need an absolute path.
          case "$arg" in
          [\\/]* | [A-Za-z]:[\\/]*) ;;
@@ -3007,14 +3018,101 @@
        else
          eval cmds=\"$archive_cmds\"
        fi
-       IFS="${IFS=     }"; save_ifs="$IFS"; IFS='~'
-       for cmd in $cmds; do
-         IFS="$save_ifs"
-         $show "$cmd"
-         $run eval "$cmd" || exit $?
-       done
-       IFS="$save_ifs"
+        if len=`expr "X$cmds" : ".*"` &&
+           test $len -le $max_cmd_len; then
+          echo "using normal linking"
+        else
+          # the command line is too long to link in one step, link 
incrementally
+          $echo "creating reloadable object files..."
+          # save the value of $output and $libobjs because we want to use them 
later
+          save_libobjs=$libobjs
+          save_output=$output
+          # clear the reloadable object creation command queue and initialize 
k to one.
+          concat_cmds=
+          objlist=
+          k=0
+          # loop over the list of objects to be linked
+          for obj in $save_libobjs
+          do
+            if len=`expr "X$test_cmds" : ".*"` &&
+                 test $len -le $max_cmd_len; then
+              objlist="$objlist"
+              if test $k -eq 0 ; then
+                # the first file doesn't need to have the previous one linked 
in.
+                eval test_cmds=\"$reload_cmds $objlist\"
+              else
+                # all subsequent reloadable object files will link in the last 
one created
+                eval test_cmds=\"$reload_cmds $objlist 
$output_objdir/$save_output-${klast}.$objext\"
+              fi
+            else
+              # the above command is almost too long
+              # increment k, the reloadable object file counter
+              klast=$k
+              k=`expr $k + 1`
+              # set the value of $output to the Kth object file
+              output=$output_objdir/$save_output-${k}.$objext
+              if test $k -eq 1 ; then
+                # the first file doesn't need to have the previous one linked 
in.
+                eval concat_cmds=\"$reload_cmds $objlist\"
+              else
+                # all subsequent reloadable object files will link in the last 
one created
+                eval concat_cmds=\"\$concat_cmds~$reload_cmds $objlist 
$output_objdir/$save_output-${klast}.$objext\"
+              fi
+              test_cmds=X
+              objlist=$obj
+            fi
 
+          done
+
+          # handle the remaining objects by creating one last reloadable 
object file
+          klast=$k
+          k=`expr $k + 1`
+          output=$output_objdir/$save_output-${k}.$objext
+          if test $k != 1 ; then
+            # all subsequent reloadable object files will link in the last one 
created
+            eval concat_cmds=\"\$concat_cmds~$reload_cmds $objlist 
$output_objdir/$save_output-${klast}.$objext\"
+          else
+            eval concat_cmds=\"$reload_cmds $save_libobjs\"
+          fi
+          i=0
+          while test $i -gt $k
+          do
+            i=`expr $i + 1`
+            delfiles="$delfiles $output_objdir/$save_output-${i}.$objext"
+          done
+          eval concat_cmds=\"\$concat_cmds~$rm $delfiles\"
+
+          $echo "creating a temporary reloadable object file: $output"
+
+          # loop through the commands generated above and execute them
+          IFS="${IFS=  }"; save_ifs="$IFS"; IFS='~'
+          for cmd in $concat_cmds; do
+            IFS="$save_ifs"
+            $show "$cmd"
+            $run eval "$cmd" || exit $?
+          done
+          IFS="$save_ifs"
+
+          libobjs=$output
+          # restore the value of output
+          output=$save_output
+
+          # expand the library linking commands again to reset the value of 
$libobjs for incremental linking
+          # Do each of the archive commands.
+          if test -n "$export_symbols" && test -n "$archive_expsym_cmds"; then
+            eval cmds=\"$archive_expsym_cmds\"
+          else
+            eval cmds=\"$archive_cmds\"
+          fi
+        fi
+        IFS="${IFS=    }"; save_ifs="$IFS"; IFS='~'
+        for cmd in $cmds; do
+          IFS="$save_ifs"
+          $show "$cmd"
+          $run eval "$cmd" || exit $?
+        done
+        IFS="$save_ifs"
+
        # Restore the uninstalled library and exit
        if test "$mode" = relink; then
          $run eval '(cd $output_objdir && $rm ${realname}T && $mv $realname 
${realname}T && $mv "$realname"U $realname)' || exit $?
@@ -3912,8 +4010,37 @@
 #          $run eval '(cd $xdir && ${LN_S} $obj $baseobj)' || exit $?
 #        fi
 #      done
+
+        eval cmds=\"$old_archive_cmds\"
 
-       eval cmds=\"$old_archive_cmds\"
+        if len=`expr "X$cmds" : ".*"` &&
+             test $len -le $max_cmd_len; then
+            :
+        else
+              # the command line is too long to link in one step, link 
incrementally
+              $echo "using incremental archive linking..."
+              objlist=
+              concat_cmds=
+              save_oldobjs=$oldobjs
+              for obj in $save_oldobjs
+              do
+                oldobjs="$objlist $obj"
+                objlist="$objlist $obj"
+                eval test_cmds=\"$old_archive_cmds\"
+                if len=`expr "X$test_cmds" : ".*"` &&
+                   test $len -le $max_cmd_len; then
+                  :
+                else
+                  # the above command should be used before it gets too long
+                  oldobjs=$objlist
+                  eval concat_cmds=\"\$old_archive_cmds~$concat_cmds\"
+                  objlist=
+                fi
+              done
+              oldobjs=$objlist
+              eval concat_cmds=\"\$old_archive_cmds~$concat_cmds\"
+              eval cmds=\"$concat_cmds\"
+        fi
       fi
       IFS="${IFS=      }"; save_ifs="$IFS"; IFS='~'
       for cmd in $cmds; do