Re: [MLB] Fix for arg list too long (another attempt)

[Robert Boehne]

Here is another attempt at an incrementally linking libtool.
I would like anyone so inclined to look at the ltmain.in patch
and point out any portability problems, and please make
suggestions if you have them!  ;)

Alexandre Oliva wrote:
> 
> On Feb 15, 2001, Robert Boehne <address@hidden> wrote:
> 
> > Alexandre Oliva wrote:
> >>
> >> On Feb 13, 2001, Robert Boehne <address@hidden> wrote:
> >>
> >> > +              output=$output_objdir/all_objs${k}.$objext
> >>
> >> This won't work in parallel builds.  You must name the temporary
> >> object files after the output.
> 
> > By named after, do you mean somthing like "all_objs$output" ?
> > Also, could you explain why this would break a paralell build?
> 
> Think of multiple huge libraries being linked concurrently in the same
> directory.  All of them would use all_objs0.o, then all_objs1.o, etc.
> If this occurs at the same time, things break.
> 
> I suggest $output-${k}.$objext.  Oh, and you may add commands to
> remove these temporary files when they've already been used.
> Otherwise, we may end up using quadratic space.
> 
> > Ah, yes, I see.  I had taken the approach that I should eval more
> > rather than less, but I see that I need to eval _exactly_ the
> > right number of times.  I will add a test for this in the final
> > version.
> 


-- 
Robert Boehne             Software Engineer
Ricardo Software   Chicago Technical Center
TEL: (630)789-0003 x. 238
FAX: (630)789-0127
email:  address@hidden

2001-02-15  Robert Boehne  <address@hidden>
        
        * ltconfig.in: Add a test to find the approximate limit
        to the length of command line arguments.  The number
        calculated here should always be lower than the actual
        limit.

        * ltmain.in: Test the length of the command line to be
        executed and use an incremtnal linking scheme if the
        command is too long to be interpreted without error.


Index: ltconfig.in
===================================================================
RCS file: /cvsroot/libtool/libtool/Attic/ltconfig.in,v
retrieving revision 1.246.2.44
diff -u -r1.246.2.44 ltconfig.in
--- ltconfig.in 2001/02/15 21:55:40     1.246.2.44
+++ ltconfig.in 2001/02/19 22:16:06
@@ -216,6 +216,7 @@
 ac_ext=c
 libext=a
 cache_file=
+max_cmd_len=
 
 ## Dependencies to place before and after the object being linked:
 predep_objects=
@@ -774,6 +775,21 @@
 no_builtin_flag="$ac_cv_prog_cc_no_builtin"
 can_build_shared="$ac_cv_prog_cc_can_build_shared"
 
+# find the maximum length of command line arguments
+echo $ac_n "finding the maximum length of command line arguments... $ac_c" 1>&6
+echo "$progname:@LINENO@:finding the maximum length of command line arguments" 
1>&5
+i=0
+testring="ABCDEF"
+while test `$0 --fallback-echo "X$testring" >/dev/null 2>&1` == `echo 
"X$testring" >/dev/null 2>&1` &&
+        new_result=`expr "X$testring" : ".*" 2>&1` &&
+        max_cmd_len=$new_result &&
+        test $i != 32 # 1 MB should be enough
+do
+        i=`expr $i + 1`
+        testring=$testring$testring
+done
+echo "$ac_t""$max_cmd_len" 1>&6
+
 # Check to see if options -o and -c are simultaneously supported by compiler
 echo $ac_n "checking if $compiler supports -c -o file.$objext... $ac_c" 1>&6
 $rm -r conftest 2>/dev/null
@@ -2221,6 +2237,9 @@
 # Additional compiler flags for building library objects.
 pic_flag=$pic_flag
 pic_mode=$pic_mode
+
+# What is the maximum length of a command?
+max_cmd_len=$max_cmd_len
 
 # Does compiler simultaneously support -c and -o options?
 compiler_c_o=$compiler_c_o
Index: ltmain.in
===================================================================
RCS file: /cvsroot/libtool/libtool/ltmain.in,v
retrieving revision 1.200.2.44
diff -u -r1.200.2.44 ltmain.in
--- ltmain.in   2001/01/24 20:08:34     1.200.2.44
+++ ltmain.in   2001/02/19 22:16:09
@@ -3007,13 +3007,90 @@
        else
          eval cmds=\"$archive_cmds\"
        fi
-       IFS="${IFS=     }"; save_ifs="$IFS"; IFS='~'
-       for cmd in $cmds; do
-         IFS="$save_ifs"
-         $show "$cmd"
-         $run eval "$cmd" || exit $?
-       done
-       IFS="$save_ifs"
+        expr "X$cmds" : ".*"
+        if test $? -ne 0 || \
+           test `expr "X$cmds" : ".*"` -gt $max_cmd_len ; then
+          # the command line is too long to link in one step, link 
incrementally
+          $echo "using incremental linking..."
+          # save the value of $output and $libobjs because we want to use them 
later
+          save_libobjs=$libobjs
+          save_output=$output
+          # clear the reloadable object creation command queue and initialize 
k to one.
+          concat_cmds=
+          objlist=
+          k=0
+          # loop over the list of objects to be linked
+          for obj in $save_libobjs
+          do
+            expr "X$cmds" : ".*"
+            if test $? -ne 0 || \
+               test `expr "X$test_cmds" : ".*"` -gt $max_cmd_len ; then
+              # the above command is almost too long
+              # increment k, the reloadable object file counter
+              klast=$k
+              k=`expr $k + 1`
+              # set the value of $output to the Kth object file
+              output=$output_objdir/$output-${k}.$objext
+              if test $k -eq 1 ; then
+                # the first file doesn't need to have the previous one linked 
in.
+                eval concat_cmds=\"$reload_cmds $objlist\"
+              else
+                # all subsequent reloadable object files will link in the last 
one created
+                eval concat_cmds=\"\$concat_cmds~$reload_cmds $objlist 
$output_objdir/$output-${klast}.$objext\"
+              fi
+              test_cmds=X
+              objlist=$obj
+            else
+              objlist="$objlist"
+              if test $k -eq 0 ; then
+                # the first file doesn't need to have the previous one linked 
in.
+                eval test_cmds=\"$reload_cmds $objlist\"
+              else
+                # all subsequent reloadable object files will link in the last 
one created
+                eval test_cmds=\"$reload_cmds $objlist 
$output_objdir/$output-${klast}.$objext\"
+              fi
+            fi
+          done
+          # handle the remaining objects by creating one last reloadable 
object file
+          klast=$k
+          k=`expr $k + 1`
+          output=$output_objdir/$output-${k}.$objext
+          if test $k != 1 ; then
+            # all subsequent reloadable object files will link in the last one 
created
+            eval concat_cmds=\"\$concat_cmds~$reload_cmds $objlist 
$output_objdir/$output-${klast}.$objext\"
+          else
+            eval concat_cmds=\"$reload_cmds $save_libobjs\"
+          fi
+          $echo "creating a temporary reloadable object file: $output"
+
+          # loop through the commands generated above and execute them
+          IFS="${IFS=  }"; save_ifs="$IFS"; IFS='~'
+          for cmd in $concat_cmds; do
+            IFS="$save_ifs"
+            $show "$cmd"
+            $run eval "$cmd" || exit $?
+          done
+          IFS="$save_ifs"
+
+          libobjs=$output
+          # restore the value of output
+          output=$save_output
+
+          # expand the library linking commands again to reset the value of 
$libobjs for incremental linking
+          # Do each of the archive commands.
+          if test -n "$export_symbols" && test -n "$archive_expsym_cmds"; then
+            eval cmds=\"$archive_expsym_cmds\"
+          else
+            eval cmds=\"$archive_cmds\"
+          fi
+        fi
+        IFS="${IFS=    }"; save_ifs="$IFS"; IFS='~'
+        for cmd in $cmds; do
+          IFS="$save_ifs"
+          $show "$cmd"
+          $run eval "$cmd" || exit $?
+        done
+        IFS="$save_ifs"
 
        # Restore the uninstalled library and exit
        if test "$mode" = relink; then
@@ -3914,6 +3991,32 @@
 #      done
 
        eval cmds=\"$old_archive_cmds\"
+        `expr "X$cmds" : ".*"`
+        if test $? -ne 0 || \
+           test `expr "X$cmds" : ".*"` -gt $max_cmd_len ; then
+          # the command line is too long to link in one step, link 
incrementally
+          $echo "using incremental archive linking..."
+          objlist=
+          concat_cmds=
+          save_oldobjs=$oldobjs
+          for obj in $save_oldobjs
+          do
+            oldobjs="$objlist $obj"
+            objlist="$objlist $obj"
+            eval test_cmds=\"$old_archive_cmds\"
+            `expr "X$test_cmds" : ".*"`
+            if test $? -ne 0 || \
+               test `expr "X$test_cmds" : ".*"` -gt $max_cmd_len ; then
+              # the above command should be used before it gets too long
+              oldobjs=$objlist
+              eval concat_cmds=\"$old_archive_cmds~$concat_cmds\"
+              objlist=
+            fi
+          done
+          oldobjs=$objlist
+          eval concat_cmds=\"$old_archive_cmds~$concat_cmds\"
+         eval cmds=\"$concat_cmds\"
+        fi
       fi
       IFS="${IFS=      }"; save_ifs="$IFS"; IFS='~'
       for cmd in $cmds; do