Re: use of wildcard function recursively

[Ted Stern]

On 16 Mar 2006 07:59:31 -0800, Jason Lunz wrote:
>
> address@hidden said:
>> Or, you may be thinking of the maximum size of the environment
>> (env. vars + command line etc.)  This is only relevant when actually
>> invoking a new process, and it's a kernel-level limit (that means it
>> doesn't matter what your program is: shell, make, whatever: the limit is
>> the same for everything).
>
> speaking of which, what do you do when you hit this limit? I have a
> makefile that builds a huge list of filenames, then selects subsets of
> this list for various purposes, ie:
>
> ALLFILES:=$(shell find -type f)
> CFILES:=$(filter %.c, $(ALLFILES))
> JAVAFILES:=$(filter %.java, $(ALLFILES))
>
> I've found this to be faster than running multiple find commands.
>
> However, on this project, ALLFILES has grown to the point where it no
> longer fits on a single command line. I can't do this, for example:
>
> allfiles.list: $(ALLFILES)
>       echo $(ALLFILES) | tr ' ' '\n' > $@
>
> because that command is just too big.
>
> What's the best workaround for this in GNU make?
>
> Jason

[CC'd to newsgroup/list]

I encountered this when I had about 10K object files I wanted to
compile in parallel and then archive in one fell swoop into a static
library.  I had the files distributed through a big directory
hierarchy with long pathnames, so it totally swamped the shell command
line length limit.

Over time (with help from Paul Smith), I developed a make function
called MAKE_LONG_LIST.  Here's how I use that function to build my big
library, for example:

    AR_LIST = $(BLDDIR)/scripts/$(basename $(@F)).list
    AR_OBJS = $(subst $(OBJDIR)/,,$(filter %.o,$?))
    XARGS = xargs
    AR = ar
    ARFLAGS = -rvsc
    TIME = /usr/bin/time -p

    include $(SCRIPTDIR)/MakeLongList.mk

    %.a:
        @printf "\n\t%s\n\t\t%s\n\n" \
                "Started archiving $@ on" \
                "`date`"
        $(call MAKE_LONG_LIST,$(AR_LIST),$(AR_OBJS))
        cd $(OBJDIR); $(TIME) $(XARGS) $(AR) $(ARFLAGS) $@ < $(AR_LIST)
        @printf "\n\t%s\n\t\t%s\a\a\a\a\a\n\n" \
                "Finished archiving $@ on" \
                "`date`"

The MAKE_LONG_LIST function generates a file with all of the contents
of the long MAKE variable, one element per line.  Then I use the
'xargs' function to generate command lines that conform to the shell
line-length restriction.  

I'm attaching MakeLongList.mk below.  It has some ksh/bash shortcuts
(I use 'SHELL = /bin/ksh') which you may prefer to replace with
something more portable.

Ted
-- 
 dodecatheon at gmail dot com
 Frango ut patefaciam -- I break so that I may reveal

#
# MAKE_LONG_LIST:
#
# MAKE function to be called when a long variable list needs to be
# put into a file with one item on each line.
# Two arguments:
# $(1) = list name
# $(2) = MAKE variable to put into list
#
# The resulting list can be used in combination with xargs to get
# around shell command-line character-length restrictions.
#
# The current hack is encapsulated in a separate included Makefile
# to hide the confusing recursive function.
#
# RECURSIVE version is requires GNU make-3.80 or higher, but is shorter and
# will handle any size list.
#

# CALC:
# Math helper function -- Use shell to do simple operations.
# Relies on ksh/bash $(( )) syntax.
# If you don't like this or need something more portable, change to
# CALC = $(shell echo "$(1)" | bc)
CALC = $(shell echo $$(($(1))))

nextindx = $(call CALC,$(3)+1)
nextword = $(word $(nextindx),$(2))
nextlist = $(wordlist $(nextindx),$(words $(2)),$(2))

# ADD_TO_LIST:
# A recursive function definition to handle iteration
# ARG 1 = filename to contain list, one entry per line
# ARG 2 = variable containing a very long list
# ARG 3 = Increment
define ADD_TO_LIST
        @printf "%s\n" $(wordlist 1,$(3),$(2)) >> $(1)
        $(if $(nextword),$(call ADD_TO_LIST,$(1),$(nextlist),$(3)),@echo)
endef

# Actual MAKE_LONG_LIST definition is a wrapper around the recursive call.
# Note that the list file is first cleared out using Bourne/ksh/bash syntax.
# I just arbitrarily set the iteration size to 500, but you can make it
# smaller if the system environment length is still too large.
define MAKE_LONG_LIST
        : > $(1)
        $(call ADD_TO_LIST,$(1),$(2),500)
endef