Loadable module support

[Paul Smith]

Boris suggested the ability to load modules (dynamically loaded code)
into GNU make back in September.  This seems like a really interesting
idea to me.  Combined with make's ability to rebuild and re-exec, it
seems like it should be possible for people to include the source code
for their module with the rest of their code, and when they run "make"
it would be possible for make to build the dynamic module from source,
then load it.

I have been thinking about this capability and I thought I'd write a
proposal for commentary. 

The first thing we need to do is provide an external interface for
defining new functions.  This is pretty straightforward.  This will
allow loadable modules to create new functions in GNU make; at least
initially this will be the only interface available to loadable modules
for interacting with makefiles.  This aspect I'm already implementing.

Next we need a syntax for loading modules.  I was thinking of something
like this:

        load <name> [<pathname>]

The semantics would be something like this:

     1. See if the symbol "gmk_<name>_init" (or something?) is defined.
     2. If not, then loop through the following paths in order:
        <pathname> (if given), or the expansion of <name> against a list
        of patterns in a variable such as .LOADPATTERNS, the default
        value of which would be something like "%.o lib%.so".
     3. For each one in turn, try loading it with dlopen().  Should we
        support VPATH search algorithms for these as well?  Do we want
        to try to suppress normal dlopen() search algorithms
        (LD_LIBRARY_PATH etc.)?
     4. If they cannot be loaded, try to build them as a target.  If
        not, go to the next one.
     5. If none of them can be built, fail.
     6. If it can be built, build it and then re-exec.
     7. If it can be loaded, look up the symbol "gmk_<name>_init".  If
        it doesn't exist, fail.
     8. Once we have a symbol "gmk_<name>_init", run it.

The algorithm for re-exec for modules would be the same as for include:
first we read all the makefiles, then we try to rebuild them all, and if
any do rebuild we re-exec.

Do we need to introduce "-load" vs. "load", like we have "-include" vs.
"include", so that we can ignore errors if a module cannot be loaded?
The algorithm for building would be the same as for included files:
after all makefiles are read in, THEN we try to rebuild them.

I was wondering if we should allow a syntax like:

        load <name> [<pathname> [<arguments>]]

and then pass the string containing <arguments> to the "gmk_<name>_init"
function... maybe this would allow the makefile to customize the
behavior somehow.  I don't know if this is just adding valueless
complexity.

This design is informed by the way POSIX systems work (dlopen(),
dlsym()).  I wonder how this would work, or not work, with alternative
dynamic loaders such as Windows (I assume MacOS uses POSIX semantics as
well)?  Is it possible for VMS to use it I wonder?

My intention is to not make any extra effort (beyond defining an
interface for registering new functions) to provide any defined internal
interfaces for these modules.  If they want to take advantage of some
other functions in make code they can but there is no guarantee that
they will exist, unchanged, in the next release of GNU make.

Comments/ideas all welcome.

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <address@hidden>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.net
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist