Re: Guile support in GNU make

[Boris Kolpackov]

Hi Paul,

Paul D. Smith <address@hidden> writes:

> On Wed, 2011-09-21 at 21:04 +0200, Boris Kolpackov wrote:
>
> > The nice thing about the GCC plugin architecture is that the plugin
> > is loaded into the GCC address space. This means that it can access
> > all the functions, data structures, etc., that built-in GCC code can
> > access. I think the same approach will work fairly well for GNU make
> > as well.
> 
> This is actually the part that worries me the most about this idea.  As
> you're aware the internals of GNU make are in no way clean and
> protected: global variables abound and there are few APIs internally
> that allow for ease-of-use.  I would be very nervous about exposing that
> to user-written code, without any control.  I guess if it breaks they
> get to keep both pieces, but still.

It is exactly the same situation in GCC, global variables, "messy"
internal structures, etc. Initially I also thought that they are
nuts to expose this. But the more I worked with it the more I
realized that this is probably the best thing that they could have
possibly done. It allows me to do pretty much anything that the
built-in GCC code can do without having to re-implement some non-
trivial algorithms (e.g., name lookup or C++ template instantiation)
myself. I think if we try to limit what a plugin can do using some
"protocol", we will find people re-implementing functionality that
is already in make or complaining that certain things are no possible.


> Second, I wonder how portable this capability really is.  It works well
> on Linux but how easy is it to get similar behaviors on the other
> POSIX-based systems, much less Windows etc.?

I know for sure it works well on Linux, Solaris, and Mac OS X. I also
believe Windows has similar support, just slightly different API. The
trickiest part are platforms like AIX with bizarre shared object models.

The other thing that I think GCC folks got right is that they didn't
try to support plugins on all the major platforms right away. Instead,
at first, they just made sure it works on Linux. In a matter of a few
minor releases, people submitted patches that fleshed out support for
Solaris and Mac OS X. I am sure we could borrow quite a bit of that
code from GCC.

The same thing about the protocol idea -- rather than figuring and
implementing (or abandoning the whole idea because things get to
complex) how to limit what people can do with a plugin, GCC folks
just let you do anything your want: use GCC global variables, call
functions, etc.


> Still it's a cool idea that make would build its own loadable module and
> then load it :-).  Making this easy might be complex though: ideally
> you'd like to add some more builtin variables that provide the rules for
> building a loadable object on the local system, based (I guess) on
> configure-discovered commands.  Would this require libtool, which I
> really hate?  Or could it be done more simply?  I don't know.

I think, as a first iteration, letting the user figure out how to
build the loadable object will be sufficient. All we will need to
do to allow this is install make headers. But, I agree, if we could
later provide portable built-in rules for building loadable objects,
that would be great.


> > And if we have something like this, it will be fairly straightforward
> > to add Guile support as just another plugin.
> 
> Actually from what I've seen I think it will be difficult to add Guile
> support in this way: because Guile is garbage-collected you have to play
> some tricks with your program's main() so that Guile can understand how
> the stack is used/ordered.  I'm not sure how possible it is to bolt on
> Guile support solely through a dlopen()'d shared object.

I have never used Guile myself but heard that it is quite popular as
an embedded language. Surely someone has figured out how to use it
from a loadable object.

The cool thing about a plugin is that it can be used to integrate other
scripting languages, not necessarily embeddable, into GNU make. Think
of a plugin that defines the $(python) function ;-).

Boris

-- 
Boris Kolpackov, Code Synthesis        http://codesynthesis.com/~boris/blog
Compiler-based ORM system for C++      http://codesynthesis.com/products/odb