RE: [Chicken-users] RE: Win32/cygwin dynamic loading

[Jonah Beckford]

Oh yeah, just forget to make something explicit ... this dynamic loading
was just a proof of concept.  The goal was to find a set of principles
to support dynamic loading on the broadest of platforms (basically, code
normalization).  Now that I think we have most of the principles
(outlined below), I would still use __declspec(dllimport) for Win32
(which should work if the "principles" are followed) etc. and only use
the "primitive" shared loading technique for trully primitive platforms
and maybe for thread safety.

Principles:

- A one-to-one mapping between units and shared libraries
- Runtime, as opposed to link-time, resolution of units (using
load/load-library)
- Splitting of libchicken into a] libchicken (and all its variants like
libembedded, etc.) for the statically linked necessities of CHICKEN
(that is, all the CHICKEN_xxx functions and main/WinMain/DLLMain) and b]
libruntime which has the remainder of runtime.c.  Libruntime actually
has both a static version and a shared version, so conceptualize
libruntime as libruntime-static and libruntime-shared.
- The call to (load "xxx") for a dynamic module must not use any prefix
or suffix (like prefix=lib, suffix=.so).  In fact, it will be difficult
to support any paths like (load "/usr/local/chicken/srfi-13") ... not
only is that system-dependent, the parsing routine used to search for
alternatives like /usr/local/chicken/cygsrfi-13-0.dll would be quite
difficult.  The dynamic modules must be locatable with a standard
mechanism (maybe search the current directory, followed by the
system-dependant library search path, followed by CHICKEN_HOME).  This
also means they must be installable with a standard mechanism (I like
William Annis' suggestion of something like Python's Distutils or Perl's
Makefile.PL; I haven't looked at "eggs" yet, but I have a suspicion they
are for pure Scheme code).
- Placing a FUNC() wrapper around the functions in runtime.c, so that
function names can be mangled if on a primitive platform
- Specification of -static or -shared when using chicken-config (so it
can pick the right libruntime) and change the compiler -D defines.  This
also means the chicken-config, or its replacement, must be useable on
all platforms.
- Using temporary variables and constants to iterate over arrays.
Correct:
    /* Mark collectibles: */
    if((msp = C_collectibles) != NULL)
      while(*msp != NULL) mark(*(msp++));
Needs 2 modifications:
  for(i = 0; i < C_symbol_table_size; C_symbol_table[ i++ ] =
C_SCHEME_END_OF_LIST);
Corrected version of above:
  C_word *sym; /* access CHICKEN arrays with a temporary pointer */
  const int TABLE_SIZE = C_symbol_table_size; /* don't check
C_symbol_table_size every iter */
  for(i = 0, sym = C_symbol_table; i < TABLE_SIZE; *(sym++) =
C_SCHEME_END_OF_LIST);

Jonah

-----Original Message-----
From: Jonah Beckford [mailto:address@hidden 
Sent: February 21, 2003 10:56 PM
To: 'Jonah Beckford'; 'felix'
Cc: 'Felix Winkelmann'; 'address@hidden'
Subject: RE: [Chicken-users] RE: Win32/cygwin dynamic loading


Changed web site to a permanent location ... is now:

http://beckford.netfirms.com/hobbies/chicken/

(and SWIG CHICKEN is at http://beckford.netfirms.com/hobbies/swig/)

MinGW has also been tested with the dynamic loading (use the
all-mingw.sh script).

Jonah

-----Original Message-----
From: Jonah Beckford [mailto:address@hidden 
Sent: February 21, 2003 1:59 AM
To: 'felix'
Cc: 'Felix Winkelmann'; 'address@hidden'
Subject: RE: [Chicken-users] RE: Win32/cygwin dynamic loading


I have the modifications to Chicken that allow dynamic loading of
modules in a portable fashion.  Right now only cygwin has been tested
(mingw had also been, but not lately).  If you have cygwin, then run the
all-cygwin.sh script first because the bootstrapping process is not easy
to do by hand.

http://beckford.homeip.net/chicken/

The web site is up most but not all of the time.

Major changes:
- (declare (uses xx yy)) is basically translated into (load "xx-unit")
(load "yy-unit") conditionally at the C source code level.
- while not enforced, the idea is that 1 unit == 1 dynamic module.  No
bundling of many units into one shared library (although you still may
if you don't care about portability).
- (load "xx-unit") will look for xx-unit, xx-unit.dll, cygxx-unit.dll
and libxx-unit.dll on cygwin.  It does .so and .sl and its variants as
well, just hasn't been tested on those platforms yet.  If can't find a
module, then will proceed with the original Chicken rules (like looking
for a xx-unit.scm).
- on "primitive shared" platforms, all function calls and variable
accesses that are defined in chicken.h are done through a data structure
(of function pointers and variables).  This is done transparently
through macros.  So a function call C_string(x,y,z) is macro expanded to
(*(C_shlib->priv_C_string))(x,y,z) and a variable access to
C_collectibles is macro expanded to (*(C_shlib->priv_C_collectibles)).
When a unit is loaded through (load "xx-unit"), its own C_shlib is
initialized with the proper values.  Basically, we simulate the
indirection method used by ordinary DLLs, but we define it in a way
(through C_shlib) such that it works on most platforms.
- there is a script gen_shlib.sh that writes all the macros and all the
code for you ... just add a new variable or function to chicken.h, run
gen_shlib.sh and nothing else to do.
- MinGW is done through the ./configure script, not makefile.mingw.

Benefits:
- Should work on all shared library platforms because each xx-unit is
completely self-contained.  In fact, the only requirement is that the
platform have some object file format that supports relocatable code
(ie. gcc -PIC) and a predefined entry point (either a symbol or an
absolute address).
- Can have circular dependencies between xx-unit and yy-unit because the
units are dynamically loaded.
- Quite unexpectedly, the technique can be used to make Chicken thread
safe as well (a thread can get its own C_shlib).  The remaining static
variables in runtime.c would have to be moved to chicken.h (trivial;
I've already done half), the static C_[xxx_]toplevel_initialized in
Chicken generated code would have to be removed (trivial) and each
C_shlib access would have to go into thread local storage (not trivial).

Drawbacks:
- Is slower because of the pointer dereference of C_shlib.  This becomes
an issue inside loops (good example: looping over C_collectibles).  The
C code developer has to be careful and introduce temporary variables
above large loops.  With the temporaries (which is nothing more than a
hint to the C compiler to not dereference the pointer every step in a
loop), the C_shlib method would be the exact same speed as access
__declspec(dllimport/export) variables.
- I'm sure there are more ...

Jonah

-----Original Message-----
From: felix [mailto:address@hidden 
Sent: February 18, 2003 4:02 PM
To: Jonah Beckford
Cc: 'Felix Winkelmann'; address@hidden
Subject: Re: [Chicken-users] RE: Win32/cygwin dynamic loading


Jonah Beckford wrote:
> I took your comment about solution #2 into mind ... I don't need to 
> pass in the toplevel pointer.  I did need to split up libchicken into 
> two libraries though (a static library that has defns that must be 
> statically linked [CHICKEN_initialize/run/invoke, main/WinMain/DLLMain

> and global data], and a runtime library for everything else).
> 

Sounds reasonable. But we still have the option of generating `main'
entry-points into compiled code. Yet, the method youu describe seems
better.

> Also, my 'better' solution to #3 is basically to resolve the
> C_<unit>_toplevel at runtime instead of link time.  Circular 
> dependencies should be fine using the new method; no need to touch the

> srfi-37 unit :)

Too late. I removed the serialization stuff from the lolevel unit and
srfi-37 does the option-parsing by hand (which is more efficient
enyway).

I'm checking in the changes just at this moment...

> 
> Also, I was using a Win32 dload_2 stub that called
> LoadLibrary/GetProcAddress.  This works on all Win32 platforms ... 
> Cygwin does have a dlopen/dlsym, but it is just a tiny wrapper around 
> LoadLibrary.

Yup.

> 
> It's mostly finished ... I will tar.gz it up and make it available for
> testing in a day or two.

Great!


cheers,
felix