[Gcl-devel] omnibus January cvs commit

[Camm Maguire]

Greetings!  I've decided to commit my work in progress inlining tree,
mostly because I'm off for a month now and I don't want to forget
where work left off.

As before stated, this is a decent sized change, so users might want
to save an old copy if desired.  Same can be retrieved from cvs by
using the date restriction switch -D.

The main idea here is to support generic lisp function inlining.  GCL
has several layers of somewhat ad-hoc inlining of particular functions
historically.  The first is the "c snippet" layer represented by
string expressions found in gcl_cmpopt.  Next came specific compiler
"c1" functions to handle each inline.  Then came a series of compiler
macro functions or 'expanders' I wrote to extend the mechanism to more
functions in a manner closer to the lisp level.  Likewise, functions
could have 'type-proagator functions defined in the compiler to fine
tune the result type given the argument types.  

Obviously, this idea does not scale to the generic lisp function.  It
has long been a goal to define the function in one place, and deduce
all necessary compiler information from this definition.  Since cvs
carries the function src in the native image, this is now much closer
at hand. 

Most of the old compiler macros have been eliminated.  sort and the
aref functions still need replacing.  compiler macros now should not
be used for function redefinition, but for call form simplification
when possible,  i.e. (+ 1 2 3) -> (+ (+ 1 2) 3).  Not quite all of the
benefit of the old compiler macros can be had with the new system, so
we may want to mitigate this policy later.  The primary reason is
&rest, which of course conses on each call and can be eliminated if
the function is translated into a macro.  &rest can be mitigated
somewhat by allocating on the C stack, and of course by a new
possibility open to us with this system -- inlining apply, which can
then share structure with &rest so there is no consing at all.  

Further mitigation might be had by defining

(defun mapcar (fun l1 &rest lists) ...)
to
(defun mapcar (fun l1 &optional (l2 nil l2p) &rest lists) ...)

then consing can be avoided unless three lists are supplied, etc.
This makes the function a bit uglier, and just delays the consing
issue to a less likely case, so I have not done this at the moment.

In sum, it might be good to check code sizes and speeds for the map
functions in the old and new paradigm for discussion on this point
later.  We can always retrieve what we might want from the compiler
macros. 

Speaking of which, disassemble now inlines assembly with source thanks
to a helpful switch from objdump.  And disassemble will work on
previously compiled functions, pulling the source from the database.
Will try to get compile to subsume si::recompile in like fashion.

The C code produced has some enhanced commenting features.  Fastlinks
are commented with the lisp function name, and inlines show the form
in a C comment.

The compiler has simultaneously been made more efficient, and loaded
more heavily -- don't know what the net effect is yet.  The primary
load is in making info structures and their associated arrays.  First
off, we use a single constant empty array for all blank infos at
significant savings.  Next, unnecessary nesting of c1 forms (let,
progn, ....) is at least partially avoided.  Unused variables are
trimmed in the c1 pass.  This latter point depended on extending the
type system to handle 'member types so that the value could be used by
the compiler inplace of the variable binding at runtime.  This was
also key to getting function inlining to replace compiler macros in
the first place, as one needs to be able to read at compile time, say,
the constant functions supplied as :test 'eq in the arguments, when
converted to a let* like
           (let* ((a1 :test)(a2 'eq)(fn (cond ((eq a1 :test) a2) (('eql)))))
                 (funcall fn ...))

The guts of the lambda list binding are in the function blla.  The
inlining occurs in maybe-inline.  I need to make sure that apply to a
known function registers the callee still.   maybe-inline pulls the
source, binds the lambda list into a let*, and returns the c1 pass
there-over if the 'size' is less than a certain threshold set
according to the prevailing value of *space*.  0 means never, and for
each increment up to 3, another 1000 nodes of the c1 tree form are
allowed to pass.  There is also an experimental facility to hash the
results for compiler performance, under the hash key of the function
name, the argument types, and whether the arguments have side
effects.  This does speed things up considerably, but is still
limited.  Any c1form which refers to constant data, or functions which
close over surrounding variables, are unhashable until we get a
different mechanism for storing the data location as a variable to be
set in pass2 in the c1form.  The hashing can be disabled at the moment
via compiler::*hash-inlines*.  In general, I'd prefer to avoid this
for a more applicative algorithm if possible.

lambda functions which are replaced by let forms normally would result
the the inclusion of an LC closure in the C file anyway, but I've
tried to minimize or eliminate this through the unused variable trimming
mentioned above.  There still is an issue that variables referenced
across such a boundary cannot be unboxed even when the boundary is
later eliminated through inlining, as the ref-ccb has already been
set.  This needs fixing. (i.e. (let ((i 0)) (mapcar (lambda (x) (incf
i)) y))).

In general, more thought needs to be given to the tradeoffs of space to
speed, but at least now there is some customizability.  the really
killer with GCL closures at the present is that 1) the environment is
consed on every call by default, and 2) arguments are passed by the
value stack.   1) can be alleviated if the previously built in
'turbo-closure' mechanism were extended.  2) awaits the decision on
whether to call vararg/mvret/closure functions with globals or special
prepended arguments on the C stack.

There are hooks in funlink.c for a non-special-gprof-build profiler
based on the idea of using the existing fastlink mechanism to count
and time calls in an optimal image when turning off fast links at
runtime.  This does not work yet.

At the moment, there is a problem with the ansi tests on the final
recompiled image that needs chasing down.

Several ansi-test errors refer to argument checking in :test and :key
functions.  I've made these fail unless fast links are turned off,
which is the way I plan on running the suite.  With fast links on, no
arg number checking is done on anonymous funcall calls for speed.
Perhaps this should be keyed to the safety setting.  Speaking of
which, tracing of such calls works, but not yet for apply -- this
needs fixing too.

I've cut down the sig conflicts requiring recompile significantly.  All
files in lsp and cmpnew are done by the time saved_gcl is written
(i.e. .o file contents need no recompilation later on).  Will attempt
same on mod, pcl, and clcs directories later.

I've rewritten a number of basic functions in lisp, as of course, C
functions cannot be automatically inlined.  In this regard, I've even
managed to get copy-tree, sublis, and subst et. al. to be inlinable
despite the apparent recursion problem, by writing them non-recursively
using a dynamic-extent cons stack, which of course uses the C stack
like recursion anyway.

Inlining is not just useful for performance, but for type propagation
too.  What is the type of (reverse (list 1 2 3))?  To tell it is a
proper-list, you must either inline and branch eliminate, or define
a special type-propagator function.  Here, even when we don't inline,
we hash the results of inline attempts.  This can be used as an
effective automatic generic type propagator in the near future.

There has been some reworking of the meanings of the safety levels.
Previously, 0 is no checking, 1 was argument number checking at
function head, 2 was prevent calls to non-checking functions via C
inlines, and 3 was the dreaded 'funcall_with_catcher'.  1 and 2 are
now conflated with lisp inlining, so what we do now is to eliminate
check-types at the head of function bodies at safety 0 only.  At
safety 1, we do the check-type, and rewrite the rest in a let with the
variable declared to the checked-type.  At all levels we read the
check-type in setting the function signature.  The idea is:

(defun foo (....)
(declare (...))
(check-type x list)...;this is the magic position for check-types

body
(check-type foo h);this check-type has no magic
)

This way we get optimal performance inlining a builtin function which
must check its args in general for compliance and other reasons.

To get the most out of apply and &rest, we need to resurrect list
types up to some limit, i.e. (cons symbol (cons fixnum null)).  At the
very minimum, the list length can be read allowing apply to convert to
a funcall if desirable.  All GCL's basic calling mechanisms (short of
possible inlining) prefer funcall.  But even more, (cons (member foo)
null) can be precessed as a constant list at compile time.

A weakness in our type system is (member (a b c)).  we treat cons
types as outer joins over other types, but this type does not appear
to be representable in this framework.  (i.e. the cons identity is
needed too.)  Perhaps someone might point out the way -- we use
Baker's 'orthogonal kingdom' type algorithm.

Cleared up quite a few C warnings -- some lisp warning cleaning
remains. 

Bootstrapping has been improved somewhat -- we simply load boot.lisp
in unixport as opposed to compiling all of lsp and cmpnew interpreted.

We still have to handle setf functions right by interning a symbol in
a setf package.

A few basic functions still need good lisp implementation -- map,
map-into, concatenate, replace, mismatch, ....  Then gcl_cmpmap should
be obsolete.

make-list, list, cons, and now list* can all be dynamic-extent and
allocate on the C stack.  Alas, this means make-list itself cannot be
automatically inlined.

portable-source still needs to capture deftypes.

characters are now eql-is-eq.

Shaved two words off all C vector types.

This stuff still needs hardening -- users beware -- still feedback
always appreciated.

More later, most likely in a month's time ...

Take care,

Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah