Re: [Axiom-developer] BAD tim

[Camm Maguire]

Greetings!

"Bill Page" <address@hidden> writes:

> Tim,
> 
> > POINT 1: Axiom needs to be future-proofed.
> > 
> >    Axiom needs to be torn down, re-architected, re-thought
> >    and rewritten cleanly so it can be modularized, maintained,
> >    and extended. Which implies a major rewrite, likely done
> >    by me.
> >
> 
> I am not convinced that this is necessary or even possible,
> although I do not doubt your knowledge or your ability to do
> this - given enough time. According to the history of Axiom
> that you have described in various previous emails, Axiom has
> already undergone several cycles of re-writing. I don't see
> any evidence that such re-writing has resulted in making
> Axiom any more "future-proof".
> 

"Future-proofing", IMHO, is a continous process by any living
project.  It requires an active developer base intimately familiar
with the system details which can only be acquired gradually through
time.  Developing this base is the key issue IMHO.

> > POINT 2: The rewriting will change existing code. 
> > 
> >    If we look at the "start-up sequence" we see:
> > 
> >               restart (lisp)
> >    init-memory-config (lisp)
> >              initroot (lisp)
> >                reroot (lisp)
> >                  spad (boot)
> >               runspad (boot)
> >            ncTopLevel (boot)
> >             ncIntLoop (lisp)
> >               intloop (boot)
> >   SpadInterpretStream (boot)
> >    intloopReadConsole (boot)
> > 
> >   and, if you follow the side-paths you'll see the constant
> >   switching between boot and lisp at every level in nearly
> >   every function.
> > 
> >   This inhibits "collapsing and rewriting" code which will 
> >   have to happen during the rewrite. Functions will either
> >   have to be collapsed into lisp or boot.
> >
> 
> As a long term strategy, I agree, however as I have stated
> I think boot is the better choice.
>  
> >   Historically this interleaving occurs because new function
> >   was added/deleted/replaced. The "nc" prefix implies "New
> >   Code", a newly rewritten parser that partially replaced the
> >   previously newly rewritten parser. The "shoe" code, which is
> >   the new, improved "boot" language exists in there. The "meta"
> >   code used to exist but is gone. Addition of Fortran support
> >   (for NAG), Foreign Language calls (for graphics and the
> >   browser), Lisp vendor specific code, etc. required lisp.
> >
> 
> Why was lisp required? Surely it is possible to do these
> things in BOOT. In fact, aren't most of the NAG extensions
> written in BOOT? E.g. ls src/interp/nag*

Existing boot code can obviously be automatically converted into lisp,
but the converse is not true to my knowledge.


>  
> > 
> > POINT 3: Boot is a dead language. 
> > 
> >    There are approximately 10 people still living who have coded
> >    in boot and every one of them is doing something else with
> >    their life. You have to be careful in boot. It is case sensitive 
> >    so APPEND is not the same as append.
> 
> Wow, that comment certainly dates you! :) Almost all programming
> languages in the last 20 (30?) years have been case sensitive.
> Usually that is considered an advantage.
> 
> >    It is often not clear what a construct will translate to and
> >    you end up reading the lisp to learn. A single, misplaced
> >    space will change the meaning of the code.
> 
> That is true of most programming languages. Understanding a
> "construct" by seeing how it "translates" is usually just a
> matter of lack of fluency in the language. This can be a result
> of inadequate documentation and/or experience. In a language
> that syntactically depends on indentation, of course misplaced
> spaces can change the meaning of the code. But this is not
> essentially different than misplaced commas or braces in other
> languages.
> 

I feel that syntax is largely a matter of taste from the language
user's perspective.  There is an immensse difference from the compiler
writer's perspective (more below).  In lisp, the user can easily blend
into the compiler writer by writing macros and compiler macros in
*very* few lines of code.  Personally, I find whitespace with syntax
quite foreign, but I can get used to just about anything if it is
*flexible enough to be widely used, thereby granting extensive
experience through doing*.  I think lisp wins over boot here.  One
might make the argument that the whole thing should be written in
vanilla C for maximal mindshare, power, and available code to learn
from, but most would find this extreme I suspect.  Going in the other
direction is bound to limit our potential contributor base even more.
The lisp user base is decent, though not spectacular, though once
was.  What I mean by contributor here is someone who can work with the
internals of axiom as well as add new mathematical algorithms in the
upper level language.

> > 
> > POINT 4: Boot SERIOUSLY complicates the Axiom system. For
> > instance:
> > 
> >   1) Bootstrapping boot
> > 
> >    The boot compiler is written in boot and needs to be compiled
> >    using itself. If you have a running Axiom system this is not
> >    a problem. However if you start from a clean lisp system it
> >    is a big problem. I cheesed up a way to do this but it is
> >    fragile and ugly. But given that we build from scratch there
> >    is no other way.
> 
> But that is exactly how the C programming language works. The
> Gnu C compiler is written in C and bootstrapped in a similar
> way.
> 
> Also, the current Axiom build starts by first building GCL from
> the C sources. This is also a kind of bootstrap process. I don't
> see that the BOOT level adds much to this already existing
> complexity.
> 

Yes, but do we want to be in the compiler business really?  Or rather
in the math business?  Axiom can be built without building gcl at
present, so boot bootstrapping is the most complex absolutely required
part.  I agree with your other statement though -- what we have works,
so its complexity is not paramount.


> > 
> >   2) Makefiles come in multiple stages
> > 
> >    Boot forces Axiom to be built in stages. Thus Axiom cannot 
> >    use modern tools like ASDF. A pure common lisp Axiom interpreter
> >    can be built and loaded directly into a lisp image, interpreted
> >    or compiled.
> 
> I thought there was already a version of interpsys called debugsys
> which provides an interpreted environment that is more friendly to
> debugging in lisp?
> 
> Not being a lisp programmer, it is entirely unclear to me what
> advantages ASDF might have over more general tools like 'make'
> especially considering that the Axiom source files are actually
> literate programming documents (pamphlets), not lisp source code.
> 

I agree with Bill here, though I don't yet grok asdf.

> > 
> >    Lisp, BOOTSYS, DEPSYS, INTERPSYS, AXIOMSYS. These can all be
> >    collapsed if Boot disappears.
> > 
> 
> Although I still recall that this looked very complicated to me
> when I first started working with Axiom, now each of these stages
> in the current Axiom build process makes sense to me. Since Axiom
> is a large and complex system, building it a "layer" at a time
> seems quite logical. If these were all collapsed into one, then
> I worry that this would all start to look impossibly complicated
> to me again.
> 
> >   3) lsp, lisp, clisp, ${LISP}
> > 
> >    All of these exist because boot exists. For example, .clisp
> >    is a translated boot file whereas a lisp file is hand coded.
> >    ${LISP} exists to smooth over this bump. I won't go into the
> >    historical reasons why these came about but they exist as a 
> >    side-effect of boot.
> > 
> 
> Just another type of intermediate file, and Axiom already has
> many of these.
> 
> >   4) The interpreter has a boot compiler built in
> > 
> >    There are complications to the interpreter to handle boot-
> >    related development from the command line which no-one is
> >    ever likely to do again. This code could disappear and, having
> >    gone away, simplify the interpreter
> 
> This still sounds a little vague, but in general if it was
> documented then perhaps someone who was working with BOOT would
> find this functionality useful and use it again.
> 
> > 
> > POINT 5: The boot language is undocumented 
> > 
> >    It is likely to remain so as one of the authors is dead 
> >    and the other one might be. There are no primary sources of
> >    documentation.
> 
> That does present a problem. But since there is still a lot
> of boot code in Axiom, figuring out how boot works and
> documenting it does not seem to me to be any more difficult
> than figuring out how the boot code works and re-writing it
> in lisp.
> 
> > 
> > POINT 6: Boot does not give you the full power of common lisp.
> > 
> >    Boot cannot handle backquote, defstruct, and other primitives
> >    not to mention CLOS, CLIM, etc. So as lisp evolves boot will
> >    "further fork" away from the living edge of lisp ideas and 
> >    limit the axiom programmer. 
> >
> 
> This is true, but I think that perhaps by accident or by design 
> it can be largely cast as an advantage. After all, the purpose
> of boot is to implement SPAD and the Axiom interpreter in a
> language that is higher level than common lisp - not to be a
> general purpose programming language.
>  

There may be a point here.  How difficult would it be to write spad
directly in lisp?  Are the restrictions of boot useful in implementing
spad, just as the applicative (i.e. functional) restrictions of acl2
assist that project?

> >    Witness the data structures used in boot, mostly lists.
> >    Frames are lists of lists, history is a circular list of
> >    lists, etc. Where are the data structures? Who will redesign,
> >    rewrite, and document boot extensions to fully cover common
> >    lisp?
> 
> This point was also recently raised by Juergen Weiss. I believe
> that his view is that it is possible to extend boot to include
> these data structures in boot syntax but that it might be
> preferable simply to use boot's built-in mechanism to "escape"
> to lisp where necessary to define these structures and then to
> call the data structure access methods in the usual way in boot.
> 
> > 
> > POINT 7: Boot loses program-data symmetry.
> > 
> >    This is a matter of religion but having been at this game for
> >    35 years I've seen a hundred languages built on lisp that try
> >    to "eliminate the parens" and all will eventually die (including
> >    Python).
> 
> I don't see that this has to do with parens or not. Using
> parenthesis to denote nested lists is just one more syntax.
> 

I think Tim means the grammar of the parens, not their lexigraphical
syntax (more below).

> > 
> >    The paren syntax is fundamental. Look at how Axiom really 
> >    works under the covers. It doesn't build boot functions, it
> >    builds lisp functions. Why? Because you can cons up a lisp
> >    function, compile it, and run it directly. Axiom code and
> >    data intermingle in the algebra lisp files. The databases
> >    have code and data in them. Data <==> Programs. Fundamental.
> 
> I don't agree that lisp has anything special to offer here
> except perhaps by being the first language to formalism this
> relationship. Any higher level language in which functions are
> first order objects has this property, although usually not in
> the way it is implemented in lisp. This is especially true in
> functional languages like Haskell and Ocaml and probably also
> in Aldor. "cons'ing up a lisp function" is just a less structured
> way of doing the same thing.
> 

This takes quite a while to appreciate for someone new to lisp, IMHO.
Sure, I can write a C program to generate and run other programs -- all
I have to do is write a printer to parse the logic of the program into
a text file, write (or use an external) parser and compiler to
generate the code, then write (or use an external) loader and
evaluator to give me my executable program withing my currently
running program.  I.e. I would at the simplest level do

f=fopen("foo.c","w");
print_c_program_from_struct_representation(f,funtion_representation_in_memory);
system("gcc  foo.c -o foo");
s=generate_command_line(args);
p=popen(s,"r");
read_result(p);
pclose(p);

Consider this vs.

fun:= a+b;
result=fun(x,y);
fun:= make_function("a","b");
result=fun(x,y);

where the functions are compiled, refer to live objects in the running
program, etc.  In lisp this is

(let* ((fun (compile nil '(lambda (a b) (+ a b))))
       (res (fun x y))
       (fun (compile nil (generate_function_lambda 'a 'b)))
       (res (fun x y))) ...)
 
> > POINT 8: Tools don't support boot
> > 
> >    Emacs balances lisp parens. It lets me do lisp function 
> >    lookups. It understands the lisp syntax including escape
> >    chars. SLIME lives in emacs and provides support. Code
> >    walkers walk lisp code. Pretty printers understand lisp.
> >    Debuggers understand lisp (I want to fix it in the language
> >    I broke it in). ASDF can manipulate lisp. Programmers 
> >    speak lisp. Blank spaces don't break lisp.
> > 
> >    Boot is unsupported anywhere by anything.
> 
> Kai Kaminiski has written a simple emacs mode for boot.
> See http://wiki.axiom-developer.org/AxiomInEmacs Boot is
> syntactically similar to Python so if there was a need for
> enhanced emacs support, then I expect a lot of this
> functionality could simply be "borrowed" from the Python
> extensions of emacs (and other syntax aware editors).
> 
> I think it remains to be seen how some of these other lisp
> tools could be used with Axiom.
> 

While I have done precious little for axiom (my apologies), there was
one bug I found and fixed early on.  I used lisp trace, debug, and
even gdb to find it.

> I don't know many people who "speak lisp" except perhaps
> during severe rapid eye movement while asleep. ;)
> 

Again, many more than boot I believe.  Still too few IMHO, but
relatively speaking lisp is much more widely used and known.

> > 
> > 
> > POINT 9: Where are the programmers?
> > 
> >    Who will write in boot? I'm the only person likely to be
> >    hacking the interpreter for the near future, mostly because
> >    it is so big, ugly, unstructured, and undocumented. Even if
> >    people code in boot to maintain the interpreter how often
> >    will they do that? How will they maintain their skill at
> >    writing boot? I've written many, many thousands of lines of
> >    lisp and use it continuously. Who are the programmers who
> >    will do that in boot?
> 
> I agree! That is the #1 question. But besides Tim Daly, where
> are the other lisp programmers? How will they maintain their
> skills at using lisp in Axiom?
> 

Here!

> Axiom is now open source and is at least potentially exposed
> to a much larger range of people than ever before. I believe
> that in the future there is a good chance that the people who
> may help to maintain and further develop Axiom will be those
> people who started by writing Axiom library code in SPAD.
> >From SPAD it is a natural step down the ladder to programming
> in boot. I expect that many of these people would never have
> had any previous exposure to lisp. So forcing them to program
> in lisp just to maintain Axiom would probably feel more like
> "falling off the ladder" than just stepping down.
> 

Well, witness maxima.  Lots more developer contributions on a project,
which while magnificent in its own right, has far less potential than
axiom IMHO.  And all of these people are writing in lisp.

> Of course this is currently mostly speculation (with the
> exception perhaps of myself and Martin Rubey).
> 
> > 
> > POINT 10: Boot was an experiment in surface syntax
> > 
> >    Historically Axiom was a platform for experimenting with 
> >    many research topics. One of these topics was a favourite
> >    of Dick Jenks and Bill Burge, that of language design.
> >    Scratchpad, now called Axiom, was changed many times to
> >    introduce new ideas. This research is no longer the point
> >    of Axiom and is better done elsewhere. We neither have the 
> >    expertise nor the research goal of surface language design.
> >    Let Python do that. Or Spad/Aldor fans.
> 
> I think the Python designers also (for the most part) consider
> their work done.
> 

???  I guess its a question of time horizon.  Lisp was standardized in
1994.  Changes in perl or python et. al. will always be possible at
the discretion of their inventors, to my limited understanding.  I've
upgraded Debian Linux from the days of 'rex', and believe me you don't
want to know what a perl incompatible upgrade can do to critical
system programs written therein.  C and lisp are no longer moving
targets -- some might consider this a disadvantage, but I find it
indispensable. 

> I thought Spad/Aldor was an essential part of Axiom and
> necessarily closely related to the Axiom interpreter, so I
> am not sure what you mean by your last comment.
> 
> > 
> > POINT 11: Do language design where it counts.
> > 
> >    If we are going to devote research effort toward language
> >    design lets do it at the Spad/Aldor level where it counts.
> >    This is where the real language expertise should be applied.
> 
> I agree. I don't see any serious need to re-design the boot
> language - simply to document it as it works now.
> 
> >    For instance, why do I have to write programs to express
> >    functions?
> 
> In Axiom you can already expression functions in other ways
> than imperative code. The Axiom interpreter, SPAD and Aldor
> have a (more or less) complete functional subset.
> 
> >    Why can't I write a domain using equations? Why doesn't
> >    the Spad language support that?
> 
> Could you elaborate on what you mean by "write a domain using
> equations"? Are you talking about an alternative to the way
> that Axiom implements "axioms" now simply by assertion?
> 
> > 
> > POINT 12: Future directions assume lisp
> > 
> >    Notice that one of the near future goals is to connect Axiom
> >    and ACL2. ACL2 understands lisp-ish kind of languages, not
> >    boot.
> 
> Besides the fact that ACL2 is implemented in lisp, I don't see the
> connection between ACL2 and "lisp-ish" languages when it comes
> to proving program correctness.
> 
> There are other tools similar to ACL2 such as Coq which is not
> written in lisp at all. Coq has already been used with Axiom
> in some research.
> 

Bill has a point IMHO that modern OSes provide a remarkable amount of
generic glue between differing programs via shell pipelines, php/cgi
scripting, etc.  But it will never be the same as integrating programs
in core memory for anyone wanting to push the limits of what their
system can do.  Lisp makes this easy, and having acl2 in the same
image opens up numerous possibilities.  Should we wish to integrate
the theorem prover into the spad compiler, for example, do we really
want to output Coq input through a shell pipe and parse the results?

> > 
> > 
> > and now, on to the recent changes
> > 
> > POINT 13: Don't break code until you understand it.
> > 
> >    I'm working hard to understand the code so the existing code
> >    is a straightforward translation (well, mostly, with some
> >    rewrites that I know are logically equivalent).
> > 
> >    Once I've completely figured out the various twists and turns,
> >    uses and references, and other points I'll be recoding the
> >    whole process. There is no reason (other than historical) why
> >    there are a dozen entry points into the interpreter that set
> >    weird, undocumented, stateful flags. All that cruft must go
> >    away.
> 
> I agree with all that, while de-emphasizing the need to remove
> all of the cruft right now. But I fail to see why recoding
> anything from boot into lisp is involved.
> 
> > 
> >    So the code is ugly. At the moment. But it will get better,
> >    cleaner, clearer, more robust, and easier to change. History
> >    and Frames, for example will use structures that don't depend
> >    on 
> >       (caar (cdddddr (cdddddr x)))
> > 
> >    It takes time. It takes work, probably 2-3 years worth of
> >    work.
> 
> I think you worry too much about how "ugly" to current code is.
> It works. That is what matters most to people who want to begin
> using Axiom now. I doubt that there is much you can do to the
> Axiom internal code in the way of re-writing it into lisp that
> would make it any more appealing to this putative new generation
> of Axiom users. Still I am hopeful that among the thousands of
> new people who have recently been exposed to Axiom, there will
> be some who will want to take up the challenge of dealing with
> what's on the inside. If/when they arrive, I think it is more
> important to have good documentation than it is to have pretty
> code.

I agree with this.  One could easily take the position that we have
more important fish to fry, and what we have works.  But if we are
considering a code simplification, I feel using lisp is clearly the
best choice from where we are now.

> 
> >   
> > and now, on to the personal...
> > 
> > POINT 14: I'm a lisp programmer 
> >    
> >    (clearly)
> > 
> 
> I don't seen anything inherently evil in that. :)
> 

Take care,

> Cheers,
> Bill Page.
> 
> 
> 
> 
> _______________________________________________
> Axiom-developer mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/axiom-developer
> 
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah