Re: Emacs rewrite in a maintainable language

[David Kastrup]

Marcin Borkowski <address@hidden> writes:

> For the record: there was once a project to rewrite TeX in Java (it
> was before the mankind learned that Java was not the solution to all
> the humanity's problems).

NTS -- New Typesetting System.

> It even got some funding (as in, real money).  The project is now
> considered a failure: while they /did/ produce a working, 99.9%
> compatible TeX replacement, it was an order of magnitude slower (or
> so), and the main goal - to have a TeX replacement written in a more
> modular fashion, so that it would be easier to e.g. experiment with
> various line/page-breaking algorithms and extensions to TeX - was not
> achieved: nobody actually performed those experiments.  (There aren't
> even many traces on the internet; while you can read about this "New
> Typesetting System" in a few places, the actual code is lost.  That is
> a pity, imho, since I hear that Java is much faster nowadays than 10+
> years ago, so it might work now.)

Well, later there was exTeX <URL:http://www.extex.org>.  The front page
of the web site and the FAQ are still accessible, but most parts of the
site are dead these days.

> Another project, LuaTeX, took off and can be now considered very
> successful.  This project, however, has different goals (instead of
> replacing TeX with something different, they embedded Lua in TeX, and
> then opend up a possibility to reimplement many parts of TeX in Lua.
> In effect, it seems that the goal of NTS - to be able to easily
> experiment with TeX internals - was achieved, though in a completely
> different way).

It did not as much clone TeX as slowly turn it into a cyborg, putting
hooks into the existing works.

> Also, it is run by a small group of dedicated individuals, and my
> guess is that one of the reasons of its success is Hans Hagen and his
> team.  One, they are exceptionally smart people.  Two, they /think/
> and /design/ before coding, not the other way round.

Uh, are we talking about the same project here?  The whole point of
LuaTeX was _not_ to design anything but rather take the existing design
and code and "open it up" by adding hooks into its code and data
structures.  This most certainly _is_ a "code foremost, to heck with
design" project.

> (Not to say that Emacs devs don't do it, but the majority of the
> software world apparently got this wrong.)  Last but not least, Hans'
> company is based around ConTeXt and LuaTeX, so he has a /really/
> strong motivation to have this system working smoothly: he /actually
> relies on it for his income/.

He has a really strong motivation to have _something_ working _today_.
And he started with an existing complex format employing the unchanged
TeX engine, hooking into TeX where the limited facilities provided by
the base TeX engine were making the format rather awkward and
inefficient even though basically "everything needed" would have been
reasonably close under the hood in the underlying Pascal code.

I once translated a video game I wrote for Z80 processors at 4MHz
(obviously in assembly language) to C.  That was the opportunity to
rewrite everything in a modular manner, make the various "monsters" have
movement code in an object-oriented manner and so on.

In the end, this ended up more or less as a literal translation of the
logic of the assembly code, partly using even register assignments as
variables.  The game was coded very well and readable in spite of
6-character limits for identifiers (uppercase and digits only) and
almost all comments consisting in accumulative cycle counts for the sake
of the game's timing.  To "rewrite" in more modern idioms would have
required basically designing the code from scratch, and the existing
code was a very good "local minimum" with nothing in the line of
"incremental improvement" being in line of sight.

The same problem pervades in a larger scale the original code base of
TeX.  You either end up with redesigning it from scratch, or with just
having a clone not inherently different, in particular not any more
modular or maintainable.

LuaTeX does not even _try_ to touch the TeX "table of equivalence" and
its related dynamic scoping mechanisms.  From Lua, you just get to touch
the current group-local settings of TeX variables but do not get to
either read, simulate, or touch the TeX grouping itself.

There still isn't any way to embed Lua code in TeX or vice versa: any
Lua you want to execute from TeX is first _tokenized_ according to TeX's
input conversion rules and then _printed_ back into characters read into
Lua.  If Lua was a language picky about differences in whitespace (like
Python), this would be totally untenable.  The reverse is similar: Lua
can "print" into TeX's idea of the current input file, and TeX will at
some later time when Lua has ceded control pick that input up and
interpret it as part of whatever it was doing when it called Lua, of
course first turning the characters back into TeX tokens according to
whatever catcode regime it might be in.

That's not an integrated application as much as it is two different
systems conversing over byte-stream pipes with synchronous but
non-symmetric flow control.

They get results as opposed to the several failed projects preceding
them.  But the main reason is that they don't bother with a coherent
design rather than getting results.  In the course of their work, they
sometimes throw out some component of TeX altogether and replace its
Pascal code with something coded entirely in Lua, in order to make it
more accessible to their personal modifications.  So there _is_ some
"rewrite TeX in Lua" work involved, but it's not really part of a design
process but done ad-hoc on an as-needed basis.

So the overall message LuaTeX sends to people wanting to redesign
systems is rather bleak.

-- 
David Kastrup