Re: [Axiom-developer] Literate programming

[daly]

On Thu, 2011-10-27 at 00:17 -0700, Mark Engelberg wrote:
> Tim,
> 
> I recall that at some point you described your setup for doing Clojure
> literate programming, and if I recall correctly, you were primarily
> working in LaTeX, relying on incremental compilation to test little
> snippets of code as you wrote them.

Yes, the idea is to write a literate version of Clojure, see
http://daly.literatesoftware.com/clojure.pamphlet
http://daly.literatesoftware.com/clojure.pdf
similar in style to "Lisp in Small Pieces". (The effort has been
stalled temporarily while I try to find new employment.)

>From the above document you extract the Makefile, type 'make'
and end up with the newly formatted PDF and a running REPL.
So any edits to the document text are immediately reflected in
the PDF and any edits to the code are immediately reflected in 
the compiled results.

> 
> Have there been any new developments on the literate programming /
> Clojure front, in terms of tools that leverage existing build tools,
> test suites, generating meaningful line numbers for stack traces and
> debugging?
> 

The question is ill-posed in sense that literate programming is not
a tool or technology but a change in mindset. To quote Knuth:

  I believe that the time is ripe for significantly better documentation
  of programs, and that we can best achieve this by considering programs
  to be works of literature. Hence, my title "Literate Programming".

  Let us change our traditional attitude to the construction of 
  programs: Instead of imagining that our main task is to instruct
  a computer what to do, let us concentrate rather on explaining to
  human beings what we want a computer to do.
    -- Donald Knuth "Literate Programming (1984) CSLI 1992 pg. 99

So imagine a world where the eloquence of Rich Hickey was expressed in
book form. Imagine the whole immutability lecture written down and 
decorated with the actual running code as illustration. Imagine that
the original authors clearly explained software transactional memory
and illustrated it with the actual code. Imagine a discussion of
argument destructuring with running code illustrations.

How many more people would be able to dive into the details of Clojure
to maintain and modify the code? How many people would find it much
easier to understand prototypes, defrecords, macros, and all of the
other struggles that populate the current email forums? How are
infinite sequences supported and what code supports it? What does
NIL mean and why was it defined that way?

We do not capture the ideas. We do not rewrite the explanations to
clarify the fine points of confusion. We do not illustrate how the
ideas of Red-Black tries are moved "from ideas to implementation".
We do not communicate the ideas to each other. We code for the machine.

So we end up with the traditional "tree of sand" pile of little files.
In order to overcome this we construct tools (IDEs) that know how to
navigate this pile, tools to store the changes (SVN, git, hg), tools
to put the sandpiles together (#include), tools to build guardrails,
tools, tools, tools... (You can, of course, keep each chapter and
section and subsection in separate files and "include" them in the
book.)

Now imagine that the "book" is multimedia where you can include
animation (watch the tree balance), video, and a running REPL.
Think Khan Academy for Clojure.

This community is willing to overthrow the traditional ideas of lisp
in order to make progress on a new path. Yet we continue to structure
the development as though we worked on a PDP 11/40 with 4k file size
limits. We continue to let the most valuable information which is the
meat of Clojure disappear while we keep only the code, the bones of
Clojure.

Open the source code. Stare at it. Ask yourself if you understand
exactly why it was needed, why it is structured that way, what would
happen if you changed it and what else depends on this code. Imagine
your job is to maintain and modify it but Rich is not available for
questions and answers.

Ultimately that is what matters. In the long term the code will be
the only remaining artifact after Rich leaves the project. Look at
Sourceforge and you will see thousands of dead projects that will
never be picked up because they are just trees of code, dead code.
Et tu, Clojure?

Literate programming is about making code live. 
I like Clojure and I really want it to live.

Think long term. Imagine a better way.

Tim Daly