You will want to look at tree_walker in the octave/src directory, which
allows you to walk the parse tree. It is used in pt-check, pt-pr-code
and pt-bp. It walks the parse tree for a function, visiting each node
in turn. You need to write the action you want for each node type.
As a first step I would like Octave to spit out a byte code for the likes
of Parrot or some other virtual machine. Parrot in particular because
it is intended to be used by scripting languages, but Java byte code or
.NET CLR would also be interesting.
All of these machines have JITs available for compiling the byte code.
A couple of years ago I did an experiment in which I translated the
an m-file version of cumsum to C++ by hand:
y = zeros(size(x));
total = 0;
for i=1:length(x)
total = total + x(i);
y(i) = total;
endfor
IIRC, I got a 6x speedup from a direct translation using octave_value
for x,y,i and total. As optimized C, this loop was about 5000x faster.
In Python, this loop was 100x faster.