Writing a lexer...

[emacstheviking]

Hi,

I am trying to write a lexer in Prolog for educating myself but
despite understanding difference lists, DCG-s etc to the point where I
can do little examples, try as I might I cannot seem to make any
headway into my problem space as it were.

Last year I released (yet another) computer language into the world to
great acclaim (i.e. nobody knows about it except me and you) and
whilst I still use it "at home" for writing PHP and Javascript stuff,
I decided that redoing it in Haskell was too easy (Parsec!!) so Prolog
was my next choice as I like learning things. Constantly, like how
little I know!!

I have two main problems.

1) How to get the contents of a file file "into memory" so that I can
scan for tokens etc.
2) How to scan for tokens!

I thought that the first step would be to read the entire file into
memory as a character code list or something and start from there.

I have the concept in my head that the prolog code would be not too
far from the BNF for an s-expression, with DCG with extra parameters
being the way to go.

Putting all that to one side then, and starting from the beginning
(nearly always a good place to start), I want to know how I might go
about my task, converting a text file with someting like this:

    (defun test ()
        (echo "Hello World"))

into something like this, a list of tokens that could then become an
AST structure as the next step:

  [ popen, atom(defun), atom(test), popen, pclose, popen, atom(echo),
strdq("Hello World"), pclose, pclose ]

Sometimes I think Prolog would make it easer to build the AST at the
same time, like I have seen done in some examples in Art of Prolog...
I will of course be furiously reading and trying things myself too!

Thanks everybody,
Sean.

PS: http://feltweb.info is where my language thing is at, still
working on Java production though...