[Chicken-users] Suggestion for new egg: Wings!

[Alaric Snell-Pym]

Ok. Those of you who lurk in #chicken on irc.freenode.net will
probably be familiar with how we all like to main about Ruby on Rails.

As it happens, before I returned to Scheme from my long hiatus out in
the horrible world of commercial software development (eg, Java, PHP,
Perl and C++) I was finding the world of web application development
rather impoverished of tools; in that whenever I wrote a web app, I
was always spending far more time on stuff I felt should be automated
- or, conversely, having to fight some toolkit that promises to
automate everything, but then makes it hard to do things outside of
its limited view of the world.

Being the kind of person who's fond of thinking of designs for
things, I naturally began tinkering with ideas in my head about how
I'd fix this, when I happened to come across Paul Graham's writings
about Lisp, take a look at the current state of Scheme
implementations again, found things muchly advanced from when I last
learnt languages for run rather than because somebody was offering me
money to code in the thing, came across Chicken, joined #chicken, and
met Peter Bex who (a) grumbled about Ruby on Rails even more than me
and (b) maintains Spiffy.

So we got chatting, and I quickly began to adapt my sketched ideas
for a web app framework to make use of Scheme's strengths. And the
name Chicken Wings was suggested, rather cleverly implying freedom
and flight rather than being stuck on a one-dimensional set of rails!

The conclusion I came to is this:

1) A web app could consist of a .scm file that loads spiffy and third-
party extensions, then proceeds to load or define application-
specific business logic procedures, thus leading to a nice global
environment full of tools, and registering resources of a more
dynamic nature in a hash table.

The framework defines a parameter, wings-environment, which is bound
to a closure that accepts a key and returns the value associated with
that key in the global hash table, or nil otherwise.

The reason for this will become clear in a moment.

2) The web app finally, having set up its environment, starts spiffy
serving files out of a document root directory.

3) These files contain the frontend of the application. Static files
(CSS, HTML, images, etc), as well as files that spiffy knows to
handle specially. All of the existing spiffy handlers will fit into
the Wings model OK, but there's a few more I have in mind. More on
those later.

4) There are web app frameworks that are "continuation based" or
"widget based" or various other paradigms. Each of these paradigms is
nice for some things, but sucks for others. Continuation systems are
great for things with a very sequential page flow, like multi-page
signup processes and complex operations. Widget systems are great
because you can bind the widgets to your data sources easily.
Frameworks like Rails are based around an object-relational mapping
to SQL databases, and if you use that supplied data store, it
automates a load of stuff to do with generating pages from data in
SQL or forms that edit same; but if you want to edit data stored in
other kinds of database, you have to do much more work yourself. Etc.

In general, I find existing frameworks too monolithic.

Therefore, with Wings, you just use Spiffy's map-URLs-to-files-and-
decide-what-to-do-with-them logic. You can have as many different
file type handlers as you like.

One could write a handler that implements continuations; when a
request for the page comes in, the handler looks for a continuation
ID in the request. If there isn't one, it starts a new 'session' from
the defined entry point, otherwise resumes a saved continuation. Then
one could use continuations for parts of the app that benefit from
it, while not using them elsewhere.

One could write a handler that provides a quick and dirty admin
interface to your SQL data. The actual file on disk just contains
metadata about how the tables link together and how fields should be
interpreted and so on. The handler turns this into an admin
interface, like Django provides.

One could write a handler that takes a page description that uses
smart widgets, and handle HTML form state in a snazzy way, with
automatic loading of widget state from a data source at the start,
and saving of it back when the form is submitted.

This model also means that you can allow front-end web developers
free reign over the document root, so they can create new pages at
will, while the back-end guys get free reign over the
application .scm file and the units it loads. And the two groups
share a nice Wiki that documents the data sources and Scheme
procedures the back-end guys provide for the front-end guys. Whereas
with Ruby on Rails, to create a page at all, you need somebody to
write an action method in the controller class. Which either means a
front-end guy bothering a back-end guy to do it, or the front-end
guys messing around in a file full of back-end code, or not having
any specialisation between front-end guys and back-end guys. Which
sucks for me, since I'm definitely a back-end guy. Gimme business
logic algorithms over Javascript and HTML form handling any day.

As a case in point, when I worked at UpMyStreet.com (which was
written in PHP at the time), I (as the resident lead back-end guy)
was always being bothered when the front-end guy had to make
relatively minor changes to what went where on pages, and changing
the page structure, and so on. So we documented all the functions
that extracted data from the database, and taught the HTML guys
enough basic PHP to call a function and output the result with nice
formatting, and let them create their own PHP pages (and rearrange
them, and split them into two separate pages, and whatever, as they
saw fit). They'd just ask us when they needed some new formatting
function or some new angle on the data, that would require us to
write and document a new function for them. This worked really well.

Also, I want to decouple the data sources. I'm designing a basic
interface for a data source to be registered in the environment so
that business logic functions and page templates can use it without
needing to know if it's an SQL database, an XML-RPC web service, or
just a function that maps inputs to outputs.

As a taster for the feel of the thing, a read-only data source called
"foo" would be made available by a mapping in the environment from
(get foo) to a function that accepts whatever the primary key of
"foo" is, and either raises a "not found" condition or returns three
values: a result, a last-modified date for use in intelligent
handling of HTTP caches (or nil if there's no global state being
referenced), and a cache validity period in seconds. The cache
information can be used to cache the data source, or all data sources
referenced in a file (plus the filesystem modification timestamp on
the file itself) can be aggregrated together to generate HTTP cache
control headers.

I'm also defining interfaces for create, update, and delete operations.

However, what the skeleton of Wings provides is a way for these
different handlers to interoprate pleasantly, bringing the benefits
of a monolithic application framework, built from loosely-coupled
independent components.

The first component I intend to write is a way to declare the GET
arguments to a page and have them automatically sanity-checked and
decoded.

For use in page template languages that let you directly embed
Scheme, it will work as a macro that reads the page metadata file
(located in a file with the filename of the page, plus ".wings") and
extracts the arguments declaration, then expands into a let that
binds Scheme variables to the processed values of the arguments.

An arguments declaration might look like this:

(positional
  ((user-id integer))
 named
  ((comment-id "c" optional integer)
   (search-terms "s" optional string))
 data-sources
  ((user userdb (user-id))
   (comment comments optional (comment-id))))

What this means is:

* If there is a path component AFTER the script name - eg, "1" in
http://www.example.com/test.ws/1 - then it's passed through string-
>integer and bound to "user-id". If it's not present, we generate a
404 Not Found. If it's present but not a valid integer, then the page
just returns with a 404 Not Found.

* If there is a GET variable called "c" then it's likewise processed
as an integer and bound to comment-id. Otherwise, comment-id is bound
to nil.

* If there is a GET variable called "s" then it's not processed
(anything is a valid string), but is bound to search-terms. Or search-
terms is bound to nil if there's no such variable.

* The "userdb" data source is accessed by looking for a key in the
wings environment called (get userdb), which should be a closure,
which is applied to the value of user-id. If it throws a not found
exception, then we return a 404 response, otherwise we bind the
result to 'user'.

* The 'comments' data source is accessed by looking for (get
comments) in the wings environment, and applying the result to
comment-id. If it throws a not found exception, it's caught and
comment bound to nil (because of the 'optional'). Otherwise, the
result is bound to comment.

Also, the last-modified timestamps and expiry times of all the data
source lookups are examined, and the most recent last-modifed and
shortest expiry time kept. The last-modified time of the file itself
is also examined, and extra last-modified times and expiry intervals
can be supplied to the macro by the script, to produce a final most
recent last-modified and smallest expiry interval.

The macro can handle conditional HTTP GETs with "If-Modified-Since"
headers, if the IMS sent by the client is less recent than the
computed last modification timestamp, by just returning a "use what
you've got" HTTP response and never processing the body of the macro.
And it can output cache control headers based upon what it's computed.

What does this get you?

1) Automatic handling of HTTP caching
2) In your code, you have nicely bound variables like "user" that are
already type-checked and converted, including referencing IDs to
actual data objects from data sources where required, with proper
handling of invalid IDs.
3) You get to use short names in your URL query string: c=123&s=search
+terms.
4) The names used to refer to arguments in your code are decoupled
from how those arguments are represented. You can make an argument
positional (part of the path) or named (part of the query string),
and change your mind later, without needing to change your code.

Also, we can provide a utility function that, given the site-root-
relative path to a file and an alist of argument names and values,
returns a URL:

(make-url "/test.ws" '((user-id . 123) (comment-id . 456)))

This would read /test.ws.wings to find the declaration, and thus return:

"/test.ws/123?c=456"

Again, if you change the way you encode your page arguments by
altering the declaration, then the behaviour of this function will
change to match the behaviour of the argument parser, so nowhere else
in your code do you need to change anything :-)

[needless to say, we'll cache the .wings files rather then rereading
them ALL THE TIME!]

But it gets better. The page argument handler knows of argument types
like "integer", "boolean", "float", "rational", "symbol" (with an
optional list of allowed values), and "string" - but also compound
types such as "list", "alist", "set", and "hash".

These would only be valid for query string named parameters. If you
declare a named argument like so:

  (foo "x" (list integer))

...the system will look for a query string like "x[0]=123&x[1]=456",
and then bind '(123 456) to foo.

If you declare:

  (foo "x" (alist integer))

...it will look for "x[wibble]=123&x[wobble]=456" and then bind
'((wibble . 123) (wobble . 456)) to foo.

If you declare:

  (foo "x" (set integer))

...it will look for "x=123&x=456" and then bind '(123 456) or '(456
123) to foo.

If you declare:

  (foo "x" (hash symbol integer))

...it will look for "x[wibble]=123&x[wobble]=456" and then bind a
hash table mapping 'wibble to 123 and 'wobble to 456 to foo.

And, of course, the make-url function will accept list, alist, or
hash table values for foo, and generate appropriate query string
contents.

Oh, and for file type handlers that don't work by running Scheme code
in some guise, I also plan to provide a function (which, I hope, the
page argument parser macro will use as a helper function) that
returns the argument bindings for this page as an alist, which can
then be incorporated into an interpreted environment of some kind for
access.

I propose to create an egg called 'wings' to contain general utility
functions for access .wings metadata files (there'll be other uses
for the files in future, so I plan to create a generic framework),
the definition of the wings-environment parameter and utility
procedures for accessing same, utility and support functions for the
data source API using the wings environment, then the aforementioned
page argument handling tools.

Followed closely by an egg called 'wings-ds-sql' that provides a
basic SQL data source generator. Data sources can be generated using
it by providing SQL query templates for SELECT, INSERT, UPDATE, and
DELETE queries, or for the simple case where the data source is a
single table, just a declaration of the columns and how they are to
be handled, for all the queries to be generated automatically.

After which, I plan an egg called 'war-rocket-ajax', in which I will
present a Spiffy file type handler that implements a full version of
a widget-based page templating system I prototyped in PHP some time ago:

http://www.snell-pym.org.uk/archives/2007/06/17/another-thing-i-hate-
about-web-application-frameworks/
http://www.snell-pym.org.uk/archives/2006/12/17/the-implementation-of-
web-applications/

The neat thing about War Rocket Ajax, and the thing that gives it its
name, is that it's designed to provide snazzy Ajaxy functionality
while providing seamless graceful degradation for clients that don't
or won't support it, with minimal extra programmer effort.

Hopefully, around then, other people will start contributing better
stuff, too ;-)

My hope is that the framework is loosely coupled enough to allow
pleasant interoperation while not constraining. The make-url
procedure defined above will gladly generate URLs that start
continuation-based sessions, that just spit static content out, that
go into complex widget-based editing forms, or that just fetch some
data from a database and slip it into an SXML template and render it,
or whatever. And the data source API should allow pages to use data
sources ranging from pure functions to global variables or any kind
of external state from the filesystem to database clusters. It allows
for purely create-only data sources, such as an SMTP client, read-
only data sources such as the current date and time or a temperature
sensor, and so on. The basic functional interface to data sources
does not constrain them to deal with 'record'-like objects as SQL
database do, and so on.

So, to conclude, for now, I request that a nice new egg directory be
created in SVN, called 'wings', with write access from my SVN account!

ABS

--
Alaric Snell-Pym
Work: http://www.snell-systems.co.uk/
Play: http://www.snell-pym.org.uk/alaric/
Blog: http://www.snell-pym.org.uk/?author=4