chicken-users
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Chicken-users] a file system using Chicken


From: Joerg F. Wittenberger
Subject: Re: [Chicken-users] a file system using Chicken
Date: Tue, 12 Jun 2007 12:14:52 +0200 (CEST)

> "felix winkelmann" <address@hidden> writes:
> 
> >>
> >> A few of you have mentioned their past interest in using Scheme
> >> for a filesystem. Out of curiosity, what ideas did you guys have
> >> in mind?

I must have missed that post.  This is, when I *must* put in my
shameless plug about having worked towards such a thing in the past
years.

> Me too! ;-)
> 
> - developing a Scheme-based domain-specific language for implementing
>   secure/safe/guaranteed-integrity filesystems, using techniques like
>   the various secure proof-carrying calculi

I did end up with something like
http://softeyes.net/A04538159df3258ea68544531bea1c006 (white paper
from 2002, PDF), the implementations looks roughly like this:

level 1: comparable to git (see "man git" section "the object data
base"): content addressed objects.  Askemos has only two types "blob"
and "place".  Blob, as in git is a static byte sequence, addresses by
it's sha256 has.  Blobs as well as places are referenced from other
places via links.  Everythink place linked from the well know place
(where - by "accident" higher level will store your current X509
private host key is referenced as blob) is kept during garbage
collection (of places).

level 2: For each place the system kernel maintains a set of
properties.  What exactly is already configurable.  However a certain
set of properties is provided.

  1.  The object type.  Called either "action-document" or "contract"
      in the context of Askemos.  (Where the former is better to
      explain the internals of the machinery, while the latter names
      the real purpose.)  Objects types are "consed" with initial data
      at object creation time.  They are static objects.

  2.  Mandatory capabilities.  (Here I need to point out that the
      scientific contribution of the Askemos project is to solve the
      distribution of capabilities "how do the capabilities get
      granted to the objects (places)" in a secure manner. [boils down
      to a few lines of set theory])

  3.  Creator, Date and some such meta data.

  4.  the actual (optional) object data (a blob).

level 3: Places communicate to each other sending messages (Erlang
style interaction).  The "contract", which governs the execution of
the process whose state is stored (in the blob and link pattern) at
the place defines via some embedded Scheme code a few operations.  At
least: "Read" - how the repls to idempotent calles to the place are
composed, and a "propose"/"accept" pair, which defines how messages
are handled, which might modify the place.  Recall MVC and the
GET/POST distinction of HTTP and some imagination, which one goes
where.  We return to the "propose/accept" instead of naive "write" in
a second.

> - developing file-systems optimized for reading and writing particular
>   types of data structures, with the ability to add new data structure
>   definitions on the fly, essentially a hybrid between file systems
>   and databases
> 
> - union directories ala Plan 9 -- combining different resources in a
>   filesystem interface; the resources could be different Scheme
>   programs and so on -- I guess this is a generalization of Felix's
>   ideas #1 and #3

By means of the contract, all types of objects (e.g., comparable to
those "tree" and "commit" records of git, but that's now for git
comparision) can be defined.  Use your OO knowledge to roll your own
MOP.

(((We have slightly optimised already for XML processing.  Parsed
state / serialized octec sequence cached side by side.  Also sqlite
available because I could not resist.)))

>From level three there are a few network protocols available.  Most
important HTTP/S to communicate with the places within Askemos from
outside.  I've been looking into fuse a few times myself and I am very
grateful for duggfs in that respect.  ;-) But for file system
operations we currently rely on WebDAV no matter how bad a standard
that one is.  At least the destop tools talk it and: it will not leave
the local machine anyway (most of the time).  See below.

The brings us to

level 4:

There is second well know object with the public host key and some
initial data intented for use as initial contract and initial process.
Since contracts are static places, but places nevertheless, their type
must be a contract.  The recursion ends where the legal recursion
regarding contracts ends in the real world to in a initial contract,
the "constitution".

Using their public key as identity, hosts running an Askemos
representative (my name for the server/proxy mix) may route requests
among each other.  (Hell strict rules regarding capabilities.)

The locations (representatives) where a place resides is a property of
the place and under control of the contract as yet another property.
So applications can choose where to run (given enought of those
choosen representative is willing to support the application).

This takes us back to the "propose"/"accept" pair above.  The last
paragraph should look slightly strange at first because of
location*s*: Places may choose any quorum out of the representatives
it can locate to run at.  In case of just one, this boils down to
parallel the design of Termite, Erlang etc.  Normal case however is
that the "write" type messages are replicated to all available copies
of the place (at those locations above) splitting the operation into a
"propose" part with all side effect held back and a "accept" part,
where side effects are applied to the data store.  A byzantine
agreement is beeing run with the result (checksum) of the "propose".

(((If you came from group communication systems like spread, such a
transaction is where you see a membership messages and need to run the
a application specific resynchronisation protocol or need to commit
your application state permsistent and run the resync even though no
membership changes.  With Askemos the resync procotol is hidden in
message passing of aplication messages.)))

A the applications level all these confusing things are hidden in the
transaction engine.  The first WebDAV prototype has been written
completely in XSLT.  (For performance sake now rewritten in Scheme and
compile with the kernel to C.)  No need to care for threads, locks
etc.

The whole thing is stable enough to run real dynamic web sites.  For
instance the Wiki, which does most of www.askemos.org but wrote more
applications already.  Currently we run about 95% percent of
transactions successful on the development network (including down
time due to bugs).  Not too bad, me thinks, given that the underlying
telco reports 98% connections beeing switched through the internet.
On everage 0.3 seconds per transaction (which has been faster before
and we struggle to locate the bottle neck).  A few days back we lost a
machine totally.  All what had to be done to recover was: compile at
the new machine, create a X509 certificate for the id of the lost
machine (if we had hada backup, we could have used the old
certificate), re-create the user list allowed to log into the machine
(from the backup), start and wait while for resynchronisation to
complete.

Given that the normal mode is to run a representative as your local
proxy to the network, HTTP and thus WebDAV is only used at the
loopback interface.  Let Moores law account for the efficiency.  ;-)
AJAX becomes bleedingly fast.  Even JavaScript free form interfaces
feel almost like AJAX, since whole page transfers are no longer that
bad.  (Except for the focus gets lost, which is annoying.)

The whole code is mostly R5RS and did run on chicken before.  manpower
was the only reason to leave it lag behind the RScheme version.  The
plan is however, still to port it to chicken, which means mostly
wrapping some C calls and write the build process.

I just wanted to tell you, before you start from scratch.

BTW: I'm mostly proud of a german lawyers expertise, (sponsored by
Prof. Niemitz, Leipzig) which finds, that contracts singed/stored in
Askemos networks should stand as "visible" evidence in court.
[provided the quorum of representatives are *not* under control of a
single legal or natural person] In contrast to normal cryptographic
signatures, which only stand as far as some law pulls them to
"probably a valid signature" level, which can be doubted and thus may
not count as evidence.  (Sorry, I the legalize I can't really
translate to english.  Please resort to the german version, I send out
in private mail at request; it may be different elsewhere anyway.)

best regards

/Joerg

PS: don't use the current version. Use 0.8.4 We tried to improve the
code, but that broke the status code in the message routing of 4:3
communication (for copies of the sender sending to a quorum of 3
receivers), in the (standard) case where the set difference is the
local proxy.

> "felix winkelmann" <address@hidden> writes:
> 
> >>
> >> A few of you have mentioned their past interest in using Scheme for
> >> a
> >> filesystem. Out of curiosity, what ideas did you guys have in mind?
> >>
> >
> > Endless things. Like,

Endless things, indeed.  ;-)

> >
> > - accessing a running scheme program over the filesystem (read
> > - globals,
> >  call functions/evaluate code)
> > - Scheme-level nfs, so to speak - network interaction over the
> > - filesystem,
> >  tuplespaces, blablabla...
> > - Using the file-system to interact with Scheme processes would open
> > - up
> >  completely new ways for shell programming
> >
> > Well, I don't know! But I want it! WANT IT WANT IT WANT IT!!!
> >




reply via email to

[Prev in Thread] Current Thread [Next in Thread]