Re: [Chicken-users] Async IO (was Re: libcurl?)

[F. Wittenberger]

Am Mittwoch, den 05.10.2005, 09:29 +0000 schrieb Thomas Chust:
> Am 05.10.2005, 03:41 Uhr, schrieb Peter Keller <address@hidden>:
> 
> > On Tue, Oct 04, 2005 at 10:21:55PM -0000, Thomas Chust wrote:
> >> CHICKEN does not have any feature that would make continuations
> >> serializable; it would mean that you somehow have to transform machine
> >> pointers to arbitrary memory locations where your continuations'  
> >> assembler
> >> code lives into stuff that can portably be transferred over a network
> >> connection.
> >
> > I'm sure there are some ideas that could help you perform a virtual
> > serialization of the continuations. For example, machine A has the
> > "original" copy of the running executable, machine B has a certain form
> > of a skeleton executable running in memory with a very small memory foot
> > print. Now, the continuation (wholesale) is transferred to the target
> > machine where it is propped up in the some executable pages in the same
> > range as before and control jumps there.  Of course, the rest of the
> > memory heap (known by the skeleton since it conspired with machine A)
> > is protected via mprotect, and when the first segfault happens, it is
> > caught, and the offending pages brought in from machine A to machine
> > B. Then the continuation happily continues. After a while, a working
> > set emerges, and machine B's heap gets managed independantly of machine  
> > A.
> >
> > Machine A knows what pages got served, so it is aware of such things
> > during garbage collection, and when machine B's pages are finished,
> > they are merged back into Machine A's domain. I'm handwaving the merge
> > process, since I haven't thought about this more than 5 minutes. Though I
> > suspect the merging algorithm can move stuff out of the pages of machine
> > B to better location on machine A since ultimately you're dealing with
> > higher level objects and the scheme implementation isn't going to care.
> >
> > I'm sure there are a lot more gotchas in a method like this (like race
> > condition between garbage collection on machine A and machine B's request
> > for the pages), especially without deep thinking about it, but it might  
> > be
> > workable.

This sound simillar to the way the RScheme persistant store works.

Just that rscheme doesn't handle persistant *continuations* without
additional work.

> your scheme for doing this is fine and basically the same thing as Kali  
> Scheme originally did. But Kali Scheme was running on a bytecode  
> interpreter, CHICKEN is running directly on the processor, so there are  
> several subtle issues involved with taking binary code from one machine  
> and sending it to another to execute. Some important points are:
>    * you may not be able to map everything to identical addresses on the  
> target machine for various reasons, so you have to compile the full  
> program as position independent code
>    * linking your program to shared libraries is a nightmare
>    * of course this model of communication is utterly importable across  
> architecture boundaries
> 
> And apart from that you also have to change some code within CHICKEN to  
> make the memory management system work with stuff that still lives on  
> other machines and to handle page faults/foreign object fetches correctly.
> 
> All this sounded like too much work for me, so the thing I seriously tried  
> was the following:
>    * compile the same program on all the target machines
>    * when the program loads, generate a two-way association between all  
> continuation addresses and "names" that have been (auto-)assigned to them
>    * instead of transmitting the code of continuations, just transfer its  
> "name"
>    * on the receiving side, map the "name" back to a code pointer and  
> construct a continuation object out of it

That's basically the implementation strategy of the Askemos.  Network
requests are replicated to all nodes.  From there a network of nodes
perform operations on the message (or derived messages).  Upon
transactions the OID (global unique object identifier) and it's current
version and a checksum of the proposed changes are verified in byzantine
aggreement.

> All other data objects, except continuations, can be serialized and  
> deserialized traditionally through the connection.
> 
> But as I said, I didn't manage to write some decent code that actually  
> does what I described here and I didn't come up with useful documentation  
> either, sorry.
> 
> What should be rather easy though, is to implement such a message passing  
> system between forked processes, because they always share code segments  
> -- you wouldn't even have to map code pointers to and from anything,  
> because they would just be the same.
> 
> cu,
> Thomas
> 
> 
> _______________________________________________
> Chicken-users mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/chicken-users