[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Chicken-users] Async IO (was Re: libcurl?)

From: Thomas Chust
Subject: Re: [Chicken-users] Async IO (was Re: libcurl?)
Date: Wed, 05 Oct 2005 09:29:57 -0000
User-agent: Opera M2/8.02 (MacPPC, build 2148)

Am 05.10.2005, 03:41 Uhr, schrieb Peter Keller <address@hidden>:

On Tue, Oct 04, 2005 at 10:21:55PM -0000, Thomas Chust wrote:
CHICKEN does not have any feature that would make continuations
serializable; it would mean that you somehow have to transform machine
pointers to arbitrary memory locations where your continuations' assembler
code lives into stuff that can portably be transferred over a network

I'm sure there are some ideas that could help you perform a virtual
serialization of the continuations. For example, machine A has the
"original" copy of the running executable, machine B has a certain form
of a skeleton executable running in memory with a very small memory foot
print. Now, the continuation (wholesale) is transferred to the target
machine where it is propped up in the some executable pages in the same
range as before and control jumps there.  Of course, the rest of the
memory heap (known by the skeleton since it conspired with machine A)
is protected via mprotect, and when the first segfault happens, it is
caught, and the offending pages brought in from machine A to machine
B. Then the continuation happily continues. After a while, a working
set emerges, and machine B's heap gets managed independantly of machine A.

Machine A knows what pages got served, so it is aware of such things
during garbage collection, and when machine B's pages are finished,
they are merged back into Machine A's domain. I'm handwaving the merge
process, since I haven't thought about this more than 5 minutes. Though I
suspect the merging algorithm can move stuff out of the pages of machine
B to better location on machine A since ultimately you're dealing with
higher level objects and the scheme implementation isn't going to care.

I'm sure there are a lot more gotchas in a method like this (like race
condition between garbage collection on machine A and machine B's request
for the pages), especially without deep thinking about it, but it might be

If you get this working, I honestly have to congratulate you -- I tried it
once (trying to cook up something like Kali Scheme on CHICKEN basis) but
failed miserably to come up with any decent, let alone portable or even
elegant solution. The day anyone has a really brilliant idea how to solve the problem, I will certainly resume this dead project of mine -- it would
be a wonderful toy ;)

Did you happen to document the methods you tried?



your scheme for doing this is fine and basically the same thing as Kali Scheme originally did. But Kali Scheme was running on a bytecode interpreter, CHICKEN is running directly on the processor, so there are several subtle issues involved with taking binary code from one machine and sending it to another to execute. Some important points are: * you may not be able to map everything to identical addresses on the target machine for various reasons, so you have to compile the full program as position independent code
  * linking your program to shared libraries is a nightmare
* of course this model of communication is utterly importable across architecture boundaries

And apart from that you also have to change some code within CHICKEN to make the memory management system work with stuff that still lives on other machines and to handle page faults/foreign object fetches correctly.

All this sounded like too much work for me, so the thing I seriously tried was the following:
  * compile the same program on all the target machines
* when the program loads, generate a two-way association between all continuation addresses and "names" that have been (auto-)assigned to them * instead of transmitting the code of continuations, just transfer its "name" * on the receiving side, map the "name" back to a code pointer and construct a continuation object out of it

All other data objects, except continuations, can be serialized and deserialized traditionally through the connection.

But as I said, I didn't manage to write some decent code that actually does what I described here and I didn't come up with useful documentation either, sorry.

What should be rather easy though, is to implement such a message passing system between forked processes, because they always share code segments -- you wouldn't even have to map code pointers to and from anything, because they would just be the same.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]