Re: [Chicken-users] Coming to Scheme from Python.

[Joerg F. Wittenberger]

Steve Freitas wrote:
> On Thu, 2006-09-21 at 11:14 +0200, Joerg F. Wittenberger wrote:
> > Porting Termite will not exactly help here.  Termite (on gambit)
> > allows you to manually transfer objects from host to host, where
> host
> > is a OS process.  On multicore machines you still have to run one
> host
> > per processor.  Then you would be left with *manually* controlling
> > object transfer and distribution.
> 
> With a threading system, you're still, at some point in your code,
> *manually* firing off new threads, so it's different work, not
> necessarily more.

Oh, I see your point.  I assumed you'd expect your Scheduler to assign
the processor automatically to your threads.

> > Gambit-C, chicken, rscheme (I don't know about Java???) and many
> > other
> > user level threading systems simly share the problem: user mode
> > threads are fast, but don't scale with multiple cores.
> 
> And that's why something like Termite makes so much sense. When you
> institute immutables, and copy by value on function arguments, it
> makes
> it easy to scale with multiple cores since you've eliminated most
> potential concurrency bugs. So you run one host per core, and dispatch
> the work. Have you look at Termite's syntax? It makes such things
> very,
> very easy.

You are true, that's exactly our experience.

I, personally, decided a few years back that I do not like to have to
care for even thread creation, mutal exclusion, dead locks etc.
That's why I came to write an Erlang-inspired (message passing)
environment with Scheme as application programming language: Askemos.

In contrast to Termite we do not (yet) have pattern matching on the
incoming queue (for some other reasons related to possibly different
message orders at hosts, that's an unresolved theoretical issue).
Also I did not know how to serialize a full continuation (as gambit
does), that's why I tricked users to supply them at application level.

In other areas Askemos is more mature than Termite: it doesn't hide in
some docs that users are supposed not to use side effects (like
set-car!), it just doesn't support them (on application level).

And eventually the highlight: it's the only environment I'm aware of,
which runs it's in virtual synchrony (byzantine agreement) at several
hosts at once.  Hence there's no single point of failure in the p2p
network, reason: legal liability; Askemos was designed to be
impervious to intentional insider attacs.  In fact, if one of the
hosts behind www.askemos.org is cut of the net, our programmers will
not even notice while they continue their work.

Ah, you where asking for chicken: once I started a port in 2003.
RScheme is not so different, except for the FFI and internals, but we
stuck close to R5RS where possible anyway.  Than we ran into
performance and UTF8 issues and had to abandon the port in favor of
featuritis.

> 
> >   Usually that's
> > a memory management issue (Felix should know) which doesn't go away
> > by
> > any amount of code in the language.  Needed is a n:m mapping from
> > user
> > level threads to system threads running the chicken VM (runtime).
> 
> If I'm understanding your suggested n:m mapping correctly, then
> Termite
> makes it really simple to establish what you're asking for.

Not at all.  I'm asking for some way to assing Scheme threads to a
processor *without* moving the thread out of the process where they
where startet.  The Askemos implementation we have today uses lots of
threads for short activities (since they are cheap).  If I wanted to
run them on multiple cores, there should be close to zero effort to
migrate threads among processors and they should still share
e.g. mailboxes with other threads on other cores.

However I understand that fast user level threads and multicore
scaling looks currently like conflicting goals.  I would *not* vote to
pay on the thread system side.  Instead close assesment of application
needs will probably reveal better ways of multicore scaling.  Example:
Askemos runs it's byzantine agreement over HTTPS (huge waste, I know).
We made the SSL layer into an extra executable, thus chances are that
the kernel will put the memory hungry server on one core and those SSL
proxies on others.

best regards

/Joerg