Re: [Chicken-users] faster threading

[F. Wittenberger]

Hi all,

Am Dienstag, den 28.10.2008, 11:37 +0100 schrieb Jörg F. Wittenberger:
> If you agree to use smarter data structures in the scheduler, I'll take
> my time to expand my proposal.

Here we go!

Attached a slightly modified scheduler.scm and a file rbtree.scm.

> * At the end of the day, we might take a minute to consider the
> question which data structure would be the best.  I took the wttree
> only because

Thankful for a suggestion from Marc Feeley I reused rbtree.scm from the
snowfort repository.  The attached version extends the snow version by
optionally generating lookup and fold procedures for the generated
trees.  (And contains some chicken specific lines on the head.)

rbtree.scm is currently literally included by the attached scheduler.scm
- but maybe we better only include the generated code or rewrite it
hygienically.  The latter is something I better leave alone until I got
use to those macros.

summary of changes

* file descriptor list (##sys#fd-list) became a priority queue
* timeout list (##sys#timeout-list) became a priority queue
* a new 'waiting and a ##sys#waiting-queue where created
* some select(2) calls (and associated processing!) where saved

The result - well with about 250 open file descriptors serve 20 parallel
http requests, which each cause several threads to serve plus some xml
processing (plus http chunk encoding with i/o flush after each chunk!).
I recall this test being impossible with the chicken version of askemos
within a 20 seconds limit.  First I moved to a quite elaborate timeout
handling anyway.  This plus just the wttree made the test run in about
8-12 seconds.  Now we are at 4.

I'd say I'll stop here.  Good enough.

Things left to be done:

The queue nodes are defined as srfi-9 records.  Maybe this would be even
better as a lolevel structure?  Felix?  Is there something to gain?

> * I would to change the interface the scheduler uses to manipulate the
> fd-list to read like fd-list:ref fd-list:ref/default fd-list:delete!
> where ref, ref/default etc. have the same signature as in srfi-69.  (And
> have those interfaces exported from the smarter data structure.)  Then a
> simple  "(import (prefix my-great-structure fd-list:))" will do all the
> work.

This has not yet been done and I might not come around any time soon.

Best regards

/Jörg

rbtree.scm
Description: Text Data

scheduler.scm
Description: Text Data