Re: [Chicken-hackers] CR #1142 and upcoming changes

[Alan Post]

On Mon, Aug 18, 2014 at 06:54:48PM +0200, Peter Bex wrote:
> On Mon, Aug 18, 2014 at 10:00:46AM -0601, Alan Post wrote:
> > I used Chicken Scheme to bootstrap my company.
> 
> That's really cool!
> 

Thank you!  It's the most successful Scheme program I've ever
written by a large margin.  I was recently trying to find a
single routine that best expressed what a bootstrap in Chicken
does, exactly, and I settled on this as a good balance of
expressiveness without requiring too much context:

  (define (alist->argv l)
    (define (fn k v)
      `(,(string-append "--" (symbol->string k)) ,v))
    (flatten1 (map-alist fn l)))

This routine was where I took any of a number of highly experimental
data structures and passed them to programs which had well-defined
interfaces, converting an experiment to infrastructure, if you will.
One end could change to whatever it needed to be, the other end was
where I could put solved problem, and this interface let me see how
far apart those two components had gotten.

I know that's not much too look at for this list--I wanted an
example I could give to people who are afraid of Scheme.

> > As of a few months ago, I am no longer running any scheme code, it
> > has been ported to either C or Python.
> 
> What is/was the main reason you had to rewrite to Python?  C I can
> understand, for performance reasons.
> 

When we started, it was not clear how much of the codebase would be
in C vs. Python.  I was using Chicken to as a high-level language
that I could compile to C (that was a hard requirement, we needed our
application to run as a shebang line).  As we developed the
application it became clear that nearly all of it could be written
in Python, and the pieces that could not were natural to express in
C, so the need for the abstraction provided by Chicken went away.

You may be happy (or disgusted!) to know that the engine I wrote in
Scheme had been implemented, quite naturally, as a series of
promises.  Depending on the user input, it calculated any needed
dependencies on the fly to produce a correctly ordered result.

In Python, to get the same result, this engine became a four-pass
compiler.  It's certainly easier to see the structure of the thing,
but I needed the flexibility provided by Scheme to see the solution,
and the feedback I got here on the user interface for genturfa'i (my
PEG compiler) was "why the hell did you write a multi-pass compiler
in Scheme?!")

A side benefit I hope to captalize on later is that my logging
infrastructure is based on association lists.  I did that because I
wanted to future-proof my logs for the time when I need to go back
to them and profile the system, but it also provided an upgrade path
from the Scheme code to Python.

> > I am still maintaining my eggs, and will be happy to do so through
> > this transition--one problem I ran in to prototyping, which frankly
> > surprised me--was hitting code- & infrastructure-related scalability
> > limitations in Chicken.  I'm happy see the effort above as a result.
> 
> It would be nice if you could elaborate on the exact things you ran into,
> so that we may try and fix them.
> 

Hands down the largest difficulty I had with Scheme was debugging.
You gave me a hint some time ago that adding type declarations to my
code would have made my job easier.  I haven't had a chance to explore
that, I'm sorry to report.

But my specific pain: I got to the point where I needed to debug a
compiled application, that was my deployment model.  In that I could
not produce results when I ran the code interpreted.  I put together
what tools I could for capturing and managing exceptions, so I could
get traces and debug output, but I often found myself pouring over
a couple thousand lines of trace trying to find the relevant section
of code that crashed.  The tools I had to write to make this visible
in the first place were the first ~hundreds of lines, and if I had
an invocation to genturfa'i (le PEG parser), I had to look at that
enormous, entirely machine generated, trace.

Thematically, Chicken let abstract my problem away to an amazing
degree--I would write two pieces of code, realize they were similar,
and merge them together.  I had a lot of sweet spots like this where
I kept seeing a new pattern, parameterizing it, and finding novel
uses I hadn't anticipated.

Much as Chicken let me do this, my ability to debug problems became
substantially constrained.  I found that the further I got away
from core the harder it was to connect the location of a failure
to the part of the code that needed to change.  My lack of ability
to debug was the significant technical driver that limited what I
could do in Scheme.

By way of an unfair contrast, I'm able in Python to run a collection
of jobs, collect any failures, and jump around to each of the stack
traces and do a post-mortem debug.  I can safely attach information
as any exception heads up the stack but still debug at the original
failure point.

If I, personally, were going to work on one exciting project in
Scheme, it would have to be a debugger.

hth,

-a

PS: this community has given me far more than I could ever hope to
    give back.  I thank each of you for what you've done for me.

-a
-- 
analytics in physical space: http://venueview.co/
my personal website: http://c0redump.org/