Re: [Chicken-users] A call to arms

[Peter Bex]

On Thu, Jan 22, 2009 at 12:40:27PM +0100, Tobia Conforto wrote:
> Ok, since you asked for it... here are some random thoughts on an  
> "ideal web application framework"

Cool, I've been having plans for web application framework development
in Chicken for a long time.

> - No database/SQL (except where truly needed) just plain S-expr file  
> storage
> 
> Using Git as a filesystem is clearly the best solution.  I would use  
> it with s-expr files though, not XML (ugh!) as XML and HTML data can  
> always be represented with sxml if need be.
> 
> Databases can be left to their true purpose (dealing with a million  
> customer records, orders, products, invoices, forum posts...) while  
> everything else is stored in plain, versioned, s-expr files.

Just like Alaric, I disagree here not because I don't like s-expr files
but because the framework shouldn't force you into a particular way of
doing things.  I *really* dislike having a rigid framework that has too
many rules for you to learn and obey to.  It's much better to be able
to define your own rules (just like in Scheme) and be able to choose the
storage backend.

Besides, how are you going to do complex queries in Git stored sexpr
files?  For storage of simple things this is fine, but I regularly work
with applications and sites that either have a lot of data or require
complex searching. I very much need good database support.

In fact, I've often wished *better* DB support existed.  Most current
web application development regards the DB as a "bit bucket" in which
data can be thrown and usually only retrieved by ID.  I'd like _real_
database support, with transactions, multi-column primary key support,
real constraint support (proper ACID, basically) and possibly even
stored procedure support.

Of course, as I said above, it's important not to bake this in, so you
can use something more lightweight (couchdb, sexprs, git, etc) when
you don't need all that.  A full RDMS really is overkill if you only
want to store stuff and retrieve it by ID again, and I think that's
the main reason y'all don't seem to want DB support at all.

I blame two things for the current DB situation: MySQL and
object-relational mapping. MySQL didn't support any of this stuff
properly for a long time, and even now it has shitty support.  Because
most hosts run MySQL, most open source software doesn't support advanced
DB features.  This is a damn shame, and I think the "right thing"
approach of Scheme would be a good fit to such a system.

Object-relational mapping is the current fad for dealing with databases,
and it does not have very good support for querying either.  Either you
make it load associations lazily and it often does too many queries, or
you make it nonlazy and it fetches too much related data. You have crude
ways to obtain data; the has-one/has-many relations are the common way
to obtain related data, but I haven't seen good support for building
complex custom queries to obtain just the dataset you need (especially
regarding security checks; those can dynamically alter a query to get
only the stuff you're allowed to see). Going through these relations is
like doing one join at a time. Inserting related data is often quite
painful too. It's too easy to do the wrong thing in these systems.

</rant>

> - Multi-tiered, component-based architecture
> 
> This is a good start:
> 
> > Presuming that some hidden machinery has arranged a chain of  
> > generators by matching the incoming url, it would call a generator,  
> > e.g.
> >
> > (lambda (next)
> >  (html (div (h1 "some headline)
> >             (next) ; <<
> >        )))
> 
> 
> But I would make it more powerful.

I don't like this either. I think pre-post-order steps work better;
see below.

> See for example sxml's pre-post- 
> order transformer, where you can express arbitrary pre-order and post- 
> order transformations.  That is, having the option to capture the  
> output of your (next) and process it further.
> 
> Better still, we could come up with a Scheme equivalent of Apache  
> Cocoon's pipelines, where you define regexp matchers on incoming URLs  
> and assign 'pipelines' to them.  A Cocoon pipeline is a chain that  
> starts with one or more 'generators' (a component that generates a  
> flow of data in a common, abstract format), passing through zero or  
> more 'transformers' (components that alter the data flow in arbitrary  
> ways) and ending in a 'serializer'.  Cocoon uses the SAX API as the  
> abstract data format, but we of course could use the List.

Ah, now we're getting somewhere. I've given this a lot of thought,
and I think the "pipeline" would consist of individual sxml
pre-post-order transformer rule components.  So you define a number
of processing steps through which the abstract sxml is transformed to
a concrete HTML (or anything else) output.  In between any two steps
any module can insert its own ruleset, or prepend their ruleset to
an existing ruleset so that during the conversion it can "break in"
to the transformation. If that ruleset overrides all the rules in
the ruleset it is appended to, it effectively overwrites it.

I haven't been able to experiment with this yet because I'm still in the
process of writing the new Spiffy (which will supply fully parsed
headers, another thing I think will make web development much nicer;
you don't need to write your own ad-hoc parsers everytime you need to
read out the value of a header, but you can just operate on rich Scheme
data).

> - Fast development turnaround time
> 
> Having the framework split into basic plumbing and a collection of  
> components, would mean that we could have the plumbing compiled  
> separately from the components, and we could keep some components  
> compiled and others interpreted (instant turnaround time) while we are  
> working on them.

Yes, being able to just place a .so file next to a .scm file and have
Chicken use that when available is a great thing to have.

Cheers,
Peter
-- 
http://sjamaan.ath.cx
--
"The process of preparing programs for a digital computer
 is especially attractive, not only because it can be economically
 and scientifically rewarding, but also because it can be an aesthetic
 experience much like composing poetry or music."
                                                        -- Donald Knuth

pgpwa_VrhGuq5.pgp
Description: PGP signature