[Axiom-developer] Re: Two stage compiler

[Tim Daly]

 Martin,

I am in the process of tree-shaking (copying only code
that gets referenced) the compiler into literate form
in volume 9.

On 11/13/2010 11:28 AM, Martin Baker wrote:
> On Saturday 13 Nov 2010 15:55:08 you wrote:
> >    Martin,
> >
> > The Spad language is converted into an intermediate lisp
> > representations of the form (DEF ... ) and then turned
> > into optimized lisp programs and then turned into C and
> > then compiled.
> >
> > Tim
> Tim,
>
> Thanks very much for the reply.
>
> I somehow got the impression (possibly from daase.lisp.pamphlet I can't
> remember now) that the SPAD compiler and the interpreter were both driven from
> the database files: (interp.daase, operation.daase, category.daase,
> compress.daase, browse.daase) and that this involved a lot of custom code
> (different for the compiler and interpreter) written in a mixture of Lisp and
> 'boot' code. Have I got this wrong?
The compiler was a mixture of boot and lisp (although boot is just syntactic
sugar over lisp and gets translated to lisp). There is no boot code
in Axiom anymore, it is all common lisp.

Axiom is using a Pratt parser which is very efficient technology.
The Spad compiler parses the top level Spad syntax into a prefix internal
language using "(DEF ...", "(IN ...", "(LEAVE ..." and other forms that
tend to mirror Spad language constructs. These are lisp s-expressions.
From there these are translated to common lisp. When using GCL, the
common lisp gets translated to C and the C is compiled using GCC.

> It just seemed to me that if, the pattern matching could be separated out
> (still driven from the tables), then the rest (Lex, Parse, Semantic Analysis
> and so on) could be done by standard tools like ANTLR.
ANTLR is a java tool. Lisp is so much better suited for doing compiler work
than Java. Once the Spad syntax is translated into s-expressions, these are
the same as AST forms in a compiler. Lisp can easily manipulate these forms
and a lot of optimizations are applied before they get compiled.

Many of these forms can be written as macros and executed directly. Boot 
code
could not exploit the full power of lisp macros. It limited you to a 
Python-like
syntax which is somewhat pointless. Boot code also could not use the 
structuring
functions (like defstruct, CLOS) properly so a lot of the code relies on 
global
variables. This will eventually get fixed.
> As I have mentioned before I would really like better error messages and
> programming environment and it seemed to me that this might help with this? I
> also thought thought it might help you if it made the code more maintainable?
> But, of course, I don't understand all the practical issues that might stop
> this working.
The issue of better error messages is a question of keeping context
around. Axiom does tell you exactly what the problem is. It prints the
failed expression in <<...>>, the type of expression it was expecting and
the expression it found. If you understand the internal compiler 
s-expressions
it is all perfectly obvious :-)

Good error messages require that you keep a trail of where you are in the
original code and how each sub-expression in the original code relates to
the compiled form. This is a matter of careful book-keeping. It also 
requires
that you related the failure to the intention of the compiler, a more subtle
issue. Even more subtle is the nuance of the language details. Good error
messages are hard.

The Spad compiler was written as a research effort to explore ideas like
encapsulation and dispatching not only on the types of the arguments but
also on the type of the returned value (something very few languages STILL
do not allow). The people who wrote it were also the only users of it.
So the error messages were perfectly obvious to us.

One of the eventual goals of the rewrite will be to include this 
book-keeping
and error messages.

Tim Daly
> Martin Baker