RE: [Axiom-developer] BAD tim

[Bill Page]

Tim,

Thanks very much for the good humour and detailed reply.

On October 31, 2005 10:00 PM you wrote:
> 
> Ah, language wars ... the only thing that out-emotions a
> license war

Damn! And I just finished saying I didn't want to start such
a war...

> 
> I suppose, having been cast as the bad guy in this morality
> play, that I'm required to have the long monologue about my
> evil ways (so the hero has time to sneak around behind me
> and kill me) :-) So sit back, grab some coffee, and watch
> the opinions fly...

I wouldn't worry about some sneaky but merely human hero. It's
the spirits of this play that haunt do most of the haunting.
Is that a long echoing evil laugh I hear? Ah, it is still
"All Hallow's Eve" here... :)
http://en.wikipedia.org/wiki/All_Hallows_Eve

What are the tools I need here? Hmmm... \cross and \dagger ?
Trick or treat?

> 
> POINT 1: Axiom needs to be future-proofed.
> 
>    Axiom needs to be torn down, re-architected, re-thought
>    and rewritten cleanly so it can be modularized, maintained,
>    and extended. Which implies a major rewrite, likely done
>    by me.
>

I am not convinced that this is necessary or even possible,
although I do not doubt your knowledge or your ability to do
this - given enough time. According to the history of Axiom
that you have described in various previous emails, Axiom has
already undergone several cycles of re-writing. I don't see
any evidence that such re-writing has resulted in making
Axiom any more "future-proof".

On the other hand I don't think it is necessary, because
Axiom is here now as open source. Axiom is certainly complex
but there are already good examples of other large and complex
open source programs that have managed to attract some very
skilful and dedicated developers who are capable of dealing
with this sort of complexity. The example of the Linux kernel
comes to mind.

Unfortunately, I think it is true that although several years
have passed since Axiom's release as open source, there is
still only one active developer with a thorough knowledge of
Axiom - Tim Daly. In order to best "future proof" Axiom, rather
than re-writing Axiom, I think the best strategy might simply
to give up programming and learn something about genetics and
cloning. ;)  Failing that, there remains the task of documentation
and education. I think you have been doing a great job at that
and I urge you to continue.

> 
> POINT 2: The rewriting will change existing code. 
> 
>    If we look at the "start-up sequence" we see:
> 
>               restart (lisp)
>    init-memory-config (lisp)
>              initroot (lisp)
>                reroot (lisp)
>                  spad (boot)
>               runspad (boot)
>            ncTopLevel (boot)
>             ncIntLoop (lisp)
>               intloop (boot)
>   SpadInterpretStream (boot)
>    intloopReadConsole (boot)
> 
>   and, if you follow the side-paths you'll see the constant
>   switching between boot and lisp at every level in nearly
>   every function.
> 
>   This inhibits "collapsing and rewriting" code which will 
>   have to happen during the rewrite. Functions will either
>   have to be collapsed into lisp or boot.
>

As a long term strategy, I agree, however as I have stated
I think boot is the better choice.
 
>   Historically this interleaving occurs because new function
>   was added/deleted/replaced. The "nc" prefix implies "New
>   Code", a newly rewritten parser that partially replaced the
>   previously newly rewritten parser. The "shoe" code, which is
>   the new, improved "boot" language exists in there. The "meta"
>   code used to exist but is gone. Addition of Fortran support
>   (for NAG), Foreign Language calls (for graphics and the
>   browser), Lisp vendor specific code, etc. required lisp.
>

Why was lisp required? Surely it is possible to do these
things in BOOT. In fact, aren't most of the NAG extensions
written in BOOT? E.g. ls src/interp/nag*
 
> 
> POINT 3: Boot is a dead language. 
> 
>    There are approximately 10 people still living who have coded
>    in boot and every one of them is doing something else with
>    their life. You have to be careful in boot. It is case sensitive 
>    so APPEND is not the same as append.

Wow, that comment certainly dates you! :) Almost all programming
languages in the last 20 (30?) years have been case sensitive.
Usually that is considered an advantage.

>    It is often not clear what a construct will translate to and
>    you end up reading the lisp to learn. A single, misplaced
>    space will change the meaning of the code.

That is true of most programming languages. Understanding a
"construct" by seeing how it "translates" is usually just a
matter of lack of fluency in the language. This can be a result
of inadequate documentation and/or experience. In a language
that syntactically depends on indentation, of course misplaced
spaces can change the meaning of the code. But this is not
essentially different than misplaced commas or braces in other
languages.

> 
> POINT 4: Boot SERIOUSLY complicates the Axiom system. For
> instance:
> 
>   1) Bootstrapping boot
> 
>    The boot compiler is written in boot and needs to be compiled
>    using itself. If you have a running Axiom system this is not
>    a problem. However if you start from a clean lisp system it
>    is a big problem. I cheesed up a way to do this but it is
>    fragile and ugly. But given that we build from scratch there
>    is no other way.

But that is exactly how the C programming language works. The
Gnu C compiler is written in C and bootstrapped in a similar
way.

Also, the current Axiom build starts by first building GCL from
the C sources. This is also a kind of bootstrap process. I don't
see that the BOOT level adds much to this already existing
complexity.

> 
>   2) Makefiles come in multiple stages
> 
>    Boot forces Axiom to be built in stages. Thus Axiom cannot 
>    use modern tools like ASDF. A pure common lisp Axiom interpreter
>    can be built and loaded directly into a lisp image, interpreted
>    or compiled.

I thought there was already a version of interpsys called debugsys
which provides an interpreted environment that is more friendly to
debugging in lisp?

Not being a lisp programmer, it is entirely unclear to me what
advantages ASDF might have over more general tools like 'make'
especially considering that the Axiom source files are actually
literate programming documents (pamphlets), not lisp source code.

> 
>    Lisp, BOOTSYS, DEPSYS, INTERPSYS, AXIOMSYS. These can all be
>    collapsed if Boot disappears.
> 

Although I still recall that this looked very complicated to me
when I first started working with Axiom, now each of these stages
in the current Axiom build process makes sense to me. Since Axiom
is a large and complex system, building it a "layer" at a time
seems quite logical. If these were all collapsed into one, then
I worry that this would all start to look impossibly complicated
to me again.

>   3) lsp, lisp, clisp, ${LISP}
> 
>    All of these exist because boot exists. For example, .clisp
>    is a translated boot file whereas a lisp file is hand coded.
>    ${LISP} exists to smooth over this bump. I won't go into the
>    historical reasons why these came about but they exist as a 
>    side-effect of boot.
> 

Just another type of intermediate file, and Axiom already has
many of these.

>   4) The interpreter has a boot compiler built in
> 
>    There are complications to the interpreter to handle boot-
>    related development from the command line which no-one is
>    ever likely to do again. This code could disappear and, having
>    gone away, simplify the interpreter

This still sounds a little vague, but in general if it was
documented then perhaps someone who was working with BOOT would
find this functionality useful and use it again.

> 
> POINT 5: The boot language is undocumented 
> 
>    It is likely to remain so as one of the authors is dead 
>    and the other one might be. There are no primary sources of
>    documentation.

That does present a problem. But since there is still a lot
of boot code in Axiom, figuring out how boot works and
documenting it does not seem to me to be any more difficult
than figuring out how the boot code works and re-writing it
in lisp.

> 
> POINT 6: Boot does not give you the full power of common lisp.
> 
>    Boot cannot handle backquote, defstruct, and other primitives
>    not to mention CLOS, CLIM, etc. So as lisp evolves boot will
>    "further fork" away from the living edge of lisp ideas and 
>    limit the axiom programmer. 
>

This is true, but I think that perhaps by accident or by design 
it can be largely cast as an advantage. After all, the purpose
of boot is to implement SPAD and the Axiom interpreter in a
language that is higher level than common lisp - not to be a
general purpose programming language.
 
>    Witness the data structures used in boot, mostly lists.
>    Frames are lists of lists, history is a circular list of
>    lists, etc. Where are the data structures? Who will redesign,
>    rewrite, and document boot extensions to fully cover common
>    lisp?

This point was also recently raised by Juergen Weiss. I believe
that his view is that it is possible to extend boot to include
these data structures in boot syntax but that it might be
preferable simply to use boot's built-in mechanism to "escape"
to lisp where necessary to define these structures and then to
call the data structure access methods in the usual way in boot.

> 
> POINT 7: Boot loses program-data symmetry.
> 
>    This is a matter of religion but having been at this game for
>    35 years I've seen a hundred languages built on lisp that try
>    to "eliminate the parens" and all will eventually die (including
>    Python).

I don't see that this has to do with parens or not. Using
parenthesis to denote nested lists is just one more syntax.

> 
>    The paren syntax is fundamental. Look at how Axiom really 
>    works under the covers. It doesn't build boot functions, it
>    builds lisp functions. Why? Because you can cons up a lisp
>    function, compile it, and run it directly. Axiom code and
>    data intermingle in the algebra lisp files. The databases
>    have code and data in them. Data <==> Programs. Fundamental.

I don't agree that lisp has anything special to offer here
except perhaps by being the first language to formalism this
relationship. Any higher level language in which functions are
first order objects has this property, although usually not in
the way it is implemented in lisp. This is especially true in
functional languages like Haskell and Ocaml and probably also
in Aldor. "cons'ing up a lisp function" is just a less structured
way of doing the same thing.

> 
> POINT 8: Tools don't support boot
> 
>    Emacs balances lisp parens. It lets me do lisp function 
>    lookups. It understands the lisp syntax including escape
>    chars. SLIME lives in emacs and provides support. Code
>    walkers walk lisp code. Pretty printers understand lisp.
>    Debuggers understand lisp (I want to fix it in the language
>    I broke it in). ASDF can manipulate lisp. Programmers 
>    speak lisp. Blank spaces don't break lisp.
> 
>    Boot is unsupported anywhere by anything.

Kai Kaminiski has written a simple emacs mode for boot.
See http://wiki.axiom-developer.org/AxiomInEmacs Boot is
syntactically similar to Python so if there was a need for
enhanced emacs support, then I expect a lot of this
functionality could simply be "borrowed" from the Python
extensions of emacs (and other syntax aware editors).

I think it remains to be seen how some of these other lisp
tools could be used with Axiom.

I don't know many people who "speak lisp" except perhaps
during severe rapid eye movement while asleep. ;)

> 
> 
> POINT 9: Where are the programmers?
> 
>    Who will write in boot? I'm the only person likely to be
>    hacking the interpreter for the near future, mostly because
>    it is so big, ugly, unstructured, and undocumented. Even if
>    people code in boot to maintain the interpreter how often
>    will they do that? How will they maintain their skill at
>    writing boot? I've written many, many thousands of lines of
>    lisp and use it continuously. Who are the programmers who
>    will do that in boot?

I agree! That is the #1 question. But besides Tim Daly, where
are the other lisp programmers? How will they maintain their
skills at using lisp in Axiom?

Axiom is now open source and is at least potentially exposed
to a much larger range of people than ever before. I believe
that in the future there is a good chance that the people who
may help to maintain and further develop Axiom will be those
people who started by writing Axiom library code in SPAD.
>From SPAD it is a natural step down the ladder to programming
in boot. I expect that many of these people would never have
had any previous exposure to lisp. So forcing them to program
in lisp just to maintain Axiom would probably feel more like
"falling off the ladder" than just stepping down.

Of course this is currently mostly speculation (with the
exception perhaps of myself and Martin Rubey).

> 
> POINT 10: Boot was an experiment in surface syntax
> 
>    Historically Axiom was a platform for experimenting with 
>    many research topics. One of these topics was a favourite
>    of Dick Jenks and Bill Burge, that of language design.
>    Scratchpad, now called Axiom, was changed many times to
>    introduce new ideas. This research is no longer the point
>    of Axiom and is better done elsewhere. We neither have the 
>    expertise nor the research goal of surface language design.
>    Let Python do that. Or Spad/Aldor fans.

I think the Python designers also (for the most part) consider
their work done.

I thought Spad/Aldor was an essential part of Axiom and
necessarily closely related to the Axiom interpreter, so I
am not sure what you mean by your last comment.

> 
> POINT 11: Do language design where it counts.
> 
>    If we are going to devote research effort toward language
>    design lets do it at the Spad/Aldor level where it counts.
>    This is where the real language expertise should be applied.

I agree. I don't see any serious need to re-design the boot
language - simply to document it as it works now.

>    For instance, why do I have to write programs to express
>    functions?

In Axiom you can already expression functions in other ways
than imperative code. The Axiom interpreter, SPAD and Aldor
have a (more or less) complete functional subset.

>    Why can't I write a domain using equations? Why doesn't
>    the Spad language support that?

Could you elaborate on what you mean by "write a domain using
equations"? Are you talking about an alternative to the way
that Axiom implements "axioms" now simply by assertion?

> 
> POINT 12: Future directions assume lisp
> 
>    Notice that one of the near future goals is to connect Axiom
>    and ACL2. ACL2 understands lisp-ish kind of languages, not
>    boot.

Besides the fact that ACL2 is implemented in lisp, I don't see the
connection between ACL2 and "lisp-ish" languages when it comes
to proving program correctness.

There are other tools similar to ACL2 such as Coq which is not
written in lisp at all. Coq has already been used with Axiom
in some research.

> 
> 
> and now, on to the recent changes
> 
> POINT 13: Don't break code until you understand it.
> 
>    I'm working hard to understand the code so the existing code
>    is a straightforward translation (well, mostly, with some
>    rewrites that I know are logically equivalent).
> 
>    Once I've completely figured out the various twists and turns,
>    uses and references, and other points I'll be recoding the
>    whole process. There is no reason (other than historical) why
>    there are a dozen entry points into the interpreter that set
>    weird, undocumented, stateful flags. All that cruft must go
>    away.

I agree with all that, while de-emphasizing the need to remove
all of the cruft right now. But I fail to see why recoding
anything from boot into lisp is involved.

> 
>    So the code is ugly. At the moment. But it will get better,
>    cleaner, clearer, more robust, and easier to change. History
>    and Frames, for example will use structures that don't depend
>    on 
>       (caar (cdddddr (cdddddr x)))
> 
>    It takes time. It takes work, probably 2-3 years worth of
>    work.

I think you worry too much about how "ugly" to current code is.
It works. That is what matters most to people who want to begin
using Axiom now. I doubt that there is much you can do to the
Axiom internal code in the way of re-writing it into lisp that
would make it any more appealing to this putative new generation
of Axiom users. Still I am hopeful that among the thousands of
new people who have recently been exposed to Axiom, there will
be some who will want to take up the challenge of dealing with
what's on the inside. If/when they arrive, I think it is more
important to have good documentation than it is to have pretty
code.

>   
> and now, on to the personal...
> 
> POINT 14: I'm a lisp programmer 
>    
>    (clearly)
> 

I don't seen anything inherently evil in that. :)

Cheers,
Bill Page.