Re: [Axiom-developer] sbcl and Axiom

[root]

CY,

============================================================
BUILD ORDER

Axiom runs a tree of Makefiles, starting with the root Makefile.
Each subdirectory contains a Makefile.pamphlet. The system does
a recursive walk of the Makefile tree by subdirectory.

In a subdirectory running a Makefile that Makefile has the
responsibility to set up global conditions for all of the Makefiles
in each child subdirectory. 
  For each subdirectory
    set up global conditions (eg directories)
    notangle the Makefile.pamphlet ==> Makefile
    ( cd subdir ; make )


You can see what the build will do by recording the console
output of a system build. The exact sequence necessary to build
the system is laid out in linear form during system build. Every
make stanza outputs a line that says what it is doing. Each line
that is output has a unique number (for the subdirectory) as a
prefix to the line. So you'll see things like:

32 making /tmp/axiom49/obj/linux/graph/view2D/write2d.o from 
/tmp/axiom49/int/graph/view2D/write2d.c

Notice the '32' prefix which gives a clue about which make stanza it is.
This line is output from the Makefile running in the 
  src/graph/view2D
subdirectory. If you look at the src/graph/view2D/Makefile.pamphlet and
search for 'echo 32' you will see the stanza:

${MIDOBJ}/write2d.o: ${MIDINT}/write2d.c ${LINC}/write.h ${HEADERS}
        @ echo 32 making ${MIDOBJ}/write2d.o from ${MIDINT}/write2d.c
        @ ( cd ${MIDOBJ} ; ${CC} -c ${CFLAGS} ${MIDINT}/write2d.c )

This stanza says: 

  if you need to build  ${MIDOBJ}/write2d.o
  then you must first build
     ${MIDINT}/write2d.c 
     ${LINC}/write.h 
     ${HEADERS}
  once these have been built it echos the '32' line and then
  it builds ${MIDOBJ}/write2d.o 


============================================================
BUILD-TIME DIRECTORY STRUCTURE


There are 4 top-level directories 
 ${SRC} -- the source tree (human written)
 ${INT} -- the intermediate (machine independent, machine generated)
 ${OBJ} -- the object files (machine dependent, machine generated)
 ${MNT} -- the final ship tree
and within each Makefile combinations of these make local variables.

The system is set up so that it is possible to copy the SRC and INT
directory to another system and restart the build. Historically this
was important because it used to take 3 weeks to build a system from
scratch (well, not scratch but from a previous system).

The makefiles are designed so you could mount NFS disk space on
multiple systems and build them all in parallel. Axiom used to build
on PCs, Symbolics, AIX, Solaris, and VM/370 all from the same 
SRC and INT. The target system is named by the ${SYS} variable,
these days usually just linux. The target system is determined from
the $AXIOM variable so:

export AXIOM=/tmp/axiom/mnt/linux
                            ^^^^^
                             the target

and you could build others by NFS mounting the source directory and
setting the AXIOM variable such as:

export AXIOM=/tmp/axiom/mnt/solaris
export AXIOM=/tmp/axiom/mnt/freebsd
export AXIOM=/tmp/axiom/mnt/symbolics
export AXIOM=/tmp/axiom/mnt/aix
etc...


So the basic idea is that you start from the sources ${SRC},
  cache a lot of machine independent work in ${INT},
    use ${OBJ} as a machine specific temp area (.o files, etc),
      and create a machine-specific "ship" system in ${MNT}.

Once the machine-specific directory exists (${MNT}/linux, ${MNT}/aix, 
etc) you can throw away everything else, including the sources.



============================================================
BUILD-TIME INTERMEDIATE IMAGES


We need to build lisp images for specific purposes. 

Initially we need a lisp image that contains our patches.
That is machine-specific, machine-generated code so it logically
belongs in the OBJ directory (see above). And since it is system
dependent it lives under the SYS subdirectory. In this case we get
${OBJ}/${SYS} == obj/linux

If we look in obj/linux we see

lisp
bootsys
depsys
interpsys
debugsys

So the key steps that use these images amount to:

step 0: build noweb

step 1: build a lisp with our patches ==> lisp
  if we had a fully patched lisp (or did not depend on any 
  special features) we could just copy a lisp rather than build one.


step 2: build a lisp with the boot translator ==> bootsys
  extract the boot translator clisp files from the boot files into INT
  load the boot translator into the lisp image
  save it as bootsys

step 3: translate (use bootsys)
  translate the boot files into clisp files using bootsys
  after this point bootsys is useless and we abandon the image

step 4: build a lisp image with all the macros ==> depsys
  we use lisp macros which are needed at compile time but not runtime
  we load those macros into a lisp image and save it as depsys
 
step 5: compile lisp (use depsys)
  we use depsys to compile the clisp/lisp/lsp files 
    clisp is boot-generated code
    lisp is lisp-original code
    lsp is GCL generated code

step 6: build a lisp image that contains the interpreter/compiler ==> interpsys
  we load all the compile files into lisp
  we can preload some algebra code for efficiency
  and save it as interpsys (eventually copied as AXIOMsys into ${MNT}/${SYS}

step 7: autoloads (use depsys)
  we have a set of files that are used in special cases but are not needed
  we use depsys to compile these into the autoload subdirectory.  

step 8: algebra (use interpsys)
  using interpsys, compile the algebra 

step 9: we may have to do deep debugging ==> debugsys
  this is purely for my use. sometimes the only way to find a 
  bug is to run the interpreted lisp code. 

Just to add to the pain each of the source files are documents so
we have to extract the required sources at each and every step.




==============================================================
BOOT HISTORY


Boot is necessary to translate the boot files to clisp files.

It is not normally used in a running system. However, boottocl
is a command line tool that allows you to dynamically translate
a boot file to a clisp file. This is a way of making dynamic
changes to the sources (it's lisp, after all). The system used to
be built on top of an existing system. Our style of working was
to keep modifying the system while it was running. Thus the boot
code was changing while we worked. Eventually this led to a system
that could no longer be built from scratch but required a running
system to build. Boot is one example of this problem. The boot
compiler is written in boot. In order to start the process we now
cache the generated lisp files in the original documents. Hacking the
boot compiler would require re-generating and re-caching these files.

The boot code is slowly disappearing from the sources.  Once that
happens the boot subdirectory is no longer necessary and boottocl will
disappear. As you can see from the steps above boot adds complexity
we don't need.

As mentioned previously in these mailing lists Dick Jenks did work
in languages. He created a Meta language to describe the boot syntax
and then a boot language.  This is one of the early attempts to build a
paren-free version of lisp (\begin{religion} similar to python and 
many other failed attempts, past and future \end{religion})

Note that not everyone agrees that boot should die as they like 
the language. Since there is nothing axiom-specific about the boot
language it would be perfectly possible to fork off a boot language
project and let it live elsewhere. 


==============================================================
FUTURE DIRECTION

I'm rewriting and documenting the system internals into straight
common lisp. At the moment I'm working on the interpreter which
will eventually become the 5th volume of the axiom series (hence,
src/interp/bookvol5.pamphlet). This fully documented interpreter
will mirror the existing system. Once it is documented we can
modify it to suit future needs.

My ultimate aim is to make the system a literate lisp program that
only needs to be compiled and run in a single pass on any ansi
standard lisp. The lisp ASDF facility will allow us to remove the
makefiles and the autoconf/autoload machinery. Boot will disappear
as will the many lisp images, intermediate directories, etc. Lisp
will be extended with native notangle/noweave so you can just
load or compile literate files.

The next version of axiom will consist of a set of volumes that are
literate documents, fully explaining the code. The compiler will be
its own volume and can be replaced by a different compiler.  The
interpreter will be its own volume and can also be replaced (possibly
by an aldor standalone version).  The algebra will be organized (in
some fashion) as volumes that fully document that algebra. Axiom will
sit on top of an ansi common lisp literate base (or aldor standalone 
base if we go there).

The build process should involve just a drag-and-drop of the volumes
and any literate papers containing algorithms you might want to use.


Tim