[Axiom-developer] Re: Build machinery -- support for out of source build

[Gabriel Dos Reis]

On Mon, 2 Jul 2007, address@hidden wrote:

| > You are wrong. A dependency tracking system needs to be part of the
| > system, not just at the Make level, but also the system
| > level. Otherwise you get into obscure contorsions as exemplified by
| > the build of depsys.
| 
| Depsys exists for a reason.

I know why it exists.  I was not discussing its exitence -- which can be
made obsolete, I'm saying that because I've done it this week-end.
Rather, I'm talking of *the way* it is constructed.

| It handles things like macros in lisp
| and it removes things that are only needed at compile time from the
| final running image. 

Make appropriate uses of 

    (eval-when (:compile-toplevel) ...

or

    (eval-when (:compile-toplevel :load-toplevel) ...


Plus, there are Lisp systems that some of us would like to support
(e.g. ECL, which shares a common ancestry with GCL) where the way you build
depsys is simply not possible: There is no notion of saving a current
Lisp image.  Yet, I succefully bootstrapped bootsys and depsys.  The
only thing that stopped me from building the whole system is that ECL insists
on ANSI Lisp compliance whereas Axiom silver insists on ANSI Lisp defiance.
   

To build depsys with such systems, you need to have a clear picture
of the dependency *expressed in the code*  -- not just outside in a comment,
separated from the code.  For example, I extended Boot with a macro
"import-module" which has the effect of tracking dependencies.  Much
like C-#include, except that it is not as brittle as C's -- it is
idempotent for, no special include guard etc, augment the compile and
load time environments as necessary.

| If you miss the depsys dependencies you'll find
| that you get "clean" lisp compiles but the generated code is wrong
| due to missing macro expansions. 

Not, if you have the system keep track of the dependency, not just
at Make level, but at the system level.  For example, with the
"import" construct I added to Boot, I can use bootsys to generate
the Make dependency, and at the same time have the Lisp system keep
track of the dependency at load time.  I've completed that work this
week-end while I was trying to finish something I promised
Ralf (have Axiom build standalone programs).

The way I use bootsys to generate dependency is not a novel idea.
Automake uses GCC to automate dependency tracking; GCC has a switch 
for generating Make dependencies.


| Make and lisp have slightly different ideas about what it means to
| depend on something. If you actually list all of the dependencies at
| the Make level you'll find that you do a lot of unnecessary work.

That is not the case of the Axiom components I've tried so far.

| This only shows up when you begin to change a single file and do
| Make again.

I've done that many times -- I did not suffer any of your predictions.

| A simple example is that the algebra code is built in layers. Each
| layer is built depending on the layers beneath it. Originally the
| algebra Makefile specified this dependency, which worked fine during
| the build. Yet if you change a single algebra file and do a second
| Make you can end up rebuilding a lot of the algebra needlessly.

If the dependencies are properly written and the system the properly
modularized, there is not need to.  There already exist a first 
approximation that works very well in build-improvements -- look into
src/alegbra/Makefile.  Please try it and tell me which aspects of it gets into
your way in everyday things you do.


I have no doubt that you have encountered many issues.  However, after
spending a fair amount of resources studying Axiom's current build setup it
appears to me that many ideas were flawed and many concrete realizations of
ideas were ill-advised and wrong headed.  So, one should be listening to your
reports on this issue; but only to be sure to move away from the current
setting, to be sure NOT to perpetuate the current build.  The setup should
have been done using best practice at time -- it apparantly didn't.  Now is an
opportunity to correct that path.

Similarly, I hope we do not lose Waldek's rework of the algebra bootstrap.
We should get it and improve it.

-- Gaby