[Axiom-developer] development process

[root]

In the past few years Axiom development has been a process where all
code changes happened on my local machine. I interacted with people
on a 1-on-1 basis, applied new code, managed local "broken" development
branches, and built new features. Once these features were working they
were tested on all the platforms I had available to me in my local
compile farm in my basement.

Clearly this process has flaws. The main problem is that it does not
scale. A second problem is that all of the parallel changes are 
wrapped into one big development swamp on my local machine (long time
readers will recognize axiomgnu/new which still exists here).

So, in order to develop a process which scales I've been looking at
the way Linus Torvolds does kernel development to try to understand
how to coordinate many people doing development in parallel. The
paramount problem is how to release quality software that is stable
and well tested.

It's not obvious to anyone but there is a long path of "process" steps
I have been taking to try to make sure that Axion "just works". Now
that the process is more open we will all end up following that process.
So I feel the need to explain the "development model" so everyone is on
the same page. Feel free to give me feedback as this is a cooperative
effort.

=====================================================================
Remember that the goal is to release a quality piece of software that
is well tested, well documented, and "just works".
=====================================================================

At the present time there are about 1/2 dozen tasks that I've "exposed"
to the world (see http://arch.axiom-developer.org). I have a dozen more
in the local code swamp that need to be exposed. They will show up as
time and interest permits.

Suppose a developer shows up with a problem, say porting axiom to the
Itanium.

Step 1 of the process is that a new "arch branch" will get created.

(Likely the branch will be named axiom--itanium--1).  This branch will
allow any developer interested in the problem to be able to read and
modify the code at will. We can experiment with any changes we want in
the "itanium" branch".

Step 2 is the documentation phase. 

All changes that survive the struggle to get itanium to work need to be
properly documented. This is my own mania, I realize, but it's part
of the 30 year horizon nonsense. We're writing code that our children
will have to change. Thus all changes need explanations that will 
make sense in the future.

Step 3 is the testing phase. 

We have to get the system to build and run on a couple Itanium
platforms to ensure that it works somewhere other than on our desk.

At this point we have a completed "axiom--itanium--1" branch and
have solved the problem of getting axion to run on the Itanium....

-------------------------------------------------------------------

Step 4 is the integration phase.

The problem to be solved here is to merge the axiom--itanium--1
branch into the axiom--main--1 branch.

There are two issues; integration and compile-farm.

First, the axiom--main--1 branch has moved.  Other developers have
been integrating their code, possibly changing the same file we changed.
So we need to merge our code into the axiom--main--1 branch. Clearly
this could be done automatically but I do it by hand so I can check
each change. (Machines don't care about quality.)

Step 5 is the compile farm (issue two)

Second, once we've integrated our changes into the axiom--main--1
the build process for axiom--main--1 has to pass the "compile farm".
That is, we need to build axiom on all of the different platforms
and make sure that our new itanium hacks don't break the main build
on other platforms.

-------------------------------------------------------------------

Step 6 is the savannah CVS change

At this point we've changed, documented, and tested a big new feature
for axiom. Now we need to check in the changes to the savannah CVS.
The changes are now going out to the world. For a lot of reasons
this step is likely to happen months after starting step 1. 

The hard part is that developers are now going to see new features
appear in the branches (e.g. graphics appears to work) but it seems to
take forever to get it into the savannah CVS. I know this feeling as
I've had various pieces of axiom running long before the world knew.
People who knew things like graphics were working wanted it NOW but 
quality is much more important than speed when releasing features.


Step 7 is the binary release

Several of the platforms have binary-only releases. These need to 
be packaged from the CVS version so they match the published reality.
Thus all of these binaries need to be recompiled and tested.

Step 8 is the spreading-tarball problem.

Several sites keep old tarballs of source around. This is convenient
but problematic as I occasionally get bug reports from old tarballs
which have been fixed in the CVS.

====================================================================
Arch? ARCH?! We don't need arch!!!
====================================================================

Why arch? Why not CVS? I've been using CVS for the past few years and
now I've run into the same problem that the Linux kernel people
solved. The problem is that CVS is unable to handle "changesets" and
"branching" easily. EASILY. Arch can.


Changesets are a group of changes to several files that are all handled
as one big change. Thus if we make a fix that involves 10 files we can
apply and retract the changes all at one time. This is technically 
possible with CVS if you carefully tag every file before applying the
change. However there are thousands of files in axiom and mistakes
are very hard to correct. Arch allows this to occur as one command.
So CVS treats each file change as different and arch combines many
semantically-related changes into one "changeset".

Branching can also be done in CVS but Arch also handles this problem
in a much more refined way. Since there will be many parallel development
efforts the ability to branch easily is important. Arch just does it 
better. 


I realize that switching to Arch is a learning curve and none of us
need yet another learning curve. But consider the fact that the kernel
has switched fron CVS to BitKeeper, a proprietary piece of code for
exactly the same reason; to manage the complexity better you need
better tools. Worldwide group development is hard.

So individually it will be painful but process-wise it will be easier.
I'll provide cookbooks schemes to anyone who asks and also provide
copies on the http://arch.axiom-developer.org page to minimize the
learning and the pain.

I ask for your patience as we open up the development process.
There is no such thing as a simple job.

Tim