[Axiom-developer] Re: Literate programming and how it scales

[Tim Daly]

(actual text is snipped out since this was not public but I want to record
the reply on axiom-developer)

...[snip]...comments about scale...

What kind of scaling are you thinking about? Axiom currently has about 
20 books
that will eventually have all of the source code in literate form. Some 
of the books
are several thousand pages long. There is hyperlinking within the files. 
Axiom is
about 1 million "things of code" and it will all reside in literate 
files so I think the
technology scales to really large systems.

If you are talking about scaling in terms of humans I think the problem 
is not one of
technology. The problem is that you really can't do a literate work with 
dozens of
authors. Literature requires one, or at most a few, authors in order to 
make sure that
the "style" and "voice" of the documentation is coherent. So I think 
that many people
can work on the system but only a very few are the "editors/primary 
authors". I don't
think there is a good label for this historically, except perhaps the 
"Chief Programmer"
from Harlan Mills "Chief Programmer Teams". I don't care for that label.

Ultimately literate programming requires a complete understanding of the 
system.
The point of literate programming is to convey this understanding to the 
reader
in a smooth, consistent way. Since programming is involved it is also 
important
that all of the details are explained, but also important that they are 
introduced
in some "motivated" fashion. It is the "motivation" for code snippets 
that is usually
left out of other documentation forms (e.g. doxygen, javadoc, etc.).

...[snip]...comments about TeX and multiple languages

I see TeX as a major strength since most technical papers are written in 
Latex.
TeX is also a strength for Axiom because it handles mathematical typesetting
very well and Axiom is a mathematical system.

Every literate system will have at least 2 languages, one for 
programming and one
for writing. I find that TeX does everything I need or can be extended 
with macros
to do what I need. I don't know how to add macros to an HTML system (without
writing in Perl as you do, which is yet a third language)

... [snip] ...comments about noweb

Axiom is moving away from noweb, although we have used it for years and 
I can
highly recommend it. Some of Axiom's literate files now have "straight 
latex" macros
for chunks so where you would write
<<noweb chunk name>>=
  your code
@

now Axiom allows you to write

\begin{chunk}{Axiom chunk name}
 your code
\end{chunk}

This uses straight latex macros which means that your file is pure Latex.
Since the file is pure Latex you no longer need "weave" so half of noweb 
is gone.

The only other remaining function is "tangle". Axiom is being rewritten
to understand Latex files directly so it can "tangle" the file without 
using noweb.
The underlying lisp system (GCL) has been modified to include a "tangle"
function directly so Axiom is not needed to extract code.

Thus we will no longer need noweb in the future.

...[snip]...comments about new tool features for code organization

Programmers have been conditioned to believe that they should work on small
files with "include files" to carry information that is shared.

If you look where this idea comes from historically you'll find there 
was a memory
limitation on the PDP 11/40 (I worked on one so I know) that limited a 
file to
about 4K characters. In order to make larger files you needed to invent 
"include files".
So C programs started the idea that files should be really small.

Then you needed to invent directories since groups of files in a 
directory made
up a single larger idea.

Then you needed to invent "make" because it becomes impossible to figure 
out
how to compile all the files in all the directories and get all the 
parts right.

Then you needed to invent "grep" to find functions.

Then you needed to invent IDEs so you could work with the whole project.


Today I have 4G of memory and a 1T of disk space. I can trivially handle
multi-megabyte files, my functions are all in one file, "make" is 
trivial, "grep"
is an editor "find" command and I don't need an IDE. Latex provides an
automatic cross-reference mechanism that allows me to find all of the places
where a function is called.

Once we stop pretending we work on a PDP 11/40 life gets so much easier

...[snip]... tool description using HTML, Perl, Javascript

So instead of Latex and Lisp, I now need Lisp, HTML, Perl,
and Javascript?

Is adding new languages an improvement? Are HTML files "documents", i.e.,
are they designed to be read as books? Can you control the layout and fonts
that are so necessary to convey certain information?

...[snip]... features of the new tool

Like any author, I keep my ongoing work in the manuscript, which is the 
literate file itself.
There are no auxillary files, no project files, no directory structure, 
no includes, nothing.

...[snip]...

I'm sure you have a good tool set that most people would find useful.
Unfortunately, the Axiom project is working to "remove the tools" that
are not absolutely required by making Axiom aware of its own file format
(pamphlet, also known as .tex, files)

Tim