[Axiom-developer] literate programming pamphlet files for MathAction

[Tim Daly]

Ralf,

re: Why should ++ comments go away?

There were actually 3 distinguished levels: + ++ and +++, an idea
lifted from lisp's use of ; ;; and ;;; comments.  The idea was a
poor-man's way of doing \section, \subsection, and \subsubsection kind
of commenting with some cheesy syntax. Later we added keywords, special
tags (like \axiom) and function signature comments.

The problems are apparent and, as one of the original developers and 
the primary author of most of the bad signature comments, I can see
the flaws. I'll contrast them with the pamphlet mechanism.

First, ++ comments are a cheap imitation of tex. You can include certain
macros like \axiom in them and certain magic post processing occurs.
What this set of magic tags are, why they are there, and how they affect
the presentation later is not clearly documented. There is a limited set
and, unless you wrote the original mechanism (i.e. are me) it is very
hard to extend. 

Pamphlet files let you use the full power of tex, including hyperlinking
packages and, now noweb and a browser. Thus we gain 4 benefits. 
 (a) the world understands tex so no documentation of the tags is needed, 
 (b) the mechanism is entirely general so anyone can make their own tags, 
 (c) less code needs to be maintained, and 
 (d) it all works with general purpose tools (mathaction, browser, etc).

Second, ++ comments live in files intended to be fed to a compiler. As a 
programmer I develop my own style of coding and, unless the compiler
enforces some syntax, I will do things my own way. In fact, to try to
overcome this problem you'll notice that the compiler actually complains
if either the ++ comments are improperly indented or they are missing.
I wrote the code that enforces that because I know programmers. But,
frankly, it still fails because the mindset is still to "make it work".
Thus programmers discovered that ++ comments can be complete nonsense
and the compiler will shut up. So I added a check for certain tags.
Same result. We shouldn't feed the compiler anything it can't use.
(With some cleverness we COULD define a function that could allow a 
program to reference its own pamphlet at runtime. This might be useful)

Pamphlet files change the emphasis from programming to writing a document.
You not only have to satisfy the compiler you also have to satisfy noweb
and tex (and l2h, etc). Thus, the emphasis is on creating a document. For
a programmer this is awkward. But pamphlet files are intended for doing
computational science which is a very specialized area of programming.
Once I got my old code back after 15 years I realized that unless we 
keep the theory with the code there is no hope of maintaining, modifying,
and extending these heavy science systems. Pamphlet files are intended
to make sure we can.

Third, you comment that "the ++ comments would greatly help programmers"
The pamphlet files are intended to satisfy what appear to be
a couple audiences, some of which are programmers who don't know the
science and some of which are scientists who don't program. I believe
that we are on the cusp of a new group of people that I refer to as
computational scientists. They are cross trained in both programming and
their particular science. Maintaining special, outdated, weak, and painful
syntax "for the programmers" is not helpful.

Pamphlet files need some "canonical examples". I have permission to use
a couple phd thesis documents, technical papers that have algorithm
descriptions, proofs, complexity analysis, etc, sections of books and
other document sources. The idea is to "set the standard" for what we
can expect from a full, professionally documented, scientifically sound,
document with execution semantics. This will evolve over time as we add
new functionality (a test harness, a semantic network, drag and drop
installation, etc). 

Fourth, you mention that sometimes a programmer just wants the documentation
of the arguments. There is no reason why we need special syntax for this
and, as mentioned above, the previous programmers did not provide this.
The Axiom compiler does extract this type information while compiling. See
)show Integer
for an example of the output. 

Pamphlet files allow tagging of arguments in such a way that we could
construct automatic hyperlinks for arguments. And the hyperlinks would
go to other pamphlet files so that not only the type of the argument is
apparent but the rational for the choice is available.

Fifth, you said "This is NOT true for the Aldor compiler". Currently the
Aldor compiler will process ++ comments and pass them along... to nothing.

Pamphlet files will work with Aldor code. There is no language restriction
on noweb files (which is why I chose noweb rather than, say, CWeb). If you
look at the Axiom sources virtually every file is a pamphlet file including
lisp, c, boot, makefiles, etc. Thus making a pamphlet file using Aldor is
already supported. And all of the magic that MathAction will support is
automatic. The Aldor compiler should not know about comments.

Sixth, you reference the javadoc-like documentation style
(http://www-sop.inria.fr/cafe/Manuel.Bronstein/libaldor/html/node18.html)

Pamphlet files can contain this information and it can be parsed out
either with noweb, tex, l2h, etc. However, I'd expect that the original
pages in the pamphlet .dvi file can not only contain this information but
format it in a similar way if you so desire with additional information.
Bill Page's MathAction approach also allows multiple people to contribute
a wide rand of expertise to the documentation (for instance, the web page
you reference has a typo which I could fix if it were on a wiki).

As for hierarchical information the src/algebra/Makefile.pamphlet contains
the current lattice of types. This information is waiting for someone to
exploit it. Bill and I have both looked at it in the past. Perhaps it will
get integrated into the MathAction site in some reasonable way.

This is a long winded way of saying that I believe literate documents
(if not exactly pamphlet files) are the way forward not just for Axiom
but for scientific systems in general. The whole Axiom library, while
in pamphlet files now, needs better documentation. The current
"canonical example", which still needs much more work, is
dhmatrix.spad.pamphlet.

I know it all seems like a lot of work but we are at the beginning of
the collision of computing with science. If we go back to the last
century and look at math proofs from that time they seem like just so
much handwaving compared to the standards of proofs today. In 30 years
lightly commented source code with ++ and -- will seem unprofessional.
Algorithms will be expected to have termination proofs, correctness
proofs, examples, test cases, complexity analysis, working source code
and a sound basis in theory in order to pass as professional
scientific work.

All of which is clearly just my opinion on the subject.  But at least
this will give you some idea why I believe ++ comments will die.

Tim