[Axiom-developer] Re: noweb, pamphlets, and TeXmacs

[Joris van der Hoeven]

Thanks for the detailed tutorial, Tim.

> Knuth and Dijkstra advocated literate programming to try to solve
> the problem of marrying the knowledge behind a program with the 
> text of the program itself. Knuth wrote Web which was designed to
> work with Pascal thus:
> 
>     .web formatted document
>         |   |
>         |   ------> tangle ----> pascal code ----> compile ---> execute
>         ----------> weave  ----> tex format  ----> latex   ---> read
> 
> As this was Pascal-specific various other language-specific versions
> were generated, e.g. CWeb for C.
> 
> Norman's innovation is that we don't need to be language specific.
> With just a few additional tags above TeX we gain great power.

I also would like to stress that 1) there is no reason to stick to
only two possible ways of extraction (tangle and weave) and
2) that the language independence might also hold for weave.

> In addition, TeXmacs is able to properly format a large subset of 
> Tex and Latex documents.

Notice that TeXmacs is not just capable to properly format a large
subset of TeX/LaTeX, but that it also provides markup which is not
available in TeX/LaTeX (hyperlinks, actions, simple folding, etc.).
What is more, like in TeX/LaTeX, you may define your own markup,
and, like in Emacs, construct special editing modes for editing
such markup. This may apply in particular to things like customized
ways of folding, direct execution of an Axiom example from
the documentation, etc.

> 0) The ability to recognize and format a code chunk.
> 1) The ability to recognize the <<defn>>=, concatenation, and <<use>>
>    features of the code chunks.

All you have to do is to add a special tag like "chunk"
with one or more arguments. A preprocessor might then rewrite
your <<defn>>= syntax into more standard LaTeX like
\begin{chunk}{...} ... \end{chunk} and then use
the standard input filter.

However, as I pointed out before, it might be better to directly
use the TeXmacs format with additional tags and write the operations
like tangle and weave in Scheme.

> 2) The ability to create a "notangled" buffer from the current buffer
>    that would contain the formatted code.
> 
>      Ideally you could make changes in the formatted code and have the
>      changes reflected back into the original buffer. Some of these
>      changes could be problematic.

I rather see this as having several possible ways of looking at
pamphlet documents. Also, David Allouche will be working next year on
a way to perform operations like "tangle" and "weave" dynamically during
the editing phase. This presupposes that "tangle" and "weave" become
native operations in TeXmacs, comparible to (but not an instance of)
the application of an XML style.

> 3) The ability to create a "noweave" buffer from the current buffer
>    that would contain the formatted document.
> 
>      The same comment as above applies. It would take some careful
>      design to properly "untangle" some changes.

Yes, as I said before, there is no reason to distinguish between tangle
and weave and no reason to limit oneself to two operations.

> 4) Bill has suggested that the folding mechanism know about the code
>    chunks and be able to fold and unfold them. Perhaps the way to 
>    make the "untangle" work would be to ignore the separate buffer
>    idea above and just use folding. I have no opinion about either
>    path yet.

In a dynamic editor like TeXmacs, it should be noticed that
folding/unfolding becomes in fact a quite subtle operation.
In fact, one has to think about the kind of semantics that one requires.

For instance, do you see folding as an operations which makes a change
in the text, or rather like an operation which modifies the view of
a text? The first interpretation is easiest to implement.
In the second case, one has to define the notion of a "view"
(and in particular: what information determines a view?).
Also, you would have to think about how to undo changes.

Another difficulty has to do with TeXmacs' ability to let the user
define his/her own macros. What should be the semantics of
a macro which is built upon a "fold" tag?

> 5) There are other ideas, not yet exposed, that it would be nice to
>    have supported. I guess I need to talk more about the pamphlet 
>    idea in depth.

Well, it would be nice to make up a list of all tags (with options and
everything) which might be useful for pamphlet files. You might also
want to take a look at some (still very rudimentary) ideas in
the tmdoc format.

At the moment, I mainly see the "chunk" tag as a variant of
our "specific" tag, which is used in order to export content
in a specific way to specific formats. We basically have to
respecify the semantics of such a tag in a very general setting.

> First, you compose a set of Pamphlet files "across the system" so that
> you could document, say, all of the matrix facilities currently
> available. 
> 
> Second, you compose a set of Pamphlet files "thru the system" so that
> you could document, say, the integration mechanism from the top level
> function all the way to the implementation details.

My way of viewing this is the use of one file for each atomic
functionality of the system. Each such file should come with
meta-information like the bigger classes to which it belongs
(linear algebra, topology, etc.). Each file may also contain
information about other relevant related files or how to
traverse files in a sensible order (cf. tmdoc style).
One may then implement special functions into TeXmacs for
extracting the information one needs.

> Thus you can insert and extract Booklets with Axiom making it easier
> to share knowledge.
> 
> Future:
> 
>  Linear Algebra Booklet
>  |   
>  |-> NullSpace.pamphlet 
>  |   |   |   |
>  |   |   |   -> notangle -> nullspace.spad -> 
>  |   |   -----> noweave  -> nullspace.tex  -> latex -> read
>  |   |--------> dereference -> load and use other code.spad ->
>  |   |--------> userdocs    -> update Axioms user documentation 
>  |   |--------> testcase    -> run test cases
>  |   |--------> examples    -> input files
>  |   |--------> textbook    -> update Axiom's current textbook
>  |   |--------> proofs      -> ACL2, MetaPRL files
>  |-> Pivots.pamphlet
>     ......

Absolutely.

> Huge dream, I realize, but except for the dishes, I see no technical
> reason why it can't be done.
>
> This implies, of course, that Pamphlets can be decomposed into a
> finer level of detail which is still under development. 

Well, it all boils down to inventing suitable markup tags which
reflect the complexity of the problem. So please give this matter
a thought and come up with some more precise proposals.
After that, we will have a discussion and see how to implement
all this in TeXmacs.

> Currently noweb needs to expand the chunk definition syntax
> to handle some more general scheme such as a URL. We need to
> be able to extract code chunks from other pamphlets so that
> you can have the following situation:
> 
> pamphlet A:  (the definition document)
>     ...
>     <<foo>>=
>     ...
> 
> pamphlet B:  (the using document)
>     ...
>     <<pamphlet:/path/A#foo>>
>     ...
> 
> It would be useful if this could happen for text blocks also
> so that generally useful descriptions could be inserted into
> multiple pamphlets. Since the text blocks currently have
> no label this becomes problematic. We need to develop text
> labels so we can follow a uniform scheme. Multiple text blocks
> containing essentially the same information already exist in
> the system. This needs to be fixed.
> 
> For larger references (e.g. whole pamphlets) I'm currently
> using the bibliography environment. However, I plan to have
> a new Latex tag, say PAMPHLETREFS, that have a bibtex-like
> reference set. Tags in this environment point to other
> pamphlet files. Perhaps the "URL syntax" proposed above 
> could use the \PCITE{} tag instead:
> 
> pamphlet A:  (the definition document)
>     ...
>     <<foo>>=
>     ...
> 
> pamphlet B:  (the using document)
>     ...
>     <<\pcite{3}{foo}>>
>     ...
>
> Anybody who understands bibtex and would like
> to take a shot at this is welcome.

As I see it now, you have the following ingredients for chunck tags:

1) A "format" which determines the program which should be used 
   in order to do extractions. More generally, it reflects the purpose
   of the enclosed content (documentation, code, example, etc.).

2) An "identifier" for specifying how several chunks should be grouped
   together. In fact, it might be nice to see this independently from
   the chunk tags, but rather as a variant of labels and references.
   In other words, logical grouping of different parts of content
   may be nice in other circumstances too.

So maybe we should see the chunk tag as the combination of
two more basic tags: one for specifying the purpose, functionality
or class of a given region of text, and one for grouping scattered
pieces of information.

> RE: TEXMACS CHANGES
> 
> Currently TeXmacs could take the following steps, probably as
> a joint effort, to support Axiom:
> 
> 1) Recognize noweb format

And we should decide whether we want to keep on working with this format,
or whether we want to switch to a format for which it will be easier to
add new features.

> 2) Integrate commands to notangle and noweave.

This should be straightforward to write in Scheme once that one has
the appropriate tags.

> 3) Possibly either support
>    a) folding out code

Please think more precisely about the semantics that
you would like to have.

>    b) notangle, noweave to "dependent" buffers

Cf. improving the markup for inclusions like in the tmdoc style.

>    c) backport changes to "dependent" buffers to the original document

Dynamic editing of depending buffers is very likely to require
the "dynamic rewriters" that David Allouche will implement next year;
this will probably take us a year of work. 

I also remind you that TeXmacs does not have a one-to-one mapping with
TeX/LaTeX. In other words, you can not longer use TeX/LaTeX as a reliable
format for explanatory text in this scheme. If you want to edit
the original document using TeXmacs, it is better to either use TeXmacs
as the document format for your pamphlet files, or at least to replace
chunks of LaTeX by chunks of TeXmacs.

> 4) Integrate noweb.sty
>      Eventually this will evolve into Axiom.sty as we need to add
>      more latex macros, like \begin{theorem}, \begin{userdoc}, 
>      \begin{pamphletrefs}, etc

You should write a noweb.ts TeXmacs style file for this.
Unfortunately, writing style files is not well documented yet,
but you can ask questions on the texmacs-dev list.