[Axiom-developer] RE: noweb, pamphlets, and TeXmacs

[Bill Page]

Everyone,

I am glad to see that this is such a "hot topic"!

BTW, I am sorry that some of you (like me) who are
members of multiple lists are likely receive
several copies of this message. It seems sort of
unavoidable for the moment while we are exploring
just how "interdisciplinary" this subject is. Perhaps
some we should absorb this subject back into the Axiom
mailing list since that is where is started. Those
people who are interested will have had a chance
to subscribe if they want to monitor this subject.

Anyway, I spent some time today thinking about alternate
mark-up languages for pamphlets. I suppose what I do
for a living could not help but intrude on my views
in this project... depressing perhaps, considering
that I would like to view this as an ambitious hobby
rather than "work" ... <frown>

Anyway, I do a lot of work with XML and XSL style sheets
these in the context of large scientific bibliographic
databases. Also, I have been reading about some ideas
expressed on the TeXmacs list about XML. A quick look
at the TeXmacs file format shows many very obvious
relationships to XML. In fact it seems that it could
almost be 1-1 except for a slightly different tag
syntax. There is an awful lot of work being done
these days on XML as the prime candidate "next
generation" structured document language. And it
turns out that "structured documents" have many
many uses. They are a kind of half-way point between
data bases record and free form text documents.

XSL (or more properly XSLT) is a structured document
transformation language that is itself in XML format.
The main purpose of XSLT these days is to generate
HTML to be displayed in a web browser. The Microsoft
browser and several others even include the XSLT
processor in the browser itself. And implementations
of XSLT processors exist for almost all server
environments. Besides HTML, other output formats are
possible, e.g. PDF format etc.

So the concept was this (expressed as a question):
How difficult would it be to write an XSLT style
sheet that would perform weave and tangle on an
XML file? I almost sat down to give it an initial
try when I thought about doing a web search on the
subject of "XML"

  http://xmlp.sourceforge.net/

and

  http://groups.yahoo.com/group/xml-litprog-l/

see also

  http://www.vivtek.com/litprog.html

Damn the web! It seems that someone somewhere has
always already thought of all the good ideas ...
<grin>.

> From: Joris van der Hoeven [mailto:address@hidden 
> Sent: Sunday, November 24, 2002 12:17 PM
> .. 
> Notice that TeXmacs is not just capable to properly
> format a large subset of TeX/LaTeX, but that it also
> provides mark-up which is not available in TeX/LaTeX
> (hyperlinks, actions, simple folding, etc.). What
> is more, like in TeX/LaTeX, you may define your own
> mark-up, and, like in Emacs, construct special editing
> modes for editing such mark-up. This may apply in
> particular to things like customized ways of folding,
> direct execution of an Axiom example from the
> documentation, etc.
> 
> > 0) The ability to recognize and format a code chunk.
> > 1) The ability to recognize the <<defn>>=,
> > concatenation, and <<use>> features of the code
> > chunks.
> 
> ... 
> However, as I pointed out before, it might be better
> to directly use the TeXmacs format with additional
> tags and write the operations like tangle and weave
> in Scheme.
> 
> ... as I said before, there is no reason to distinguish 
> between tangle and weave and no reason to limit oneself
> to two operations.
> 
> > 4) Bill has suggested that the folding mechanism
> > know about the code chunks and be able to fold and
> > unfold them. Perhaps the way to make the "untangle"
> > work would be to ignore the separate buffer idea
> > above and just use folding. I have no opinion about
> > either path yet.
> 
> In a dynamic editor like TeXmacs, it should be noticed
> that folding/unfolding becomes in fact a quite subtle
> operation. In fact, one has to think about the kind of
> semantics that one requires.
> 
> For instance, do you see folding as an operations
> which makes a change in the text, or rather like
> an operation which modifies the view of a text?
> The first interpretation is easiest to implement.
> In the second case, one has to define the notion
> of a "view" (and in particular: what information
> determines a view?). Also, you would have to
> think about how to undo changes.
>

I really like the idea that the code appears "folded"
into the document, but still part of it. When the folds
are collapsed, the result is just documentation. But
when the folds are expanded, they are visible and
editable just like any other part of the document. So
in that sense I am only talking about a "view" of
the document (with one, several, or all folds collapsed). 

I would definitely say that folding should not be
an operation that changes the text, however we it
might be desirable if it (optionally?) affected how
the documented was printed as well as displayed.

This sort of folding is now a common feature of
math packages like Maple and Mathematica. It helps
a lot to be able to read a document in a "top down"
fashion. In this sense the programming code is at
the "bottom" of the document fold structure.
 
> Another difficulty has to do with TeXmacs' ability
> to let the user define his/her own macros. What
> should be the semantics 
> of a macro which is built upon a "fold" tag?
> 
> > 5) There are other ideas, not yet exposed, that
> > it would be nice to have supported. I guess I
> > need to talk more about the pamphlet idea in depth.
> 
> Well, it would be nice to make up a list of all tags
> (with options and everything) which might be useful
> for pamphlet files. You might also want to take a
> look at some (still very rudimentary) ideas in the
> tmdoc format.
>

I think that Anthony Coates (see links above) has
already done quite an excellent job of defining a
simple set of tags and attributes. These could
probably be used almost directly as a model for
TeXmacs.
 
> At the moment, I mainly see the "chunk" tag as
> a variant of our "specific" tag, which is used in
> order to export content in a specific way to specific
> formats. We basically have to re-specify the semantics
> of such a tag in a very general setting.
> 
> > First, you compose a set of Pamphlet files "across
> > the system" so that you could document, say, all of
> > the matrix facilities currently available.
> > 
> > Second, you compose a set of Pamphlet files "thru
> > the system" so that you could document, say, the
> > integration mechanism from the top level function
> > all the way to the implementation details.
> 
> My way of viewing this is the use of one file for
> each atomic functionality of the system. Each such
> file should come with meta-information like the bigger
> classes to which it belongs (linear algebra, topology,
> etc.). Each file may also contain information about
> other relevant related files or how to traverse files
> in a sensible order (cf. tmdoc style). One may then
> implement special functions into TeXmacs for extracting 
> the information one needs.
> 
> > Thus you can insert and extract Booklets with Axiom
> > making it easier to share knowledge.
> > 
> ... 
> > Huge dream, I realize, but except for the dishes,
> > I see no technical reason why it can't be done.
> >
> > This implies, of course, that Pamphlets can be
> > decomposed into a finer level of detail which is
> > still under development.
> 
> Well, it all boils down to inventing suitable mark-up
> tags which reflect the complexity of the problem. So
> please give this matter a thought and come up with
> some more precise proposals. After that, we will have
> a discussion and see how to implement all this in
> TeXmacs.
> 
> > Currently noweb needs to expand the chunk definition
> > syntax to handle some more general scheme such as a
> > URL. We need to be able to extract code chunks from
> > other pamphlets so that you can have the following 
> > situation:
> > 
> > pamphlet A:  (the definition document)
> >     ...
> >     <<foo>>=
> >     ...
> > 
> > pamphlet B:  (the using document)
> >     ...
> >     <<pamphlet:/path/A#foo>>
> >     ...
> > 
> > It would be useful if this could happen for text
> > blocks also so that generally useful descriptions
> > could be inserted into multiple pamphlets. Since
> > the text blocks currently have no label this becomes
> > problematic. We need to develop text labels so we
> > can follow a uniform scheme. Multiple text blocks
> > containing essentially the same information already
> > exist in the system. This needs to be fixed.
> > 
> ... 
> As I see it now, you have the following ingredients
> for chunk tags:
> 
> 1) A "format" which determines the program which
> should be used in order to do extractions. More
> generally, it reflects the purpose of the enclosed
> content (documentation, code, example, etc.).
> 
> 2) An "identifier" for specifying how several chunks
> should be grouped together. In fact, it might be
> nice to see this independently from the chunk tags,
> but rather as a variant of labels and references.
> In other words, logical grouping of different parts
> of content may be nice in other circumstances too.
> 
> So maybe we should see the chunk tag as the
> combination of two more basic tags: one for
> specifying the purpose, functionality or class of
> a given region of text, and one for grouping
> scattered pieces of information.
>

I think you (Joris) may be surprised how close
your ideas are to Anthony's on this!
 
> > RE: TEXMACS CHANGES
> > 
> > Currently TeXmacs could take the following
> > steps, probably as a joint effort, to support
> > Axiom:
> > 
> > 1) Recognize noweb format
> 
> And we should decide whether we want to keep on
> working with this format, or whether we want to
> switch to a format for which it will be easier
> to add new features.
> 
> > 2) Integrate commands to notangle and noweave.
> 
> This should be straightforward to write in Scheme
> once that one has the appropriate tags.
> 
> > 3) Possibly either support
> >    a) folding out code
> 
> Please think more precisely about the semantics
> that you would like to have.
> 
> >    b) notangle, noweave to "dependent" buffers
> 
> Cf. improving the mark-up for inclusions like in
> the tmdoc style.
> 
> ... 
> I also remind you that TeXmacs does not have a
> one-to-one mapping with TeX/LaTeX. In other words,
> you can not longer use TeX/LaTeX as a reliable
> format for explanatory text in this scheme. If you
> want to edit the original document using TeXmacs,
> it is better to either use TeXmacs as the document 
> format for your pamphlet files, or at least to
> replace chunks of LaTeX by chunks of TeXmacs.
> 

Regards,
Bill Page.