Re: [Axiom-developer] Literate documentation

[Ralf Hemmecke]

> > I know that the "clone" idea is powerful. Without clones LEO
> > is just a fancy way of showing the table of contents and
> > clicking there to get to the corresponding section. Plus,
> > LEO can extract code from the sections. Minus, there is no
> > real weave command.

> Oddly enough - at least from our point of view - this is a
> standard but apparently undeveloped part of Leo. Ed Ream, the
> developer of Leo, says that basically he was waiting for someone
> to come along (like us?) who really wanted to use both Leo and
> noweb together. He has invited us to prepare an example of how
> we think the interaction between Leo and noweb should work.

OK. But I am not convinced anymore that switching to LEO is such a big 
step forward. If the Axiom code ends up by something that looks like 
LeoPy.leo and this is understood as a literate program, then I would say 
I don't understand the code and the ideas behind it. LeoPy.leo is *not* 
a literate program, rather it is a program whose reading order is 
perhaps a bit different than the compilation order, but otherwise I miss 
 a lot that drives people to write understandable code. And 
understandable code is, to a great deal, independent of the underlying 
programming language.

In fact, a literate program should make it very easy to translate a 
program written in one language into another language.

Maybe that is not the only guiding sentence behind LP, but when one 
documents and thinks about that, then one probably adds more 
documentation explaining the tricks needed for the particular language 
and also the goal that this or that piece of code actually serves. In 
the end a literate program might survive even a change in the underlying 
programming language.

> I find that a little surprising because it would seem that Leo
> already has a large number of users. It seems strange to me
> that I was only able to find one posting by another user who
> seemed to have concerns similar to ours.
>
> Anyway I am thinking that we should start with something like
> this:
>
>   1) It should be possible to import a noweb file into and run
>      Leo's weave command to produce an output file identical
>      to that produced by applying noweave to the original file.
>
>   2) After adding the necessary @root attribute, it should be
>      possible to run Leo's tangle command to produce an output
>      file identical to that produced by applying notangle to
>      the original file.
>
>   3) After importing a noweb file into Leo, saving it in Leo
>      format and after opening it at a later time it should be
>      possible to edit the file in Leo and then export the file
>      to noweb. From the exported noweb file it should be
>      possible to generate the same weave and tange outputs
>      using noweb noweave and notangle.
>
> In other words, this would make Leo a kind of interactive
> replacement for noweb.

My way of connecting noweb with something like LEO are a bit different.
I don't want to see myself exporting from .leo to .nw then call 
noweave+latex to get .dvi. (The tangle part can probably be done from 
within LEO.) Rather I would like to have leo work directly on the .nw 
files. A .leo file or rather a kind of index file file would keep meta 
information about the outline. Let that have the extension .meta.

Forgetting for a moment that a .nw file contains LaTeX, then there are

<<blah>>=

lines that start a code chunk and

@

or

@ %def ...

lines that start a documentation chunk. If I am not completely wrong, 
that is enough information to split the file into nodes where each node 
contains the documentation that comes right before it. (Maybe there is 
need for a little thought, since there might be code chunks that 
_continue_ previous code chunks.)
If LEO reads a .meta file, then if finds there lots of references to .nw 
files together with a precomputed outline from the previous save time of 
the file. LEO checks that the outline is not in conflict with the .nw 
files and reads the nodes into its memory. Then work is done as usual in 
leo. Saving any modification, saves the .mete file and all the various 
.nw files. (If the outline changed, then that might result in reordering 
the chunks inside the .nw file.---Maybe there should be a special 
nwfile-outline.)

Now if the .meta file is lost or if someone makes drastic changes to .nw 
files without using LEO, then most of or even all the cloned information 
is lost. Well, that is sad, but one has just lost the views on the code 
from different perspectives.

Before I started to look into LEO, I thought something like this 
scenario can be achieved with LEO. That would probably make most of the 
current Axiom developers happy. Everyone can work on .nw (.pamphlet) 
files as before, and there are people who could maintain different 
perspectives on the code+doc base.

Basically, I see LEO as a help to organise the code from a higher level, 
but I don't want it to mix with the pamphlet/noweb structure.

> But then things get a little more complicated and interesting.
> Of course Leo has some features that go beyond that supported
> by the noweb file format

What exact features do you like most.

[snip]

> Because of Leo's untangle command, it is possible for programmers
> to edit generated source files and have Leo automatically update
> the associated Leo files from which they are (normally) generated.

I wonder if we should make "untangle" a desireable feature. It basically 
means that you update code and forget about the documentation.
I tend to say, that hasn't my support.

My experience with developing code for Aldor-Combinat with the help of 
ALLPROSE is that I never felt the need to edit the generated .as files 
directly. One can easily find oneself around in the corresponding .as.nw 
file. Having need to look at code files directly just says that you 
haven't changed your view to programming.

With LP-aware debuggers I don't think that the .as or any other 
generated code file is anything else than some generated thing. You can 
generate any additional information into such a file to help a debugger 
or any other tool to find its way back into the actual .as.nw source, 
but I don't want to look at .as files. That is as uninteresting (and 
only rarely needed) as the .c file that can be generated from Aldor sources.

> Maybe this would be the best of both worlds.

Bill, can you state a few items why exactly you think LEO would help Axiom?

[snip]

> I think my suggestion above is a possible answer to that
> question - it would mean that you just go ahead and use the
> usual dvi - LaTeX file linkage to edit the Leo-generated LaTeX
> file (being careful not to distrub any of the weave-anchor
> comments in the LaTeX code that where inserted by Leo). Then
> you would expect Leo just to unweave these changes back into
> the associated Leo source file the next time you openit in
> Leo.

But I don't want my texts be stored inside .nw and inside .leo. There 
will be a point when nobody knows anymore what the actual sources are.

> > > In particular this was a follow-up to comments from
> > > Waldek, Gaby, and me that were critical of the creation
> > > of 'bookvol5.pamphlet' as literate documentation of the
> > > Axiom interpreter. We all expressed the opinion that
> > > 'bookvol5' was more difficult to appreciate even than
> > > the original code.
> > Sorry, I haven't even looked at bookvol5. But if it is
> > recognised as worse than the actual pure code, then this
> > is not a problem of whether using LEO or not, but rather a
> > problem that there is no general guideline of how to write
> > good literate programs, i.e. easily understandable
> > documentation.
>
> Yes, perhaps. But maybe the issue is deeper. I think Waldek
> said it best in a recent email when he coined the new verb:
> to "chunkify". He used this is a somewhat derogatory manner,
> suggesting that to re-arrange code into chunks in order to
> produce a pamphlet (noweb) style documentation of the Shoe
> (new boot) compiler would be very undesirable. Clearly he
> would prefer (at least in this case) that the code and the
> documentation remain separate. I am sure that this view is
> drive at least in part by reluctance to be reading and editing
> a cumbersome and large pamphlet file every time one wants to
> make a small change to the source code. Maybe Leo's untangle
> and unweave features would help prevent this sort of fear of
> using this literate format.

I can understand Waldek. Unfortunately, I have never read code in 
wh-sandbox, so I actually cannot judge. But suppose Waldek likes having 
documentation here and code there. That does not mean that in the future 
somebody else could rearrange that to make the code even more 
understandable. Everyone has his coding style and for Axiom it is 
currently probably the best to let everyone work as they please. We need 
something working, eventually. And if Waldek does this and puts the 
documentation different from where I would put it, I don't care. I 
cannot do Waldek's work so I am rather satisfied that Waldek writes 
documentation at all even if it lived in another place.

All my LP efforts currently mainly concern the Algebra code.

> > My effort in documenting my own code is probably not really
> > literal programming. The best thing I've seen up to now is
> > Cliff's description of cl-web.

> I agree that would Cliff wrote is a good example. One trouble
> (and this has always been a problem with literate programming)
> is that most programmers are poor or even terrible technical
> writers.

Hmmm, I really think that this is very much due to the fact that LP is 
not normally thought in university courses. Everyone wants to have code 
quickly. Nobody pays for good documentation. If a customer gets a 
sparsely documented product, one can get even more money out of that 
customer since somebody has to be paid for the maintenance (which might 
even be another company).

> Thus by it's very nature I think literate programming
> has to normally be a collaborative effort between the highly
> technical and specialized programmers and other people who are
> willing to contribute to the end result via their writing skills.

Let's see how this works out in our Axiom experiment. ;-)

> > I think one of my next tasks is to investigate what TeXmacs
> > misses.

> I think the biggest thing that TeXmacs lacks from out point
> of view is some form of literate programming or interface to
> noweb.
>
> Have you already looked at Lyx?

Oh thanks for that hint and yet another evaluation work. ;-)

Ciao
Ralf