Re: [Axiom-developer] Literate documentation

[C Y]

--- Ralf Hemmecke <address@hidden> wrote:

> In fact, a literate program should make it very easy to translate a 
> program written in one language into another language.

YES.  My take on a literate program is that if the tasks it is
performing are explained well enough, the only question is how to
perform those tasks in the target language.  (Of course, some
explanation has to be language specific but the task should be clear.)
 
> Maybe that is not the only guiding sentence behind LP, but when one 
> documents and thinks about that, then one probably adds more 
> documentation explaining the tricks needed for the particular
> language and also the goal that this or that piece of code actually
> serves. In the end a literate program might survive even a change in
> the underlying programming language.

That is my hope.  My "guiding principles" for Literate Programming at
this stage can probably be expressed as:

1.  A Literate Program should document the background of a task, the
particulars of the task, and "tough spots" with implementing the task. 
Ideally, only the last item is language specific.

2.  A Literate Program should contain enough background material on a
topic to allow understanding and application of an idea without having
to refer to non-included sources.  (For example, key algorithms
published in journal articles should be explained in the pamphlet, on
the assumption that some readers may not have access to/be able to
afford said articles.)  This has the side benefit of forcing the author
to greater awareness/knowledge of the subject in question - writing a
"proper" literate program on a topic should be a demonstration of
knowledge sufficient to solve the problem in a reasonable fashion.  I
can't claim to be there yet, but that's the eventual goal.

More on the wishlist side of things:

3.  "Turtles all the way down." My dream would be to have Axiom built
on only literate programs, down possibly even to the hardware design. 
This is one of the reasons I prefer working in Lisp, as every language
added to the support stack (except perhaps those implemented inside
Lisp like SPAD) makes that goal more remote.  In some specific cases
(for example the ANSI lisp specification) it is not possible to
incorporate the literature and the code directly, and we must state
that we rely on a "working ANSI Lisp environment as defined in ANSI
standard *whatever-the-ref-is*".  I dislike this even in those cases,
but clarification of the dpANS3 documents is impractical at this time
and converting sbcl into a complete literate program  without them
would be a monumental project requiring much in-depth knowledge of
compilers and low level issues that I don't have :-(.  Maybe someday. 
For the 30 year horizon (maybe beyond even that length of time) I think
it is relevant, but for now - assume Lisp and work from there.

> I wonder if we should make "untangle" a desireable feature. It
> basically means that you update code and forget about the
> documentation. I tend to say, that hasn't my support.

I agree.  It might be possible to support the idea of editing the code
in code view and having that update appear in the pamphlet view, but I
would tend to be wary of it.  There may be some nifty tricks for
viewing a chunk in code context or pamphlet context which can be
implemented, and I think a sufficiently smart literate programming
editor would be able to maintain code formatting and syntax
highlighting from the tangled file point of view inside the chunks, but
changing the code outside of the pamphlet context is something to be
wary of.

In cases where programmers simply don't want to work with pamphlet
editing and update the code only, I would propose a "chunkified diff"
utility where the edited code file is diffed against the current tangle
output, and the diffs are propagated back to chunks for incorporation
in the pamphlet context by the pamphlet author.  Each chunk would have
the diff highlighted until the pamphlet author identifies the merge as
complete.

None of these features require the "outline" approach and should be
possible with normal pamphlet files, given the correct tool and editor
support.

> Having need to look at code files directly just says that you 
> haven't changed your view to programming.

Bingo.  My take on this is pamphlet files should be maintained as
pamphlets, with tools to help merge non-pamphlet based updates.  That
process can't be automatic - some minor changes may not need
documentation changes, but many (most?) probably will.

> With LP-aware debuggers I don't think that the .as or any other 
> generated code file is anything else than some generated thing. You
> can generate any additional information into such a file to help a
> debugger or any other tool to find its way back into the actual
> .as.nw source,

Right.
 
> but I don't want to look at .as files. That is as uninteresting (and 
> only rarely needed) as the .c file that can be generated from Aldor
> sources.

Off topic, but a question about Aldor - if you compile Aldor routines
into c and build that c code, can other Aldor routines compiled into
Lisp still interact with the compiled c based modules?  Does Aldor
enforce compatibility?

>> Yes, perhaps. But maybe the issue is deeper. I think Waldek
>> said it best in a recent email when he coined the new verb:
>> to "chunkify". He used this is a somewhat derogatory manner,
>> suggesting that to re-arrange code into chunks in order to
>> produce a pamphlet (noweb) style documentation of the Shoe
>> (new boot) compiler would be very undesirable. Clearly he
>> would prefer (at least in this case) that the code and the
>> documentation remain separate. I am sure that this view is
>> drive at least in part by reluctance to be reading and editing
>> a cumbersome and large pamphlet file every time one wants to
>> make a small change to the source code. Maybe Leo's untangle
>> and unweave features would help prevent this sort of fear of
>> using this literate format.

That's of course in the eye of the beholder.  I can understand that
view - Waldek is a skilled coder and to him it would feel like a lot of
overhead.  I think that's where less skilled but learning coders can be
a big help to Axiom.  Dr. Sit and I discussed this a bit at ECCAD -
someone who needs to learn the basics behind a program can take working
code, dissect it, research the background, and write a pamphlet.  In
the process they may clean up the code or introduce some new features,
but they are building off of very skilled work by someone who may not
feel the need for literate programming.  In essence, this is what
happened with cl-web - Waldek provided the original finite-state based
code, and I expanded it into a literate document while in the process
of learning.

Similarly, I think someone having very little knowledge of languages,
compilers, BNF notation, etc. could look at boot or spad and start from
the beginning.  People capable of making boot and spad work may not
feel any particular interest in writing an introduction to compiler
theory as the first chapter of the pamphlet, but someone who is
learning will need to find this out anyway and in the process can write
it up.  This a) forces them to learn the ideas well enough to express
them coherently and b) means the next person coming behind them will be
able to start with the pamphlet and do a lot less "grunt research work"
to get up to speed.

> I can understand Waldek. Unfortunately, I have never read code in 
> wh-sandbox, so I actually cannot judge. But suppose Waldek likes 
> having documentation here and code there. That does not mean that
> in the future somebody else could rearrange that to make the code
> even more understandable.

I think you mean "couldn't rearrange"?  If so, I agree.  Waldek is
doing very good and very important work, and even if someone comes in
to write a literate version they will be building off of his efforts. 
Studying the solution to a hard problem is usually simpler and less
time consuming than trying to solve it for the first time (or second or
third if the previous solutions were lost...)

> Everyone has his coding style and for Axiom it is currently probably
> the best to let everyone work as they please. We need
> something working, eventually. And if Waldek does this and puts the 
> documentation different from where I would put it, I don't care. I 
> cannot do Waldek's work so I am rather satisfied that Waldek writes 
> documentation at all even if it lived in another place.

Exactly.  Right now, most of Axiom is not literate except in the
technical sense of being inside a pamphlet file.  I have no particular
qualms about Waldek's work being done in a non-literate fashion.  My
take on it is that when we DO start to make the files literate for
posterity we probably want tools that allow non-pamphlet edits to be
intelligently and interactively merged back into the pamphlet.  When we
get there, understanding Waldek's updated code won't be significantly
harder than the current code, and may be easier.

Pretty much any work I do in a serious way for Axiom (the Emacs mode
and cl-web, so far) has to be literate because I am learning from a
basic level as I go.  This has the advantage that I know what questions
a non-skilled programmer will want to have answered, since they are the
ones I'm asking :-).

Alasdair McAndrew asked recently on the list about learning Axiom and
writing code for Axiom.  Since I'm in a similar position, I can supply
my take on it - you don't want to go in with the mindset of writing
code that "just works" but being a beginner is not necessarily a minus
and can in some situations be a plus.  Fortunately for beginners, Axiom
is currently in a state where many hard problems are solved but the
solutions need cleanup, documentation, etc.  Selecting a particular
problem and exploring it is a GREAT way to learn and will also
contribute to Axiom.  Beginner questions are sometimes the most useful,
since experts in a subject may be so accustomed to what they are doing
they forget the unstated axioms.

Nor does a solution always have to be the "final word" on solving a
particular problem.  Ideally, if there is more work to be done
(theoretical or practical) a pamphlet can indicate this and the next
person knows where to start.

As for Axiom being a workhorse - it isn't now, true.  That doesn't mean
it will always be a "specialists only" game.  Most "user level"
mathematics has its foundations in more rigorous theory, and being able
to supply user expected behaviors while retaining the rigor behind them
is a major future selling point for Axiom.  We are doing the ground
work now, but the future is bright.

Cheers,
CY


 
____________________________________________________________________________________
8:00? 8:25? 8:40? Find a flick in no time 
with the Yahoo! Search movie showtime shortcut.
http://tools.search.yahoo.com/shortcuts/#news