[Axiom-developer] RE: noweb "bug" (was: article "standard" header/footer)

[Page, Bill]

On Tuesday, December 13, 2005 8:17 PM Tim Daly (root) wrote:
> 
> re: noweb bug
> 
>    i don't agree with Norman. *I* consider it a bug.
>    especially so since Ralf fixed the same "feature" independently.
>    noweb does the wrong thing with certain non-chunks.
>    if the chunk isn't in the hashtable complain and continue.

Norman defines it this way: If a chunk isn't in the hashtable
treat it as empty text. That is the behavior that he expects
and presumably what some other users of noweb might also expect.
Your patch would change noweb's behavior for everyone.

Ralf did not patch noweb to achieve the effect he desired so I
don't think Ralf would classify this as a bug. It is just the
way noweb works by default.

> 
>    i diagnosed the bug, fixed it, and sent off a diff -Naur
>    patch. which is exactly the way open source is supposed to
>    work. Norman disagreed with my patch and rejected it. that's
>    his call. which is exactly the way open source is supposed
>    to work.

I agree.

>    but we still NEED the patch. Axiom won't build without it.

No. We don't need a patch, we just need to use noweb a little
differently, the way it was designed to be used. No patch is
needed.

>    so we get the UNMODIFIED sources, apply the patch, and
>    build.
>

This is **not** the way open source is supposed to work unless
you are prepared to possibly alienate the original developer
and take on the responsibility for maintaining a separate version
of noweb. This is called a fork.

There is a standard well-documented way to change noweb's behavior
and Norman showed us exactly how to do it.
 
> re: awk
> 
>    since i don't write awk (but you do) i find the awk
>    script unreadable.

I don't know much about awk. All I do is 'man awk' and read
a little. For example:

http://developer.apple.com/documentation/Darwin/Reference/ManPages/man1/
awk.1.html

>    it really ought to be documented. we had this discussion 
>    before when you wrote awk scripts into the makefile. awk
>    is linenoise to me and i hate having to learn a whole new
>    language to maintain 8 lines of code.

But you are developing on linux. awk and friends have been in
all unix and unix-like systems for nearly 30 years. As far as
I am concerned you have no excuses for not being able to read
it.

http://www.gnu.org/software/gawk/manual/html_node/History.html

> 
>    this may be an acceptable way to fix the "feature" but i'm
>    unclear when and how to apply awk/gawk/nawk scripts to
>    noweb. i looked at the filtering thing but since it is all
>    in awk/nawk/gawk it made no sense.

That is not true. The -filter option is very clearly documented
in detail by it's author. It has nothing to do with awk. The
filter could be written in the C programming language if one
wanted. It is just a filter.

> if it was so clear why didn't Ralf apply Norman's awk code?

Probably because he did not know about it. It was buried under
more than 3 years of discussion about other things.

>   
>    does this work with awk/nawk/gawk? all three?

Norman's awk script is vanilla awk. I would say "all three".

>    already i have to customize the build scripts because of
>    this trivial feature. it's the ONLY shell variable besides
>    AXIOM that needs to be set before the system build begins.

Why do you need a shell variable?

> bug #1 on MACOSX is the lack of nawk/gawk.
>

You must be kidding. See:

http://developer.apple.com/internet/opensource/opensourcescripting.html

and

http://developer.apple.com/documentation/MacOSX/Conceptual/OSX_Technolog
y_Overview/Tools/chapter_10_section_9.html

awk is the first tool on the list.

> re: gcl
> 
>    if it helps, think of it as automating a cvs get for the 
>    version we want. we get the UNMODIFIED sources and apply
>    axiom-specific patches. we'll still need a patch mechanism
>    otherwise the browser, sman, and graphics are broken.

That is not true. Camm does not patch gcl sources in order
to build Axiom on Debian.

>    yet the axiom lisp patches will *never* be part
>    of EVERY lisp distribution, not even part of GCL.

There is no need for such patches.

> 
>    an alternative is to modify GCL to support UFFI (Universal
>    Foreign Function Interface) and modify Axiom to use it.
>    I looked at this path and it is a lot of work. When Axiom
>    goes ANSI it'll have to be done but that's still a long way
>    in the future and GCL may have UFFI by then. Only a few other
>    common lisps support UFFI so far.

UFFI is not required in order to build Axiom.

> 
> re: forking
> 
>    you seem to feel that maintaining axiom-specific patches 
>    to standard distributions (we use the standard tgz files as
>    a base) is equivalent to forking a distribution.

Yes I do. See:

http://en.wikipedia.org/wiki/Forking

"In software engineering, a project fork or branch happens when
a developer (or a group of them) takes code from a project and
starts to develop independently of the rest."

>    my definition of forking differs. in my view we are not
>    forking.

Humpty Dumpty: When I use a word, it means just what I choose it
               to mean - neither more nor less.
Alice: The question is, whether you can make words mean so many
       different things.
Humpty Dumpty: The question is: which is to be master - that's all. 

>    we are fixing axiom-specific bugs. we don't claim to distribute
>    common lisp or GCL or noweb or anything but Axiom. we don't
>    even claim that OUR version of noweb works for anyone else.
>    we don't distribute a patched version on websites, we don't
>    advertise that ours is better/faster/sexier. we're not trying
> to compete.

Forking has nothing to do with competing. It is a result of the
breakdown of a collaborative effort. Developers who want to do
their own thing without regard to cooperation with others.

> 
>    GCL and noweb sources in axiom are EXACTLY as distributed
>    and could (in a more perfect world) be fetched from CVS
>    automatically. that's hardly a fork. GCL and noweb need
>    source-level patches to work.

That is not true. Axiom is built on Debain without patches
to gcl or noweb.

> 
> re: advocacy
> 
>   ok. you're advocating a change.

No, I am just complaining that you did something wrong 3 years
ago. It was Ralf who was advocating a change. I just agreed
with him. ;) The only change that I am advocating is that we
do everything we can to attract more Axiom developers.

>   that's good. but advocacy is volunteering
>      - check out a --patch-46 copy of the system, 
>      - make the changes, 
>      - document the changes,

The documentation would go something like this.

In noweb when the syntax <<xxx>> appears in the code, it
is supposed to be interpreted as the substitution (inclusion)
of the contents of the chunk named xxx at that point. The
problem that we have in Axiom is that sometimes the appearance
of <<xxx>> in the source code should not be interpreted as the
use of a chunk name but should be interpreted literally as
'<<xxx>>'. If <<xxx>>= does not occur as the actual name of
a chunk, then by default noweb simply omits the text '<<xxx>>'.

It is possible to add escapes to the source code so that <<xxx>>
is interpreted literally. But since it is unlikely that <<xxx>>
will appear as the name of a chunk, another possibility is to
change the default behavior of noweb so that it treats <<xxx>>
as if it was defined as

<<xxx>>='<<xxx>>'

The following noweb -filter accomplishes this.

As explained in Norman's noweb documentation, each occurrence
of <<xxx>> results in the internal code
  @use xxx
Each occurrence of <<xxx>>= results in
  @defn xxx
The complete definition of a chunk looks like this:

  @begin code
  @defn xxx
  @nl
  @text <<xxx>>
  @end code

The script works as follows:

1) Awk reads one line at one at a time.
2) If line begins "@use " then add rest of line as
   a key of the 'uses' array.
3) If line begins "@defn " then add rest of line as
   a key of the 'defns' array.
4) Output each line as it is.
5) At the end of the file, for each key in 'uses' that
   is not in 'defns' this script adds a definition of a
   chunk that is the same as the name of the chunk.

To modify the Axiom source distribution to use this method,
create a file named 'axiom-noweb' with the following
contents:
---------
#!/bin/sh
awk '
/@use /  { uses [substr($0, 6)] = 1 }
/@defn / { defns[substr($0, 7)] = 1 }
{ print }
END {
  for (i in uses)
    if (!defns[i])
      printf "@begin address@hidden address@hidden@text <<%s>>address@hidden 
code\n",
i,i
}'
exit 0
------

Make it executable by

 % chmod a+x axiom-noweb

and change every occurrence of 'tangle' in the Axiom make files
to 'tangle -filter axiom-noweb'.

>      - test them,
>      - test the whole system build,
>      - "diff -Naur old new" to create the patches
>      - mail the patches to me, one set for noweb,
>        one set for gcl. 
>      - i'll apply them and test them
>      - we'll discuss the results
>      - i'll add them to the main sources

If anyone wants to do this, I would be very glad to help them
do it. We need more Axiom developers. This is a simple change.
All of the technical work has already been done by someone else,
but it would be a good chance to learn how to submit a patch.

> 
>   sending documented, running, and debugged code that "just 
>   works" goes a long way toward winning the argument.
>   complaining that i'm doing it wrong and expecting me to do
>   the work isn't nearly as effective.

I don't expect you to do the work. This is an argument about
principles not pragmatics.

>   at best if i agree with you i'm likely to add it to an already
>   overlong queue. if i disagree with you and your code works it
>   is much more likely to be included (witness the awk script
>   already in the Makefile).
> 
> re: fixing working code
> 
>   the system build works. i can see hundreds of things in Axiom
>   that need fixing but GCL and noweb aren't on that list (at
>   least for me). basic functionality in Axiom is still broken
>   (like ")cd" or ")hd") and are MUCH higher on the todo list.
>

I agree. But what is broken about ")cd" and what is the meaning
of the command ")hd" ?
 
>   i'd much prefer that you try to tackle something like the
>   Hyperdoc browser since you're advocating redoing the front
>   end to use firefox. why not take the lead on Volume 8:
>   Hyperdoc? it needs to be documented and we need to understand
>   it so we can redesign/rewrite it. you have more ideas in that
>   direction than anyone else.
> 

I was really hoping what Kai Kaminski has already done would
encourage either him or someone else to take on that sort of
project. I am willing to continue to try to help anyone who
wants to do this.

Regards,
Bill Page.