[Texmacs-dev] Roadmap for the reorganization of the Guile/Scheme interface

[Joris van der Hoeven]

My ideas about the reorganization of the Guile/Scheme interface
get more and more precise. I will probably start the reorganization
during autumn, so this leaves you some time to think about it too and
let me know wishes/suggestions/etc.

1. Generalities about Scheme
----------------------------

First of all, I would like to recall that one of the main reasons
why scheme is so useful for us is the fact that it is so flexible
that we can in fact "design our own language" inside scheme.
Although we do not really need a new language, we do need primitives
which naturally suit our needs, like setting up dynamic menus,
associating properties to functions and extracting documentation
from the source code.

So I rather think of Scheme as some kind of skeleton which allows
us to design something well-suited for TeXmacs than a precise
language with several standard libraries. A corollary of this
attitude is that I prefer only to use some very elementary routines
from Guile/Scheme and none of the additional functionality
which is provided. Rather, we will define ourselves part of
this functionality if we really need it.

The advantage of this strategy is that it makes our code independent from
the particular implementation of Scheme. I already noticed that *each*
time that I relied on something more specific from Guile for some reason,
this gave rise to bugs, which were sometimes very hard to debug.
I estimate that at least 25% of the debugging time of TeXmacs was spent
on issues related to Guile. In other words, the Scheme code inside TeXmacs
should not contain much else as very standard primitives.

A disadvantage of Scheme is that it is only weakly typed.
Because we want to be able to use other extension languages in the future,
it is important that additional type information is specified by
the programmer whenever possible. This will also make it possible to
automatically convert the Scheme code into other compiled languages
in the future.

Another disadvantage of Scheme maybe its (((())))-notation. Nevertheless,
it should be stressed that this notation is very well suited for data 
structures and that much of the Guile/Scheme code will be data-structure-like.
The (((()))) notation is also very well suited for extensions of
the language with new primitives: we do not have to care about grammars.

2. About the general architecture
---------------------------------

Since many of the future extensions of TeXmacs will be written in Scheme,
it is important that the collection of all code remains as readable
as possible. Since Scheme is only weakly typed, the choice of the names of
functions will therefore be very important, so we need something very
systematic for this. Also, we want to encourage modular design and
of course this is intimately related with naming conventions.

The problem with modular design is that there are two main types of
extensions: new dtd's and new functionality. I think that new dtd's
will be the most frequent type of extension and that most new functionality
will in fact be subordinate to such new dtd's. This means that
the new structure of the progs directory should be something like

progs
* boot
* cas
* convert
* edit
  * dtd1
    * edit-dtd1
    * kbd-dtd1
    * menu-dtd1
  * dtd2
    * edit-dtd2
    * kbd-dtd2
    * menu-dtd2
  etc.
* fonts
etc.

The names of routines should be similar: dtd-markup-functionality or
dtd-markupclass-functionality and simply tag-functionality or
tagclass-functionality in the case of the "standard" dtd.
For instance,

  (fraction-insert) or (math-fraction-insert) instead of (make-fraction)
  (thm-insert "proposition") for inserting a proposition,
  (docbook-emphasize-insert) for the particular docbook dtd.

Of course, certain functions are not related to any particular dtd.
In that case, we should still try to use a prefix for classifying
the function. As a general rule, it should be easy to find
the definition of a function from its sole name.

This architecture is most natural for people who want to adapt TeXmacs
to a particular dtd-related need. However, it is different from
the current architecture which puts the emphasis on functionality and
it will become harder to *a posteriori* insert additional functionality.

3. Some elementary logical programming
--------------------------------------

The fact that it is so hard to foresee which kind of functionality might
be added later on and so hard to design an *a priori* architecture which
would be fit for all future pruposes, one main novelty which will be
introduced in the new Guile/Scheme interface is some "elementary
logical programming".

The idea is to attach logical properties to both data structures and
functions, as well as logical implications for such properties.
These properties provide powerful "meta-information", which may
for instance be reused when we want to add a new functionality
without knowing the particular dtd to which it applies.

For example, we might attach a property 'theorem-like' to
each of the markup primitives 'theorem', 'proposition', 'lemma', etc.
One might automatically deduce from 'theorem-like' that each of
the primitives 'theorem', 'proposition', etc. have arity 1.
All existing properties of the standard markup, as found in tree.cc
should become properties of this new kind.

Another example: we may associate a property like 'checkmark' to
functions. Such a propery may be reused automatically to add
checkmarks to the corresponding menu entries. More generally,
I would like to replace all function declarations like

(define (f args)
  body)

by something like

(tm-define (f args)
  properties
  .
  body)

The properties should in particular contain type information,
a short description, a short description for each of the arguments
(this makes it also possible to automatically generate documentation,
but also dialogue boxes or interactive input on the footer),
category information (like saying that a given function belongs
to a certain group of functions; this may for instance be reused
when generating automatic documentation), checkmark information
(for use in menus), potential restrictions on when the function
is valid (can be used for grey menu items, but also for automatically
determine the predicates for keyboard shortcuts, etc.).

4. Further notes
----------------

a) We need a precise rule when to use the ! suffix.
   For instance, we do not want to write (fraction-insert!).
   Probably a good rule is to use the ! suffix only if
   one of the *arguments* of the function gets modified.
   So (set! var val) but (fraction-insert) or (window-toggle-footer).
   I am not sure about (window-set show-footer #t) though.

b) We need a comprehensive set of useful extensions to Scheme with
   very short function names, like 'rcons', 'display*', 'cons*',
   'map*' (sequential variant of map), etc.

c) We need a comprehensive set of basic editing routines.
   This set clearly comprises 'assign', 'insert', 'remove', etc.,
   but we might also need routines for manipulating paragraphs and such.

d) We have to decide how to specify where the routines apply.
   on the current edit-tree at the current cursor position,
   on the current edit-tree at another position, for another document, etc.
   By default, the routines should probably apply at the current cursor
   position in the current window. We may provide primitives
   'with-at', 'with-buffer', 'with-window', etc. to do operations
   elsewhere and variants 'fraction-insert-at' of routines like
   'fraction-insert' if such a variant is used particularly often.

e) The menus should probably not be declared as variables,
   but rather be regrouped in a hashmap. So we woould need
   a routine like (set-menu which list-of-entries).