Re: comments on new-model.txt [long]

[Lynn Winebarger]

Ok, I've had some time to think more, and so I want
to go back the beginning.  I've also included two
other emails I haven't yet responded to in chronological
order for completeness.

On Thursday 05 September 2002 14:06, Marius Vollmer wrote:
> Actually, I have tried to keep the module system out of the model as
> much as possible.  The interface between the module system and a
> hypthetical compiler is quite minimal.  The module system is very
> important for the overall behavior of the system, but my intention is
> to allow large changes to the module system without having to change
> the compiler.

     The problem is the very definition of what modules are.  That's what
I'm objecting to, and it is in there.  I can only assume this is in preservation
of the current module system, which is way too dynamic in nature.
"interactive-namespace" or "interactive-environment" would be a more
accurate term (as I've noted before).  Well, actually even that's not
quite accurate as an "interactive-namespace" would just be a type
of environment that might occur near the top of a lexical closure
- and that would be fixed for that closure (there would be no
changing the top-level of a particular closure in "interactive-namespace").

> The biggest requirement that the proposal puts on the module system is
> that modules can only ever grow; you can not remove bindings or change
> existing ones.  I expect this to be a popular point for debate, and I
> don't know if I can defend it successfully.  But the current model
> rests on this requirement.
      The biggest one is that there's a "current module" at evaluation
time and that it can be changed.  That's a _huge_ requirement.

> Option 'body' for (3) is harder to enforce in the module system, but
> it is easy to add in the macro expansion pass which is also not
> treated in detail in the proposal.
      But, as I understand it, this is one of the problems that has
been encountered in practice (the interaction between macro expansion
and the "module" system).

> >     There is no reason to require macros to always be macros.  R5RS does
> > not have any such requirement and there are plenty of compilers for it.
> 
> You can change a macro into a function, but you need to
> recompile/reload your code for it to take effect.  That's about all
> that the model implies.  Also, you can't redefine a name in a module,
> but I think that is reasonable.
      You see, I think this especially shows where my mental model of a
module differs from yours (and perhaps others).  A module should not
have anything to do with the current (interactive, or whatever) top-level
environment.  It's environment should have at its root its own top-level
environment (or not, if the compiler can determine it does not need one).
It might interact with the current environment by explicitly importing from
it (using some sort of reserved identifier for that purpose), and the 
linker/loader could decide (at run time) what top-level environment
to link it to (for example, if each thread has its own top-level environment
besides the thread-global top-level environment, the user might specify
linking to some specific thread, or might specify a fresh environment to use.
    So part of the miscommunication is that I've stopped thinking of
"top-level" (lexically open) environments necessarily being at the actual
top-level of a lexical environment.  It complicates define a little, but not
much.  

> >     To handle optimization, I believe we could adopt special forms
> > that import values and bind them to local (possibly immutable)
> > variables. [...]
> 
> This sounds more complicated than the 'declarations' feature of the
> proposal.  The declarations have the advantage that you don't need to
> do anything special when importing a name, the burdon is on the
> definer of the name.

     I almost always prefer code I'm using not to have some oddball
implicit semantics.  If it's going to do something weird, I should specify
that it does something weird.  Of course, you might provide special
forms to handle some of these things implicitly, but the core
mechanisms should not (and particularly, it should be the library
user's choice and not the library implementers about what is
done with the user's variables).  I object to global dynamic scoping
in general on the same grounds.
    I don't think it will hurt to reflect on this statement from R5RS either:

Following Algol, Scheme is a statically scoped programming language.
Each use of a variable is associated with a lexically apparent binding
of that variable.

   Now you may think I'm being pedantic, but I think discarding
that second sentence should at least give pause.

>>        If we have extlocs, this can be just a plain lambda that
>> happens to let import take references to some of its variables.
>> Then we can use the keyword "module" instead.
>
>Is this related to the 'module' feature of recent versions of Dybvig's
>syntax-case implementation?  If so, I think that it is implemented
>solely with renaming of identifiers and needs no support from the
>interpreter.  Also, I think that the syntax-case modules are less
>powerful than ours since their semantics are fixed while ours can be
>cleanly extended.

   To the extent that we're talking about lexically scoped closures
with exportable bindings, yes I suppose.  Though I'm not sure
what you mean by "cleanly extended".  Maybe you can give
an example of what you mean.
    Otherwise, see below.

>>     And then there's Dirk's view of modules as parameterizable to
>> get a handle on too.
>
> Lynn Winebarger <address@hidden> writes:
>
>> > What would you do with that pointer?
>> 
>>     Actually, I believe I was originally thinking of using it for macro
>> expansion purposes (so if a macro was bound to an external variable,
>> that top-level environment could be stacked on whatever the top-level
>> environment "currently" is for macro lookups.
>
>I plan to use the :module-ref 'assembly instruction' for this.  Macro
>expanders would use it to refer to specific top-levels aka modules.
>We need to be able to encode these references in object files, so we
>need to use the names of modules, not pointers to the run-time objects
>that implement modules.
           
    No.  There's no (good) reason for modules to have inherent names.   
There are good reasons for them to have fixed file names, but not
names that inhere in the module itself (any more than there is for functions, 
at least).  Indeed, if you have parameterized modules (that is, 1st there's
possibly a parameter of "current top-level enviroment to link to", and then
there are explicit imports (by the module) to be specified by the module
user, that will result in an instantiation of the module's data environment
that links to those.  Indeed, there's no particular reason to disallow modules
from being loaded multiple times with independent instantiations.  Like this;

(define a (load-module "foo"))
(define b (load-module "foo"))
(import a (prefix "a->"))
(import b (prefix "b->"))
(set! a->bar 5) 
(set! b->bar 7)
a->bar => 5
b->bar => 7

  So in this model, you could use modules you object with compiled methods
if you like.
   Heck, it might even be as simple as this to create a (trivial) module (in 
shell)
cat >"foo.ss" <<EOF
(import core)
(define a 5)
(export a)
EOF
cat >"compile-foo.ss" <<EOF
(compile-module "foo.ss" foo)
EOF
cat >"use-foo.ss" <<EOF
(load-module foo ((a -> b))
b => 5
(set! b 18)
b => 18
EOF

   If you can't tell, I think Guile should handle its own linking and loading, 
at least
as far as its own compiled modules go.      
   As regards syntax-case and macros/modules:  I don't believe Guile should use
Dybvig's portable Scheme implementation.  Even if it we had a compiler, it does
explicitly use symbols for renaming that somehow have to be unique across
compiler runs.  This may be necessary for a portable implementation, but I
think it's pretty undesirable.  The "renaming" doesn't have to involve
actual new names.  
   In my (unfinished) implementation, I represent syntax objects with
explicit lexical (here I mean compile-time lexical - no data) environments
and marks, and when an identifier is encountered, it gets resolved to a
frame in its lexical environment, and this frame is stored as part of the
syntax-object.  This is sufficient for comparing identifiers for free and bound 
equality.  In expanding macros, newly introduced identifiers are labeled with
the environment of the macro instantiation (actually, of the "syntax" form)
(which can result in what Dybvig calls "displaced lexicals" but I prefer to 
think 
of as a simple attempt at importing variables).   For those who don't know
what I'm talking about:
(define foo-macro 
   (lambda (x) 
     (syntax-case x ()
         ((_ y ...)
          (let ((z +))
             (syntax (z y ...)))))))
(let-syntax ((foo foo-macro))
   (foo 10 11)) => "Error: identifier out of context z"  (or a similar error 
message)

   Really, this is just an attempt to import z from the let inside of foo-macro
into the environment in which "foo" is expanded.
   Now, what should happen when a macro imported from the module is
used is the same:  newly introduced identifiers should get labeled by 
the module environment, and everything else should retain whatever
lexical environment it had.  This will avoid both capturing syntax from the
interactive top-level, and missing syntax from the module.  Since
macros are perfectly capable of relying on values of imported variables
(and side-effects on those and resulting dynamic macro behaviour), 
And macros can introduce macros they imported from yet another
module, it will all work (I think, anyway).
    That's how I think all this should work, anyhow.  It's a lot of work
to get there, though.
     I still haven't read Mattew Flatt's paper, maybe he's got something
better.
    Pshew!

Lynn