[swarm-hackers] ObjC runtime abstraction

[Scott Christley]

Hello hackers,

If anybody gets CVS emails, you may have noticed that I've been  
committing some major changes to Swarm, albeit on a separate branch.   
I've been working on creating an abstraction interface for the ObjC  
runtime library, and having Swarm use that interface instead of  
directly manipulating the ObjC runtime.  The motivation for doing this  
work is my long-standing goal to get Swarm to work on Mac OSX using  
the Apple ObjC runtime.  With the release of OSX Leopard, Apple made a  
significant change to the ObjC runtime, dubbing it version 2.0.  One  
of the fundamental changes is that all of the objc associated data  
structures like for classes, methods and instance variables have made  
opaque; programs can no longer access the fields directly in the data  
structure.  Instead Apple provides functions for accessing this data.

As a result, the current Swarm code cannot be directly ported to Apple  
ObjC 2.0, so I proposed a re-architecture of Swarm so that it used a  
functional interface to the ObjC runtime.  This is work funded by  
SDG.  I'm posting this email to let everybody know about my progress  
with some design notes, what is left to be done, and any outstanding  
issues.

One of the core capabilities of Swarm is the create-phase protocol.   
While end-users likely do not use this functionality, it is  
fundamental to the internal organization of the Swarm code.  One of  
the design goals is to allow objects to customize themselves when they  
are being created.  The implementation has an object changing its  
class (going through phases), allowing it to be customized during each  
phase, then finalized with that final object being returned to the  
application.  This capability requires that new classes be dynamically  
created at runtime and for objects to change their class.

The old Swarm architecture implementation utilized a number of facts  
about the GNU ObjC runtime:

* objc class data structure is public.
* objc runtime has no formal functionality for creating a new class,  
all one does is allocate an objc class data structure, then populate  
it with appropriate values.
* some of the fields in the objc class data structure are under- 
utilized by the objc runtime, leaving them open for programs to use  
them for their own purpose.

Further more, Swarm uses an overlay technique with an ObjC class,  
specifically DefClass in defobj, so that the memory layout of the  
instance variables matches the objc class data structure, thus  
allowing instances of DefClass to correspond to new ObjC classes.

To re-architecture Swarm so that it uses a functional interface to the  
ObjC runtime library requires eliminating this overlay technique such  
that DefClass instances are separate from any new ObjC classes.   
Besides this major change, other minor changes involve using functions  
to access things like the class name, superclass, methods and instance  
variables instead of accessing the class data structure directly.

I have completed all of these changes (as far as I can tell) and have  
Swarm working with the new ObjC runtime abstraction interface for the  
GNU runtime on Linux.  My next task is to get this working for the  
Apple runtime.

The new Swarm architecture isn't much different from the old, there is  
just addition of an extra pointer hop (and/or hash lookup) to move  
from the actual ObjC runtime class to the phase definition structure  
(i.e. BehaviorPhase instance).

* The global singleton classes, like OrderedSet, Activity, etc etc.  
now point to the actual ObjC class created within the runtime, not to  
a  constructed DefClass instance.  The exception to this are the  
implemented Types which never actually get instantiated.
* The classData structure which holds additional meta data for ObjC  
classes is expanded and is now hashed with the pointer to the ObjC  
class in the runtime instead of the "class number" which is not well- 
defined on Apple.
* The classID variable with the classData structure points to the  
actual ObjC class.
* Instances of BehaviorPhase still represent a phase of a class, but  
the instance is no longer an overlay for the actual ObjC class,  
instead the definingClass instance variable points to the actual ObjC  
class for the phase.  The nextPhase instance variable still points to  
the next BehaviorPhase.

Essentially by maintaining pointers that go both ways, Swarm can  
access the ObjC class, the class meta-data (in classData structure),  
and the class phases (in BehaviorPhase instances), from any of the  
other structures.

hashLookup(Actual ObjC class) --> classData structure
classData(classID) --> Actual ObjC class
classData(initialPhase) --> BehaviorPhase instance
BehaviorPhase(definingClass) --> Actual ObjC class
BehaviorPhase(nextPhase) --> BehaviorPhase instance

It is hard to give much detail about all of the complexities of the  
defobj functionality in a short email, so I apologize if this doesn't  
all make complete sense.

I encourage anybody feeling brave to try compiling Swarm from this  
branch and test out your applications.

cheers
Scott