[Monotone-devel] [RFC] Monotone NETSYNC Hook Extension & Abstraction Layer

[Ralf S. Engelschall]

On my road of trying to adopt Monotone for use in my various larger
Open Source projects like OSSP, OpenPKG, etc I've last week worked on
an important issue (at least for me) which perhaps looks strange to the
"maximum distribution is everything which matters" guys in the VCS camp:

   How to best combine the developer-requested distributed VCS nature
   of Monotone with the central ACL nature required at the "master"
   repository of those projects?

To get you an impression how "deep" the problem is we try to solve
here: we want that the developers can use full distributed VCS outside
the central repository. Nevertheless, they have to fulfill a mandatory
contributor agreement (to protect the licensing, etc) which especially
means they are allowed to propagate/merge/pluck only revisions within
a particular branch tree (e.g. "openpkg.*") and inside this tree only
developers who have signed the contributor agreement are allowed
to commit their stuff. As long as a developer is fulfilling these
constraints he will be able to share his revisions (which indirectly
usually carry the revisions of others due to merging) with the master
repository (from which official release tarballs and packages are
rolled). If he breaks out of these constraints, he is _forced_ to stay
out at all.

Please do not confuse this with the idea of policy branches some of
you already discussed. My understanding is that policy branches are
about distributed policy _recommendations_, while I'm talking about
central policy _enforcement_ here. So, this stuff is the other side
of the "policy branches" medal. For a final solution one might wish
both AFAIK. The "policy branches" are for the distribution, while our
solution is for the "master" repository which usually is the middle of
our proposed partial star topology.

My partner Thomas Lotterer and I came up with the idea to solve the
problem at the server-side of the master repository by directly
enforcing the access control on keys, revisions, and certs during
NETSYNC and even _ROLLING BACK_ the _WHOLE_ NETSYNC transaction in
case an ACL fails. And I've now also developed an experimental
implementation for this and I have written a little bit more about this
stuff in my BLOG a few days ago (see the following URL for details:
http://trainofthoughts.org/blog/2007/09/16/distributed-vcs-central-acl/).
We are currently testing this stuff in practice with a few test
repositories. Looks promising so far.

Now comes the time where I would like to let you Monotone hackers review
this stuff in more detail. The stuff mainly looks like this (see the
attachments to this mail for a snapshot of this code):

1. monotone-netsync-hooks.diff

   Monotone was extended in C++ with four additional NETSYNC Lua hooks:
   one for the receiving and storage of each type of data: epoch, key,
   revision, cert. The hook for each data can decide whether to accept,
   ignore or reject it. And for rejecting it can decide whether after
   aborting the NETSYNC session the already received data is kept
   (database transaction commit) or thrown away (database transaction
   rollback).

2. lua-logfile.lua
   lua-dump.lua

   These are reusable auxilliary Lua extensions (not depending on
   Monotone at all actually). The "lua-dump.lua" is like Perl's
   Data::Dumper and a great tool for debugging Lua scripts during
   development. The "lua-logfile" is a little OO class for logfile
   abstraction. It can be optionally be used by "monotone-netsync.lua"
   and "monotone-aaa.lua" to log to a (even common) logfile.

3. monotone-netsync.pod
   monotone-netsync.lua

   This is the generic NETSYNC hooking abstraction layer for Monotone
   Lua which allows arbitrary Lua extensions to easily hook (even in
   parallel) into the NETSYNC processing. The primary consumer certainly
   is the "AAA" extension for Monotone, providing Authentication
   (Access), Authorization (Content) and Accounting (Logging). But is
   can be also used for hooking a mirror extension or whatever else
   interesting into Monotone. All without interference.

4. monotone-aaa.lua
   monotone-aaa.bio

   This is the first cut for an AAA Lua extension which at least
   provides the access control part. It uses a basic_io based ACL file
   for granting read-only or read-write access on certain branches
   to certain keys (corresponding to the users). The difference to
   Monotone's "read-permission" and "write-permission" files is that it
   especially allows the control of write operations to certain branches
   (and not just in general to all branches). Also, it uses a single
   "basic_io" file and leverages from the new NETSYNC abstraction.

5. monotone-server.lua

   This is the --rcfile of the "mtn serve" process for running an ACL
   controlled Monotone server on the "master" repository side.

If you review it, review it in this order, please. Else it certainly is
not such easy too understand as the later parts in the above list depend
on the earlier parts.

My proposal would be that at least "monotone-netsync-hooks.diff" (1)
and "monotone-netsync.{pod,lua}" (3) should be provided by Monotone
out-of-the-box as these are absolutely generic and reusable parts lots
of people could benefit from. The other parts are more specific, but
still could be worth to be included with Monotone (in contrib/ only).

Feedback?
                                       Ralf S. Engelschall
                                       address@hidden
                                       www.engelschall.com

PS: If you think I like Lua just because I coded all this stuff in Lua,
    please see http://trainofthoughts.org/blog/2007/09/16/lua-popularity/
    and give your feedback there, too ;-)

monotone-netsync.pod
Description: Text document

monotone-netsync.lua
Description: Text document

monotone-netsync-hooks.diff
Description: Text document

lua-dump.lua
Description: Text document

lua-logfile.lua
Description: Text document

monotone-server.lua
Description: Text document

monotone-aaa.lua
Description: Text document

monotone-aaa.bio
Description: Text document