[Vrs-development] Re: SEE and Goldwater integration (was Re: [DotGNU]Network SEE architecture, v2.0.1)

[Chris Smith]

On Friday 11 October 2002 11:32, Stephen Compall wrote:
> Note: I will be gone (that is, offline) from before 11 Oct 2200 UTC to
> 15 Oct sometime probably.

> Serveez is a server framework that allows you to write callbacks that
> act as servers whereas Serveez handles all the network communications.
> It has some weaknesses, but was evidently designed to be compilable with
> MSVC. That is valuable to me, because I don't know anything about
> writing code that can compile with MSVC, and don't really want to learn.
> Good for cannibalizing y'know :)

I've got a similar thing called netengine.  It does all the network stuff for 
you and invokes callbacks to pass you data chunks etc.  It's a single 
threaded internal-multiplex model (as used in Zeus http://www.zeus.com) and 
is extremely fast.  It should port to MS with negligable effort.

MSVC is good.
The reason for abstraction in the middleware has absolutly everything to do 
with portability and hiding the implementation details.

>  > VRS is being built around a middleware that does all the
>  > single-machine inter-process-comms for you and also inter-machine
>  > comms too. However, we (the VRS) will still need some sort of network
>  >  presence for the request/reply transport. How this will be done is
>  > undecided - possibly use off-the-shelf daemons like Apache and Jabber
>  > for this. What are your thoughts on this area for SEE?
>
> see*user and see*port are the network players in the SEE model. SEE
> clients and SEE daemons don't even know they're there; they only see the
> FIFOs.

Same with the VRS/Goldwater model.  Components only know about queues (FIFOS).

>  > So given that both our designs are so modular with a significant
>  > functionality cross-over, I'm kind of hoping that we can factor out
>  > the comminality (which is effectively the receive, decode, fetch,
>  > execute and return bit) and provide a SEEinterface module and a
>  > VRSinterface
>
> My concern is not about the defined-ness of the API, but the weight.
> That is, I intend it to be trivial to write from-scratch
> see*port/see*user pairs, w/o using existing libs, provided that the
> language can open/close files and has an XML parser.

Light as a feather light thing.
goLDWAter = [L]ightweight [D]istributed [W]eb [A]rchitecture to be precice.

ie to send an arbitary message to a queue called 'MyNiceQueue' you do this:

char *data = .......
int   length_of_data = .....;

gw_svc_call( "NyNiceQueue", data, length_of_data, ... );

and that's it.
WHAT the data that you're sending is is up to you, (but it helps if the thing 
at the other end of the fifo can understand it!)

The api allows fire and forget, and fire and wait for response data.

> Hate to say it, but the prospect of shared memory makes me a little
> nervous. :]

Its just how the middleware keeps track of what processes it has started, 
what they are doing, whether they've locked up/crashed and need to be 
restarted, what queues are available, how loaded they are, whether there are 
othere queues it can use instead etc....

> Yes they do...the ones in my head that is. You have seen the
> outside-my-head diagram :)

Like I said - conceptually they're not too different beasts!

> However, my head depicts startsee off to the side, then see*ports next
> to the network interface, and behind them various webservices, and all
> of the services being children of seedaemons (which, BTW, refers to 1
> process), but it's not behind the children because the see*ports need to
> talk to it too. Actually, I'm thinking of moving the FIFO creation to
> startsee, because seedaemons *must not crash ever*

The middlware has a 'guardian' process that watches over all other processes 
using the middleware.  Thus if something like the VRS equivalent of seedaemon 
did crash it would be re-started instantly.

> see*users are not important in this diagram, they're just clients.

Yep, they're not important for the VRS either.

> The message queues (by which you mean FIFOs, I'm sure) are not in a big
> block, they're really independent, only names are decided rather easily
> (index to number of connection see earlier post (and the xml block in
> see-arch 2.0.1). Timeouts are implemented through deleting the FIFO.

Goldwater is a message queueing middleware, hence the importance of the queue 
in the diagram!

> Finally: it's not so much the VRS and SEE overlapping, it's SEE and
> Goldwater, I think. Make of this what you will.

So, the VRS too will have seedaemon and see*port-esque processes.  The 
network listening bit could very easily be serveez as all it has to do is 
listen for data and send it to the relevant FIFO/queue.

The seedaemon-esque processes are a bunch of processes with ilrun embedded 
within them that are sitting on the 
'AnythingPlacedOnHereWillGetRunByASeeDaemonThingy' FIFO/queue. 


These are the bits I see as being common (noting that no design of see*port 
type processes has been done for the VRS, we've (I've) just identified the 
queue that these processes will need to send the data they receive.


I'm *Not* trying to coax you into using Goldwater Stephen :o)  but several 
mutually benificial things come to mind if we at least share the same 
inter-process comms API...

1. Goldwater handles the startup/shutdown/restart of processes for you, so 
that would be something not to worry about designing/coding/testing.
2. Golwater provides a lightweight FIFO, the implementation details of which 
are hidden from the user.
3. The VRS project could then use any See*ports written for SEE and vice 
versa ('cos they're using the same API).
4. The SEE and VRS would share the same sandboxed secure execution 
environment implemented by the VRSSEEDaemon processes.
5. Portability means making the same messaging API available on the target 
platform.  It (the middleware) may not be as fully featured on Windows as on 
Unix, but that's not a concern.


You've identified things in SEE that the VRS will need but hasn't qualified 
properly yet.  I'd like to be able to interchange things between the SEE and 
the VRS - from a user point of view as it avoids the confusion of haveing two 
incompatible applications within dotGNU that do very similar things.


Chris.

-- 
Chris Smith
  Technical Architect - netFluid Technology Ltd.
  "Internet Technologies, Distributed Systems and Tuxedo Consultancy"
  E: address@hidden  W: http://www.nfluid.co.uk