Re: [Vrs-development] Distributed Filesystem

[Eric Altendorf]

On Friday 12 July 2002 07:23, Chris Smith wrote:
> I really haven't got my head around how the propogation of data
> segments is going to work.  However I do feel strongly that it
> should be transparent to the application layer (the LDS itself) and
> handled by the RM layer directly.

Certainly...

> [Which brings in a previous email of mine which stated that there
> is a little crossover between the RM and the CM as the CM needs to
> keep the RM updated as to what LDS's are currently active in the
> cluster (VRS).  This is just messaging in the end, the CM
> dispatches an event to the RM to say that something has changed and
> supplies it with enough info to sort out the Resource image.
> I'm tackling everything from a messaging and abstraction point of
> view as I see it as the key to breaking the problem up into
> managable sub-projects].

Right... the thing is it is more than just messaging.  Messaging is 
the physical means by which nodes communicate, sure.  However, the 
real difficulty comes when you try to structure the communication so 
that it's clearly defined what nodes are in the cluster at a given 
time, that all nodes agree on that, that a write operation on a piece 
of data will block read operations on that same piece of data on ALL 
other LDS nodes until the write operation has completed and ALL LDS 
nodes holding that data have an updated, synchronized copy of it..., 
that when an LDS node comes back online, its data is marked invalid 
until it has a chance to synchronize with the rest of the cluster, 
that that synchronization can happen while data is being re-written, 
etc.

This might be a weird analogy, but... messaging is just the means for 
passing data and requests between machines, the way function calls 
are the means for passing data and requests between parts of a 
program on a single machine.  It doesn't change the fact that we have 
an incredibly difficult and complex problem to solve.

I'd like to remind people once again that nobody has *ever* built a 
filesystem like this.  Not NFS, not Andrew file system, not any of a 
number of other experimental distributed filesystems I've read about.  
There are filesystems that guarantee consistency by storing a single 
copy of data, or a master copy plus caches.  There are filesystems 
that guarantee accessibility by allowing many copies of the data, but 
not guaranteeing consistency between them (the Andrew filesystem is 
*weird*!).  There is no filesystem out there (AFAIK) that both 
distributes the master copy of the data and guarantees Unix 
filesystem semantics and ACID transactional properties.

I hate to be defeatist, but I honestly, honestly don't think that this 
is a feasible project unless we get a *real* expert in distributed 
filesystems or transaction processing on here.  And by a "real 
expert" I mean someone who's had a few years experience implementing 
transactional data storage at a place like Oracle, you know? :-)  I 
think our time would probably be better spent on a more ... realistic 
project.

If you guys are still in doubt, I'll sit down and write up a more 
detailed requirements doc that explains the technical details of what 
we'd need to do in order to acheive the goals for this filesystem.... 

eric

-- 
"First they ignore you.  Then they laugh at you.
 Then they fight you.  And then you win."             -Gandhi