DiscField

[James Crotinger]

Hi John. Just wanted to update you on our DiscField replacement. It consists
of two classes, FileSetWriter and FileSetReader (there are supporting
classes, but these are the two that the user will deal with). The interfaces
are simple - once you've initialized the appropriate class and opened a file
set, you either invoke "read" or "write" with an Array or Field argument.
There is no support to random access within a record (is this desireable?)
nor is there currently support for reading views. Oh, and currently these
only deal with single file sets, which are read/written via context 0.

Today Scott moved FileSetWriter to the blanca branch, along with a bunch of
supporting changes. I have not tested these in parallel, nor have I finished
testing FileSetReader (which is still just on the CVS trunk). FileSetReader
works in serial on the PC and nirvana, and probably Linux, although I
haven't tried writing a linux file set to test with.

My immediate plans are to test in parallel and move FileSetReader to the
blanca branch. That'll give you the ability to read whole Fields from a
single fileset on nirvana. Once that is working, I have three improvements
in mind:

 1. Multiple filesets - we think this is straight-forward, and in fact the
only code that is not in there is the communication of the layout
information so that all contexts know the complete layout (this is for
reading - writers know the full layout). Scott has already added some
supporting changes for doing this and I just need to add the communication
code to FileSetReader (or DiskLayout) and it should work.

 2. Views - this also should be straight forward. We're using Arrays with
remote engines to write the patches and it shouldn't be too hard to mask the
assignments and only read data that is going to cause elements to be
written. 

 3. Machine independence - Currently, FileSetReader detects byte-reversal
and fixes it (the heuristic is that it looks at the domains in the .layout
file and examines the stride to see if the 1 is in the first or last byte -
this does rely on saved Fields always having stride 1). I want to go further
and standardize on the layout of the OffsetData field in the .offset file as
being that stored on nirvana with "long long" offsets. We can write the code
to always write this arrangement of bytes to disk. That plus detecting
byte-ordering should make the files almost machine independent. The main
problems will be files written by r1 on machines other than SGI/IRIX 6.5. We
could deal with these via some new keywords in the .meta file.

Anyway, that's where things are at. I'll try to get some work done on these
this weekend.

Could you please forward this to Lee and Peter if they're not on the
pooma-dev list.

  Jim