Re: [pooma-dev] Problem with Field::makeOwnCopy() and Relations

[Richard Guenther]

On Wed, 21 May 2003, John H. Hall wrote:

> Richard:
> >
> > Uh, while I dont use dependend Field feature (yet), I cannot see how
> > the centering can make a difference here.
> >
> It been a while since I used POOMA so some of my cobwebs need to be
> shaken out. In moving from POOMA R1 to later versions of POOMA R2 we
> removed centering as a template argument, so you are correct, the real
> issue is that the current instantiation of relations inside POOMA
> requires fields with the same number of subFields because of the way
> Scott wrote the loops. So when I referred to centerings, I didn't mean
> Vert and Cell, I meant Edge and Cell would be incompatible since they
> have a differing number of subFields.

Ah ok, I see what you are talking about. But these expressions involving
multiple subfields are error prone and sometimes quite surprising in the
outcome. Not only because you cannot take views of fields with >1
subfields and calling field.physicalDomain() is "undefined" for fields
with >1 subfield.

> > ...
> > Yes, but program (or better, expression) flow would be very hard to
> > follow
> > then. So from a maintainance point of view I doubt this would be
> > useful.
> > I see it can be useful for sort of CSE, where you set up a dependent
> > field
> > for the expression.
> >
> Since we were trying to investigate modern computer science we
> developed tools allow us to do full round-trip engineering. So the
> dependencies were actually quite simple to track and we worked in a
> graphical environment in maintaining them using a customized version of
> a technology called "Describe" (originally GDPro) from Embarcadero
> Technologies. We had to introduce and work through a lot of concepts,
> like "state" to make things work. Also, the way we implemented
> relations you had to think of them as having views of the things they
> were dependent upon which always led back to the same data, otherwise
> you would get divergent unexpected copies of your data floating around.
> We had some additional structure to support this.
>
> One of the major contributors to code fragility is implicit contracts
> between subroutines which are violated when an unaware subroutine is
> placed between them that violates the "contract". Computers were meant
> to track these things and the use of relations led to a significant
> reduction in user-supplied code. As with all technologies, there are
> pathological cases that would eventually result in trouble if not
> designed around, but, they were fairly straightforward to avoid. Since
> most of them led to infinite loops, that also made it simpler to figure
> out when you had fallen into a trap. We also defined a Macro which was
> placed at the beginning of every method by our automatic code generator
> that would allow us to conditionally turn on the call chain to allow us
> to see if we had any cascades or unnecessary loop updates going on as a
> result of having our dependencies set up incorrectly.
>
> A question we never had to ask was "Has this variable been updated?"
> When you are integrating a large set of physics packages, it becomes
> important to have these contracts explicitly spelled out or remove them
> with a scheme such as ours. Avoiding unintended side-effects becomes
> the name of the game.

Yes, I see. Some highlevel tools to at least view the dependence graph
would be useful. Should be extractable at runtime and displayable by
f.i. the dot package ... sounds like a project ;)

> > As you are (were?) a POOMA user, did you have tricks to overcome the
> > overly simplistic handing of the inner guards and their exchange? I.e.
> > the fact that only a simple flag is kept for the state of the internal
> > guards, so you cannot optimize f.i. directional splitted CFD and
> > instead
> > do tree times the communication you need to?
> No, we talked about it a lot. Our hardware was not bandwidth limited,
> it instead had a large latency, so we bundled all the communications
> into as few messages as possible, but, we really didn't care what the
> message size was. We spent much more time thinking about the external
> guard cell fills and while we came up with some ideas for optimizing
> these, most of them have never been implemented.
>
> Our team did come up with the deferred guard cell update idea which was
> wonderfully implemented by Scott Haney (that's the flag you are
> referring to). Before this the guard cells were updated at the end of
> every assign statement, which was truly problematic for performance.
> His code in R1 would only require updates if the RHS expression were
> using guard cells. I believe the R2 version just always requires an
> update if the field is on the RHS of an expression (stencil or no). A
> major driver for recognizing the difference between Dependent and
> Independent Fields was that Dependent Fields would almost never need to
> communicate to fill their guard cells (only if they stenciled across
> the Independent Field). We didn't get far enough along to take
> advantage of that optimization.

Yes. If I have enough time I'll look into these issues, but a one-day
try had no success sofar.

> >
> > Btw. - what is/was the application you were using POOMA on? Is it
> > available somewhere, so people can learn from it?
> >
> There were a bunch of CFD codes all developed under a project at Los
> Alamos National Laboratory known as Blanca. AFAIK none of these codes
> are available to the public and most of them were export controlled so
> they probably never will be.
>
> In the current version you do lose some optimizations if you stick to
> the strict data parallel syntax (e.g. loop fusion, directional guard
> cell fills for operator split algorithms). But it is still the most
> advanced array/field abstraction developed to date and I hope to get
> back to moving it forward again soon.

I hope so, too. POOMA is really the best designed package for CFD I
know, and I'm glad to have stumbled over it.

Richard.