Re: [lmi] Editor for product files

[Greg Chicares]

I've gotten suggestions from several users, and I'll combine them
with this older email without trimming any part of it, so that we
have all these ideas for future directions in one place.

On 2005-8-10 15:22 UTC, Greg Chicares wrote:
> The editor we've been discussing handles several types of files:
>   *.db4 *.fnd *.pol *.rnd *.tir
> The '.pol' file names the others to be used for any product;

This is just an accidental artifact of the data-storage techniques:
see below.

> a 'product' is defined by the collection of product files it uses.
> 
> The files are separate because the data they contain are of
> different types. Eventually (later this year), we'll want to find
> a way to use xml for all of them. When we do that, there will no
> longer be any good reason to keep the files separate, so we'll
> want the editor to handle the whole collection of data as a unit,
> rather than as separate groups.

That way, of course, we won't need one file to name the other
  *.db4 *.fnd *.pol *.rnd *.tir
files a 'product' uses: each product can be embodied in one file.

This eventual change would lose what might seem to be an advantage
of the current implementation: that it enables something like
"programming by difference". Real-world experience suggests that an
insurance company might have fifty products but only five '.rnd'
files, because many products are frequently "derived" from others
but rounding rules rarely change. Yet implementing only five
physical '.rnd' files and sharing them confers only an illusory
advantage: each is 84 bytes, so 3780 [84*(50-5)] bytes are saved,
but a user who wants to know how product X's calculations are
rounded needs to consult 'X.pol' to see that it uses the rules in
'Y.rnd', and users would rather spend a few kilobytes to avoid that.

'.pol' files that point to '.db4', '.fnd', '.rnd', and '.tir' files
demonstrate one form of indirection that we can thus do without.
But there is another, less apparent form of indirection--some data
are stored in external tables that follow this specification:
  http://library.soa.org:8080/xtbml/docs/standard.html
[Actually, today we're using a binary format that preceded that
xml specification.] This is a good idea because it's an industry
standard, and because the Society of Actuaries publishes carefully
checked versions of tables we must use in that format (and in its
predecessor binary format). Established practice is to use that
format for a company's proprietary data as well, so there will
exist more than one file in that format. Each binary file can
contain potentially thousands of tables.

Today, tables in SOA format are referred to as follows:
  '.pol' file specifies the table file name
  '.db4' file specifies the table(s) to access in that file
When we eventually merge lmi's data files, both those things will
be specified in one file, which is slightly more convenient for
users. Several users have asked for the product editor to hide
that indirection and edit these SOA-format files directly; in
the more remote future we'll consider that.

The largest data tables are also those most suitable for reuse.
By exporting them to SOA-format files, today's lmi data files
  *.db4 *.fnd *.pol *.rnd *.tir
can reasonably be kept to about 25 Kb per product. An xml file
resulting from merging those together won't be prohibitively
large. They will, however, become larger, and we'll want to
consider compressing them with, e.g., zlib or libbzip2, which
will confer the extra advantage of making it difficult to read
data that an insurance company wants to keep proprietary. Of
course, this can be combined with encryption of whatever strength
may be desired, e.g. by using GPGME. This is all on the distant
horizon today, but worth recording now.

> I think this could be done with almost the same GUI currently
> used for '.db4' and '.tir' files. The tree control on the left
> side, with the selected item's data on the right, is a paradigm
> that works well. The right-hand panel would have to change
> dynamically depending on the datum's type, which would of course
> be specified in xml.

The 'wxTreeBook' control we've discussed recently might help here.

> For the '.rnd' files that specify rounding, we currently offer
> a set of owner-drawn radiobuttons. That seems to work OK, though
> perhaps we should find a different way if that's difficult. It
> is sort of nice to see all rounding rules on the same screen,
> but if we show only one at a time due to a new tree-view
> paradigm, I don't think anyone will miss seeing them all at
> once--and if we add lots more rounding rules, then they won't
> all fit on the same screen anyway, and the genericity of the
> tree-view paradigm will be welcome in that event.

Here's a different idea that I think Steven suggested: put the
whole set of rounding rules into one entity, displayed in one grid.
Let's declare that to be the definitive decision.

Thus, there will be no owner-drawn radiobuttons. Instead, a grid
with exactly two columns suffices. One column represents rounding
direction as an enumerative choice, naturally displayed in a
drop-down combobox embedded in the grid cell. The other column
represents number of decimals, and it's just an integer in a
normal numeric cell. All rounding rules for a product can be
viewed at once--or, if there are too many to fit on the screen,
then at least they're in the same grid, and grids are scrollable.

This strongly resembles the two-column grid we already need for
the data in '.tir' files. I feel a slight pang of wistful regret
at losing the owner-drawn buttons, but introducing an exotic
element for only one purpose makes the interface less coherent
overall.

The '.fnd' data should be handled similarly. It's of the same
basic nature as '.tir' data, except that it's three columns:
two strings, and an integer.

> I think it would be best to design the new editor with this
> sort of change in mind, but actually convert the data to xml
> later. The xml conversion is a large, separate project that
> affects much other code in lmi, and deferring it would keep
> different developers from changing the same code for now.
> Creating the new wx-based editor first would also fill a
> short-term business need by making the old program now used
> for that purpose obsolete.

Here's an important idea that I don't think we've discussed yet.
Many data in '.db4' (and '.rnd') files are enumerative by nature.
Today, merely as an artifact of the '.db4' file's binary layout,
such data are stored as double, and the legend at the bottom of
the screen lists the enumerative values, e.g.
  Separate-account spread and fee subtraction:
  0 = annual (effective), 1 = daily (nominal)
I believe wxGrid already permits cells to be edited as drop-down
comboboxes, which is a much nicer way to display the available
choices; it should also contrain values to those choices, which
isn't possible in the legacy implementation. End users can't
define new enum values, but lmi needs to do that sometimes.
The example above is entity 'DB_SepAcctSpreadMethod', which
lmi knows is of type 'e_spread_method', which is a class that
has a member function
  static std::vector<std::string> const& all_strings();
which returns
    {"Effective annual", "Nominal daily"}
which are suitable for display in a combobox. It would seem
ideal to use that existing facility in the product editor,
perhaps by including lmi's enumerative-class object files:
that way, whenever lmi adds a new enumerator, a new combobox
item would be shown automatically. Maybe that's harder than
I'm thinking, but it does seem most desirable.

Here are some other ideas for future enhancements. We don't need
them right now, but ought to bear them in mind.

Users have asked for a search facility. It's a great idea, though
I don't have any particular idea how to implement it. Each '.db4'
entity already has a descriptive string, like the example above:
  Separate-account spread and fee subtraction
and it's sufficient to search for words or portions of words in
those strings. Searching strings is easy, but I don't know what
to do if we find, say, three matches--or, rather, I can think of
several ways to treat search results, but don't know of one that's
clearly better than all others.

Guaranteed vs. current isn't a '.db4' axis now, but should be
someday. That's been discussed already in its own thread here:
  http://lists.nongnu.org/archive/html/lmi/2005-08/msg00030.html

This is an exhaustive list of all the ideas users suggested.
We probably can't do them all now, but at least we can avoid
making design decisions now that would make it hard to implement
these ideas later.