GLR, GLT, and the memory array GL

[Todd Boyle]

Somebody emailed me today,

> Its just redicularous to store purely in memory (outside of large
> scale caching) as its so primative and you have so many issusses.
> Issues like network sharing, scalablilty, large data sources, OS's
> doing strange things to your VM, system crashes, mis-addressed ...


I proposed that the general ledger of GNUE is a descriptive structure
and not a transactional one.  The GL would serve disparate business
applications as well as GNUE.  My proposal is that conceptually, a GL
is a data structure to which every business module must report the
the balance sheet and income statement impacts of its events, in CDEA
rows.  Surely the reader understands this is not technically difficult.

I think the real controversy between GNUE or FERMS developers and my
proposal for a memory array GL is that developers regard the General
Ledger as a transaction oriented backplane or data bus which their
applications would use as an anchor, as the primary or sole data
storage for their applications which compose tranactions.

I believe that *no* transaction database that is reasonably optimized
for GNUE applications, such as payroll, can ever successfully serve as
a root ledger for the range of disparate applications that exist
today.  It will be a good thing, that GNUE have a well-optimized
transaction database.  Perhaps, further, it would be a good thing if
the separate GNUE applications should share a common database a hub
for interoperating with each other, as an integrated whole.  (I don't
think so.  I think the applicaitons should be uncoupled.)

In any case, such a database would be far from optimal for supporting
disparate non-GNUE applications.  It would be a big, honkin thing that
requires the SMB to maintain a relational database, and accept
whatever limitations and dependencies and upgrade cycles are necessary
for *the whole GNUE complex*, when often, they only need a root ledger
to provide an aggregated view of two or three BSPs and subledgers.

By now you must realize my belief is that it is more important that
SMBs have a tool to support disparate applications, than that they
have GNUE applications.  Fortunately, a rootledger does not impede
GNUE architecture at all.  It complements it.

The General Ledger may be broken down, conceptually, into two pieces--
as I posted Sep 2, "There should be a GLT (GL for transactions) and GLR
(GL for reporting).

The GLT manages transaction posting, and the GLR part would support
downstream reporting.  The posting engine runs on an RDBMS and is
integral to GNUE business applications.  The posting engine will also
have to support posting business transactions originating externally,
compliant with standards such as ebXML.

The part that supports downstream reporting may be uncoupled from the
posting engine or its RDBMS.  My proposal for this half of the GL is a
memory array GL based on flat rows, which persists its data into
compressed XML files on a storage device.  It would never modify or
delete a file.  It would persist data as often as necessary, and
divided into as many smaller transaction files as necessary, to meet
the organizations' requirements.  If a system failure occurred, the
MAGL would bootstrap itself by collecting the most recent set of
indexes and XML row files that were cache consistent.   Disk storage
would grow endlessly.  A sister program might monitor and cleanup
files but I don't need one, as a small business with tens of GBytes of
excess disk space, having transaction rows so simple that 10,000
transactions would fit inside of 1 megabyte.

The MAGL I described had an interface for loading streams of data
directly from a data source such as PostgreSQL.  It persisted
data to disk only when told to save data, and only in the method
instructed.  Accordingly it would serve alongside GNUE posting
engine and its RDBMS.

Still, nobody has refuted my main theses.

1. supporting disparate applications is important, and would serve
hundreds of times more users than GNUE itself.

2. some division between a GLT and GLR is essential to support
disparate applications, and

3. the memory array GL is a reasonable approach to a GLR component of
GNUE, achieving sufficient reliability, and providing many benefits
since it can be run without GNUE.  It does not impede the scalability
or efficiency of GNUE since it accepts streams from memory which
could originate from detail or summary queries from GNUE RDBMS.

GNUE ane FERMS members: please email me privately if you want me
to shut up and go away, or my emails are too long, etc.!  So far,
nobody has ever said that yet  :-)

Todd

> if your ever written a database engines or caching systems (as i have)
> you would know to need to have time buffered write back caches (or no
> write back cache, if you can't be bothered writing it and your
> operating a mostly read operation.)
>
> Its just redicularous to store purely in memory (outside of large
> scale caching) as its so primative and you have so many issusses.
> Issues like network sharing, scalablilty, large data sources, OS's
> doing strange things to your VM, system crashes, mis-addressed
> pointers, needing as much ram as your data on every client etc etc
> etc.
>
> The application object server I designed for gnue uses caching
> exentisivly, and i would have no desire at all to move all my data
> into memory as my sample data for the gnue payroll system is about
> 800mb.


  From: address@hidden (Todd Boyle)
  Newsgroups: alt.accounting
  Subject: Re: General Ledger Faultlines
  Date: Sat, 02 Sep 2000 05:45:35 GMT

  It is clear to me now, that the "General Ledger" needs to be divided
  into two modules.  There should be a GLT (GL for transactions) and GLR
  (GL for reporting).   "Transactions" are fundamentally processes between
  the enterprise and external parties.  These transactions have
  fundamentally turned inside out, they now span company boundaries.

  There is no other way to reach the efficiencies and process improvements
  needed, than some type of external interface with 3rd parties.  I hope
  everybody gets that, by now.  There is no way to put the genie back in
  the bottle, and the rest of the world is moving forward on this. We need
  interparty integrations and the only way is forward.  All of the
  intellectual vitality today is focussed on B2B, B2C commerce. Witness
  ebXML as a defining forum for all these issues.

  In future, a GLT might be architected along the triple-entry theme I have
  tried to express in http://www.gldialtone.com/str.htm.  It is inevitable
  ther will be some kind of shared transaction table someplace behind any
  successful ecommerce for individuals and small business.

  The fortune 500 is converging around Point-to-point architectures, based
  on middleware, as they redo EDI into XML.  Do not be confused by these
  battles of the titans--small business is never going to maintain complex
  multiple-step commerce on a point-to-point basis.  That would require
  every party to maintain an online, realtime, 24x7 responsiveness and
  complex logic to implement.  Almost a two-phase commit.  Big business
  needs that.  They need to click on things and get an answer from their
  supply chain, and will not delegate stewardship over data.  They are
  willing to install MQSeries, Biztalk server or whatever it takes. We
  cannot afford that.  We need an asynch message forwarding hub like the
  STR.

  The STR is a subset of the rootledgerXML schema, a format for
  representing CDEA (classic double-entry accounting) transaction rows.
  Nobody is going to have any problem implementing GL interfaces to any
  legacy GL with the rootledgerXML schema.  It is certainly backwards
  compatible, and forwards compatible.

  Triple entry accounting is this:  You form a transaction in your
  GLT.  Every GLT transaction requires naming an external party.
  You have a party table.  A party service, perhaps. Perhaps DUNS.
  Perhaps an ebXML registry, or an LDAP directory provider, or
  an authentication provider on the internet.

  When you POST, the entry is stored in your internal GL (the GLT, if you
  have split your ledger, which is my thinking) just like in the past.
  But it is also submitted to the triple-entry table in whatever STR
  system you choose. Perhaps your own STR server, or an STR module on your
  GL, or a big STR server out at Exodus or your ISP or someplace. The same
  information you stored in your GLT entry suffices to complete the shared
  entry and your private stub in the STR.  If the entry is part of a
  pattern, you must provide the GroupID.  Otherwise, the STR assigns a
  GroupID in case you need it later.  See #4 below.

  The STR diligently forwards the entry to the reciprocal party. If the
  reciprocal party is interested in integration they will reply to the
  transaction sent by the STR, and optionally, complete their private Stub.
  That is triple entry.   STUB - SHARED ENTRY - STUB.  This is a
  subledger, folks.  Your work is done, if you trust your STR's stewardship
  of data.  Of course you will still need a rootledger for control
  totals of this, and other STRs, and for your GLR.

  The GLR is not so different from the GLT except that the architecture
  is uncoupled from the GLR so that it can travel in different
  directions.

  1. the GLR is much more private, containing all the capital
  accounts, permanent histories of fixed assets, all the accruals
  and adjustments, the overall picture of the enterprise.  This
  reduces risk and provides more granular security.

  2. the GLR is the component which supports financial reporting
  integration.  Financial Reporting is obviously going to be subject
  to very demanding, ongoing technology challenges, for example
  requiring a full XML parser, and supporting very demanding tax
  and management reporting needs.  These have manifestly become a
  separate problem domain, with separate requirements, from the
  processing of B2B transactions, supply chain, demand signals,
  etc.

  3. the GLT is something that is almost becoming a community asset.
  You just cannot get the kind of integrated economy we need, without
  some real consensus among practitioners, to move certain parts
  of the transaction to a shared place.  I am not saying public;
  rather, the amount, date and description of a deal are inherently
  shared between two parties and should be stored visible to those
  two parties alone, encrypted.

  The storage of shared transactions will experience security glitches.
  That is what happens.  When the automobile was invented the first thing
  everybody did was kill themselves in headon collisions.  Airplanes. same
  deal.  But if you want to save 10 million person-years of work every
  year, it is well worth the price of a certain messiness. That is why you
  split the GL into a GLT and GLR.

  4.  The GLT must be evolved to support patterns.  ebXML is calling
  these "Commercial Transactions". They are exceedingly important.

  A particular transaction in business almost never stands alone. They
  come in patterns.  For example offers/acceptances form a transaction but
  seldom encapsulate the entire fulfillment and payment cycle. Even if
  there has been a payment accompanying a PO message, the customer is
  now waiting for fulfillment.  There is a whole science and literature
  built around this, and adopted by Business Process workgroup of ebXML.
  REA talks about this, in their Dualities.

  Patterns are not the same thing as XML "conversations" such as
  RosettaNet "pips" or OAG "scenarios" which are more like

     hello are you there, send me your price for a widget
     yes, im here, widget model 234 costs $45.33.
     ok here is my PO for 12 widgets. plese confirm
     no, you don't have credit.  give me a credit card.
     ok my credit card is 0000-000-000-000
     Ok order accepted.

  You can see that the above is a real-life PO submission. That's what
  happens, in an XML PO submission between two computers. A pattern
  includes the PO and also, any later delivery, or settlement of accounts
  thru an OFX transfer etc.   A GroupID is needed to associate all the
  dealings surrounding a particular "Commercial Transaction" pattern.
  ebXML will come with these patterns baked in.  There will be registries
  defining standard commercial patterns.  ebXML mailing lists seem to
  indicate this will described in documentation soon.

  The GLT, freed to optimize for all the transaction oriented stuff, can
  evolve to satisfy things like ebXML patterns, will be a better animal.

  For example you would never have two GLRs.  But you might have numerous
  GLTs which could be managed as subledgers in consolidations.  I don't
  know. Perhaps your rootledger GLT would only contain summary entries,
  like control accounts, pointing to STRs.  That would be elegant; you
  would eliminate redundancy in detail rows..... in any case, the
  principle is that the GLT developer can ponder those things without
  getting sucked into the whole XBRL, XML parsers, and years of disruptive
  changes in requirements in reporting technology.  You almost get a sense
  that the GLR might reside at a BSP, or be administered by a CPA firm
  perhaps.  That is yet another reason to split the GL into two
  components.


  * Todd F. Boyle CPA    http://www.GLDialtone.com/
  * address@hidden    Kirkland WA    (425) 827-3107
  * XML accounting, web ledgers, BSPs, ASPs, whatever it takes