Re: [Bug-apl] ISO Component File API preview

[Elias Mårtenson]

No, I used the correct acronyms in my email. DBM is a general term referring to (at least) three different products: dbm, ndbm and gdbm. All of these use a similar API.

BDB is a different product, and has similarities with DBM in the sense that it too is a key/value store. It has a different API (although it is generally similar) and is a bigger product.

DBM is certainly much more lightweight and is more like using "plain files" if you like.

Regards,

Elias

On 11 July 2014 13:10, David B. Lamkins <address@hidden> wrote:

Did you mean to name BDB in the second sentence, or is that a limitation
of DBM as compared to GDBM. Either way, thanks. I'll start with BDB and
compare it to SQLite.

On Fri, 2014-07-11 at 13:07 +0800, Elias Mårtenson wrote:
> BDB is much heavier than DBM (of which GDBM is an implementation). DBM
> only allows a single process to open the database for writing.
>
>
> I'd like to start with DBM support, as it is simple and efficient, and
> also because I have more direct experience with it. BDB is cool
> though, and I can add that as a second phase.
>
>
> Regards,
> Elias
>
>
> On 11 July 2014 13:04, David B. Lamkins <address@hidden> wrote:
>         Could you also do BDB? I'd like to try both GDBM and BDB.
>
>         I've quickly skimmed both manuals; it looks like GDBM provides
>         no
>         support for transactions or journals. I'd like to be able to
>         keep that
>         functionality without having to write it myself (at least in
>         the initial
>         implementation).
>
>         SQL is optional to BDB on Linux builds. I can live without the
>         SQL
>         support...
>
>
>         On Fri, 2014-07-11 at 12:24 +0800, Elias Mårtenson wrote:
>
>         > Would you be willing to move the CF stuff on top of GDBM?
>         It's a much
>         > better fit for it, and will give you everything you do with
>         SQLite
>         > today, without the extra overhead (SQLite is also not very
>         efficient
>         > if you open multiple databases).
>         >
>         >
>         > Regards,
>         > Elias
>         >
>         >
>         > On 11 July 2014 12:21, David B. Lamkins <address@hidden>
>         wrote:
>         >         On Thu, 2014-07-10 at 18:52 -0500, Blake McBride
>         wrote:
>         >
>         >         >
>         >         >
>         >         > The component file system is for APL-only
>         applications.  The
>         >         databse
>         >         > API is to communicate with the rest of the world.
>          It
>         >         depends on what
>         >         > you need.
>         >         >
>         >
>         >
>         >         and...
>         >
>         >         >
>         >         > The problem is that you are creating a component
>         file system
>         >         that
>         >         > _depends_ on features of SQLite and is
>         _needlessly_
>         >         dependent on those
>         >         > features.  I understand it works on other
>         databases, but it
>         >         is an
>         >         > unreasonable burden on those systems.  You could
>         just as
>         >         easily create
>         >         > the same system that doesn't depend on particulars
>         of
>         >         SQLite.  That
>         >         > would work everywhere in a way that is reasonable
>         to any of
>         >         the
>         >         > systems.
>         >         >
>         >
>         >
>         >         Ahhh... OK. Perhaps I've had an "aha!" moment. :)
>         >
>         >         If I understand you correctly, generalizing the
>         current
>         >         component file
>         >         implementation to work with databases other than
>         SQLite is
>         >         problematic
>         >         for all of the reasons you've discussed: poor
>         generality, poor
>         >         use of
>         >         resources, and - in general - a poor fit to SQL.
>         >
>         >         Therefore, here's my proposal:
>         >
>         >         I'll drop all further work on generalizing the
>         current
>         >         component file
>         >         API to work with SQL databases. IOW, the component
>         file system
>         >         will
>         >         remain - literally - a component *file* system. The
>         quick
>         >         patch that I
>         >         worked up last night will never become part of my
>         code.
>         >
>         >         Programmers who want a way to persist data for an
>         APL-only
>         >         single-user
>         >         application and a shallow learning curve and no
>         sysadmin or
>         >         dbadmin
>         >         concerns will (I hope) appreciate and use the simple
>         >         component-file API.
>         >         Programmers who need the functionality of a real
>         database -
>         >         and all that
>         >         it offers in terms of access control, concurrency
>         and remote
>         >         access -
>         >         can use lib_sql with a PostgreSQL backend.
>         >
>         >         The presence of SQLite will remain an incidental
>         >         implementation detail
>         >         of the component file library. All further concerns
>         specific
>         >         to the use
>         >         of SQLite as an implementation vehicle will be
>         treated as
>         >         out-of-scope
>         >         for the project.
>         >
>         >         The current component file code can serve as a
>         reference
>         >         implementation
>         >         to validate the design choices not implicitly
>         required by the
>         >         ISO APL
>         >         Annex. There are a number of such choices that I was
>         forced to
>         >         make in
>         >         the absence of detailed guidance from the spec. For
>         example:
>         >         the choice
>         >         to ensure that component IDs assigned by CF_APPEND
>         are always
>         >         nondecreasing for a given component file. There are
>         many other
>         >         detailed
>         >         choices that had to be made. If you have a concern
>         with how
>         >         I've
>         >         implemented the API to meet the spec, we can work
>         out the
>         >         details
>         >         together.
>         >
>         >         Meanwhile, if someone feels that it's important to
>         layer a
>         >         component
>         >         file API on top of lib_sql *and* allow a choice of
>         database
>         >         backends,
>         >         they may either start from scratch or fork the
>         existing code.
>         >         Personally, I don't see the need for this particular
>         layering
>         >         of
>         >         abstractions; for anything more complex than an
>         >         integer-indexed file of
>         >         APL objects, I'd prefer to work directly with
>         lib_sql. But if
>         >         someone
>         >         else sees value in the layered approach, by all
>         means pursue
>         >         it as a
>         >         separate project.
>         >
>         >         Meanwhile, I'll continue working on design sketches
>         for a
>         >         component file
>         >         system that doesn't build upon SQLite. The primary
>         reason for
>         >         doing so
>         >         will be to increase performance while not requiring
>         use of
>         >         explicit
>         >         transactions to reduce single-record insert times
>         below the
>         >         100ms mark.
>         >         Of course the content of the component file will
>         need to
>         >         remain reliable
>         >         in the face of software and hardware failures;
>         that's one of
>         >         the
>         >         implementation attributes that I got "for free" by
>         choosing
>         >         SQLite as a
>         >         short-term implementation tactic. An alternative
>         >         implementation will be
>         >         challenging, no doubt, but worth the effort in the
>         long run.
>         >         Whether
>         >         such an implementation should be done as a native
>         library or
>         >         as a hybrid
>         >         is not clear to me at this time.
>         >
>         >         The take-away from all of this should be the
>         following: if you
>         >         use the
>         >         existing component file library (and I certainly
>         hope that you
>         >         will do
>         >         so in cases where it best serves your needs), you
>         can be
>         >         assured that
>         >         any future work that I do with the component file
>         library will
>         >         preserve
>         >         your data. Any change in the file format will be
>         accompanied
>         >         by a
>         >         utility workspace to migrate your existing component
>         files.
>         >
>         >         >
>         >         > This is the most I've been involved with a
>         community
>         >         project.  Fun!
>         >         >
>         >
>         >
>         >         Indeed. :)
>         >
>         >         Also, it's worth once again acknowledging Jüergen's
>         work in
>         >         making all
>         >         of this possible. All of these little libraries and
>         utilities
>         >         are
>         >         penny-ante stuff compared to the core interpreter. I
>         just want
>         >         to say,
>         >         again, that I am grateful for the ability to work
>         with a
>         >         full-featured
>         >         open-source APL interpreter for the first time ever.
>         If you
>         >         look at the
>         >         history of free APL systems, they've traditionally
>         been thin
>         >         on the
>         >         ground, undermaintained and not very interesting.
>         >
>         >         A lot of technologies have developed or matured in
>         the past 30
>         >         years to
>         >         get to the point where we are today with free APL:
>         Unicode,
>         >         Linux,
>         >         Emacs, gcc, fast hires video and the x86 hardware
>         platform
>         >         being
>         >         foremost among a plethora of candidates. None of us
>         would be
>         >         here on
>         >         this mailing list today, however, had Jüergen not
>         put in an
>         >         immense
>         >         amount of design and implementation effort to create
>         GNU APL.
>         >         I
>         >         sincerely appreciate his continuing support and
>         involvement.
>         >
>         >
>         >
>
>
>
>
>