Re: [Bug-apl] ISO Component File API preview

[Elias Mårtenson]

BDB is much heavier than DBM (of which GDBM is an implementation). DBM only allows a single process to open the database for writing.

I'd like to start with DBM support, as it is simple and efficient, and also because I have more direct experience with it. BDB is cool though, and I can add that as a second phase.

Regards,

Elias

On 11 July 2014 13:04, David B. Lamkins <address@hidden> wrote:

Could you also do BDB? I'd like to try both GDBM and BDB.

I've quickly skimmed both manuals; it looks like GDBM provides no
support for transactions or journals. I'd like to be able to keep that
functionality without having to write it myself (at least in the initial
implementation).

SQL is optional to BDB on Linux builds. I can live without the SQL
support...

On Fri, 2014-07-11 at 12:24 +0800, Elias Mårtenson wrote:

> Would you be willing to move the CF stuff on top of GDBM? It's a much
> better fit for it, and will give you everything you do with SQLite
> today, without the extra overhead (SQLite is also not very efficient
> if you open multiple databases).
>
>
> Regards,
> Elias
>
>
> On 11 July 2014 12:21, David B. Lamkins <address@hidden> wrote:
>         On Thu, 2014-07-10 at 18:52 -0500, Blake McBride wrote:
>
>         >
>         >
>         > The component file system is for APL-only applications.  The
>         databse
>         > API is to communicate with the rest of the world.  It
>         depends on what
>         > you need.
>         >
>
>
>         and...
>
>         >
>         > The problem is that you are creating a component file system
>         that
>         > _depends_ on features of SQLite and is _needlessly_
>         dependent on those
>         > features.  I understand it works on other databases, but it
>         is an
>         > unreasonable burden on those systems.  You could just as
>         easily create
>         > the same system that doesn't depend on particulars of
>         SQLite.  That
>         > would work everywhere in a way that is reasonable to any of
>         the
>         > systems.
>         >
>
>
>         Ahhh... OK. Perhaps I've had an "aha!" moment. :)
>
>         If I understand you correctly, generalizing the current
>         component file
>         implementation to work with databases other than SQLite is
>         problematic
>         for all of the reasons you've discussed: poor generality, poor
>         use of
>         resources, and - in general - a poor fit to SQL.
>
>         Therefore, here's my proposal:
>
>         I'll drop all further work on generalizing the current
>         component file
>         API to work with SQL databases. IOW, the component file system
>         will
>         remain - literally - a component *file* system. The quick
>         patch that I
>         worked up last night will never become part of my code.
>
>         Programmers who want a way to persist data for an APL-only
>         single-user
>         application and a shallow learning curve and no sysadmin or
>         dbadmin
>         concerns will (I hope) appreciate and use the simple
>         component-file API.
>         Programmers who need the functionality of a real database -
>         and all that
>         it offers in terms of access control, concurrency and remote
>         access -
>         can use lib_sql with a PostgreSQL backend.
>
>         The presence of SQLite will remain an incidental
>         implementation detail
>         of the component file library. All further concerns specific
>         to the use
>         of SQLite as an implementation vehicle will be treated as
>         out-of-scope
>         for the project.
>
>         The current component file code can serve as a reference
>         implementation
>         to validate the design choices not implicitly required by the
>         ISO APL
>         Annex. There are a number of such choices that I was forced to
>         make in
>         the absence of detailed guidance from the spec. For example:
>         the choice
>         to ensure that component IDs assigned by CF_APPEND are always
>         nondecreasing for a given component file. There are many other
>         detailed
>         choices that had to be made. If you have a concern with how
>         I've
>         implemented the API to meet the spec, we can work out the
>         details
>         together.
>
>         Meanwhile, if someone feels that it's important to layer a
>         component
>         file API on top of lib_sql *and* allow a choice of database
>         backends,
>         they may either start from scratch or fork the existing code.
>         Personally, I don't see the need for this particular layering
>         of
>         abstractions; for anything more complex than an
>         integer-indexed file of
>         APL objects, I'd prefer to work directly with lib_sql. But if
>         someone
>         else sees value in the layered approach, by all means pursue
>         it as a
>         separate project.
>
>         Meanwhile, I'll continue working on design sketches for a
>         component file
>         system that doesn't build upon SQLite. The primary reason for
>         doing so
>         will be to increase performance while not requiring use of
>         explicit
>         transactions to reduce single-record insert times below the
>         100ms mark.
>         Of course the content of the component file will need to
>         remain reliable
>         in the face of software and hardware failures; that's one of
>         the
>         implementation attributes that I got "for free" by choosing
>         SQLite as a
>         short-term implementation tactic. An alternative
>         implementation will be
>         challenging, no doubt, but worth the effort in the long run.
>         Whether
>         such an implementation should be done as a native library or
>         as a hybrid
>         is not clear to me at this time.
>
>         The take-away from all of this should be the following: if you
>         use the
>         existing component file library (and I certainly hope that you
>         will do
>         so in cases where it best serves your needs), you can be
>         assured that
>         any future work that I do with the component file library will
>         preserve
>         your data. Any change in the file format will be accompanied
>         by a
>         utility workspace to migrate your existing component files.
>
>         >
>         > This is the most I've been involved with a community
>         project.  Fun!
>         >
>
>
>         Indeed. :)
>
>         Also, it's worth once again acknowledging Jüergen's work in
>         making all
>         of this possible. All of these little libraries and utilities
>         are
>         penny-ante stuff compared to the core interpreter. I just want
>         to say,
>         again, that I am grateful for the ability to work with a
>         full-featured
>         open-source APL interpreter for the first time ever. If you
>         look at the
>         history of free APL systems, they've traditionally been thin
>         on the
>         ground, undermaintained and not very interesting.
>
>         A lot of technologies have developed or matured in the past 30
>         years to
>         get to the point where we are today with free APL: Unicode,
>         Linux,
>         Emacs, gcc, fast hires video and the x86 hardware platform
>         being
>         foremost among a plethora of candidates. None of us would be
>         here on
>         this mailing list today, however, had Jüergen not put in an
>         immense
>         amount of design and implementation effort to create GNU APL.
>         I
>         sincerely appreciate his continuing support and involvement.
>
>
>