|
| From: | Blake McBride |
| Subject: | Re: [Bug-apl] ISO Component File API preview |
| Date: | Fri, 11 Jul 2014 09:15:00 -0500 |
No, I used the correct acronyms in my email. DBM is a general term referring to (at least) three different products: dbm, ndbm and gdbm. All of these use a similar API.BDB is a different product, and has similarities with DBM in the sense that it too is a key/value store. It has a different API (although it is generally similar) and is a bigger product.DBM is certainly much more lightweight and is more like using "plain files" if you like.Regards,EliasOn 11 July 2014 13:10, David B. Lamkins <address@hidden> wrote:Did you mean to name BDB in the second sentence, or is that a limitation
of DBM as compared to GDBM. Either way, thanks. I'll start with BDB and
compare it to SQLite.
On Fri, 2014-07-11 at 13:07 +0800, Elias Mårtenson wrote:
> BDB is much heavier than DBM (of which GDBM is an implementation). DBM
> only allows a single process to open the database for writing.
>
>
> I'd like to start with DBM support, as it is simple and efficient, and
> also because I have more direct experience with it. BDB is cool
> though, and I can add that as a second phase.
>
>
> Regards,
> Elias
>
>
> On 11 July 2014 13:04, David B. Lamkins <address@hidden> wrote:
> Could you also do BDB? I'd like to try both GDBM and BDB.
>
> I've quickly skimmed both manuals; it looks like GDBM provides
> no
> support for transactions or journals. I'd like to be able to
> keep that
> functionality without having to write it myself (at least in
> the initial
> implementation).
>
> SQL is optional to BDB on Linux builds. I can live without the
> SQL
> support...
>
>
> On Fri, 2014-07-11 at 12:24 +0800, Elias Mårtenson wrote:
>
> > Would you be willing to move the CF stuff on top of GDBM?
> It's a much
> > better fit for it, and will give you everything you do with
> SQLite
> > today, without the extra overhead (SQLite is also not very
> efficient
> > if you open multiple databases).
> >
> >
> > Regards,
> > Elias
> >
> >
> > On 11 July 2014 12:21, David B. Lamkins <address@hidden>
> wrote:
> > On Thu, 2014-07-10 at 18:52 -0500, Blake McBride
> wrote:
> >
> > >
> > >
> > > The component file system is for APL-only
> applications. The
> > databse
> > > API is to communicate with the rest of the world.
> It
> > depends on what
> > > you need.
> > >
> >
> >
> > and...
> >
> > >
> > > The problem is that you are creating a component
> file system
> > that
> > > _depends_ on features of SQLite and is
> _needlessly_
> > dependent on those
> > > features. I understand it works on other
> databases, but it
> > is an
> > > unreasonable burden on those systems. You could
> just as
> > easily create
> > > the same system that doesn't depend on particulars
> of
> > SQLite. That
> > > would work everywhere in a way that is reasonable
> to any of
> > the
> > > systems.
> > >
> >
> >
> > Ahhh... OK. Perhaps I've had an "aha!" moment. :)
> >
> > If I understand you correctly, generalizing the
> current
> > component file
> > implementation to work with databases other than
> SQLite is
> > problematic
> > for all of the reasons you've discussed: poor
> generality, poor
> > use of
> > resources, and - in general - a poor fit to SQL.
> >
> > Therefore, here's my proposal:
> >
> > I'll drop all further work on generalizing the
> current
> > component file
> > API to work with SQL databases. IOW, the component
> file system
> > will
> > remain - literally - a component *file* system. The
> quick
> > patch that I
> > worked up last night will never become part of my
> code.
> >
> > Programmers who want a way to persist data for an
> APL-only
> > single-user
> > application and a shallow learning curve and no
> sysadmin or
> > dbadmin
> > concerns will (I hope) appreciate and use the simple
> > component-file API.
> > Programmers who need the functionality of a real
> database -
> > and all that
> > it offers in terms of access control, concurrency
> and remote
> > access -
> > can use lib_sql with a PostgreSQL backend.
> >
> > The presence of SQLite will remain an incidental
> > implementation detail
> > of the component file library. All further concerns
> specific
> > to the use
> > of SQLite as an implementation vehicle will be
> treated as
> > out-of-scope
> > for the project.
> >
> > The current component file code can serve as a
> reference
> > implementation
> > to validate the design choices not implicitly
> required by the
> > ISO APL
> > Annex. There are a number of such choices that I was
> forced to
> > make in
> > the absence of detailed guidance from the spec. For
> example:
> > the choice
> > to ensure that component IDs assigned by CF_APPEND
> are always
> > nondecreasing for a given component file. There are
> many other
> > detailed
> > choices that had to be made. If you have a concern
> with how
> > I've
> > implemented the API to meet the spec, we can work
> out the
> > details
> > together.
> >
> > Meanwhile, if someone feels that it's important to
> layer a
> > component
> > file API on top of lib_sql *and* allow a choice of
> database
> > backends,
> > they may either start from scratch or fork the
> existing code.
> > Personally, I don't see the need for this particular
> layering
> > of
> > abstractions; for anything more complex than an
> > integer-indexed file of
> > APL objects, I'd prefer to work directly with
> lib_sql. But if
> > someone
> > else sees value in the layered approach, by all
> means pursue
> > it as a
> > separate project.
> >
> > Meanwhile, I'll continue working on design sketches
> for a
> > component file
> > system that doesn't build upon SQLite. The primary
> reason for
> > doing so
> > will be to increase performance while not requiring
> use of
> > explicit
> > transactions to reduce single-record insert times
> below the
> > 100ms mark.
> > Of course the content of the component file will
> need to
> > remain reliable
> > in the face of software and hardware failures;
> that's one of
> > the
> > implementation attributes that I got "for free" by
> choosing
> > SQLite as a
> > short-term implementation tactic. An alternative
> > implementation will be
> > challenging, no doubt, but worth the effort in the
> long run.
> > Whether
> > such an implementation should be done as a native
> library or
> > as a hybrid
> > is not clear to me at this time.
> >
> > The take-away from all of this should be the
> following: if you
> > use the
> > existing component file library (and I certainly
> hope that you
> > will do
> > so in cases where it best serves your needs), you
> can be
> > assured that
> > any future work that I do with the component file
> library will
> > preserve
> > your data. Any change in the file format will be
> accompanied
> > by a
> > utility workspace to migrate your existing component
> files.
> >
> > >
> > > This is the most I've been involved with a
> community
> > project. Fun!
> > >
> >
> >
> > Indeed. :)
> >
> > Also, it's worth once again acknowledging Jüergen's
> work in
> > making all
> > of this possible. All of these little libraries and
> utilities
> > are
> > penny-ante stuff compared to the core interpreter. I
> just want
> > to say,
> > again, that I am grateful for the ability to work
> with a
> > full-featured
> > open-source APL interpreter for the first time ever.
> If you
> > look at the
> > history of free APL systems, they've traditionally
> been thin
> > on the
> > ground, undermaintained and not very interesting.
> >
> > A lot of technologies have developed or matured in
> the past 30
> > years to
> > get to the point where we are today with free APL:
> Unicode,
> > Linux,
> > Emacs, gcc, fast hires video and the x86 hardware
> platform
> > being
> > foremost among a plethora of candidates. None of us
> would be
> > here on
> > this mailing list today, however, had Jüergen not
> put in an
> > immense
> > amount of design and implementation effort to create
> GNU APL.
> > I
> > sincerely appreciate his continuing support and
> involvement.
> >
> >
> >
>
>
>
>
>
| [Prev in Thread] | Current Thread | [Next in Thread] |