Re: [Bug-apl] ISO Component File API preview

[Blake McBride]

Dear David,

I will answer in-line below.

On Thu, Jul 10, 2014 at 3:16 PM, David Lamkins <address@hidden> wrote:

Hi Blake,

I feel that we're talking at cross purposes.

Yes, I thought of using those exact words.

 

What I've been trying to convey (and clearly have failed on every attempt so far) is that an APL component file system must be file-centric. Every APL component file system that I've worked with or have read about has created a file to contain its APL data components.

Conceptually, yes.  Actually, what goes on under the hood is implementation specific.  In our present case, each component file would equate to an SQL table - not an SQL database.  This is very important.  So, if your applications uses 15 component files, that would be represented as 15 SQL tables in one SQL database.

 

I know you don't like looking at the ISO document, but please: read Annex A before continuing this conversation. The Annex is concise and informal and describes what a component file system is expected to do.

Okay, I read the spec.  It is way too minimalistic IMO.  It would be best to make our component file system a super-set of that definition.

Their "Library" should map to an SQL database.

Their "file" should map to an SQL table.

 

My use of SQLite is an implementation detail - nothing more - underneath the covers of the component file system implementation. Had Elias never read the code, all you'd know is that my implementation does satisfy the requirements of the design outlined by the Annex of the ISO APL spec.

Satisfying the spec is insufficient.  Your satisfaction of the spec must also be reasonable.  It is unreasonable to store one table per SQL database.

 

Please, let's acknowledge that, as a baseline, a component *file* system requires *files*... Look at the component file systems as implemented by STSC, Dyalog, MicroAPL and others: they all store data on files; not database systems.

But *file* is an abstract notion.  The underlying implementation abstracts what is really going on.  On one hand, utilizing an SQL database for a component file system is over-kill, but it gives us everything we need to easily implement an APL component file system.  Creating a native component file system is likely more effecient than riding on top of a huge database engine.  That is why all of the vendors you quote did that.  The problem is that developing a native component file system is significantly more work than riding on top of SQL.

A component file system written in APL that rides on top of an SQL engine is reasonable.  A native component file system implementation would have to be written in something like C.  That would be somewhat unreasonable in APL.

Writing a native component file system is very significant compared to riding on top of an SQL engine.

 

I feel that strong assertions that this library *must* do things a certain way, when those assertions seem based upon a fundamental misunderstanding of what the library is designed to do, are not constructive.

If I were new to computers, and I told you I had a bunch of normal sized files, and that I intended to store each file on its own disk drive, would you tell me "that's wrong, you can and should store many files on one disk drive."?  I can argue as much as I like, but you know I'm wrong.

 

Perhaps I should have been more clear regarding the intent of the library at the start of this conversation; I incorrectly assumed that everyone involved had already familiarized themselves with the reference material and the implementation and was making suggestions about possible futures.

I've used many component file system over the years, and I've written my own file systems, so I am familiar with what they are in general terms.  Having now read the spec, it is even less than I would have expected.

I still believe that it may be useful to allow different database backends as an experimental conforming extension of the component file library. That, however, is not and has never been a design goal. I'm going to insist that we address the correctness and effectiveness of the library relative to the design expressed by the Annex before we proceed further with proposals for extensions.

Are you mapping one APL component file into one SQL database?  (This is what I understanding of how your system is working.  If you answer this one question, I can reply better.)

 

Thanks for your feedback so far. I do hope you'll make the time to evaluate this library in the context of Annex A of the ISO spec.

Just like you don't need to see the implementation if someone tells you they store one file per disk drive, I don't need to see your implementation (if you store one table per database), and how well it conforms, is irreverent.  Are you mapping one APL component file into one SQL database?  That is the key question.

I hope you don't take my directness as disrespectful.  I've seen some of your work, and I respect and appreciate it.

Thanks.

Blake

 

On Thu, Jul 10, 2014 at 11:45 AM, Blake McBride <address@hidden> wrote:

Dear David,

I haven't actually had time to look at your work yet.  I am going off of what Elias said, my fair amount of experience with SQL databases, my experience writing the keyed files system, and the comments you have made.  Forgive me if I am wrong, but based on your comments, it seems like you don't have a lot of experience with SQL databases.  

In SQL terminology, there are no 'files'.  A 'database' is analogous to a D: drive in Windows, or a partition, file system, or mount point on Unix.  What you would thing of as a 'file' is called a 'table'.  A database can have any number of tables.  You would never think of putting one table per database.  That would be like creating a new drive for each file.

Each table can have any number of rows (or records).  Each record in a single table will have the same data elements as the other records in that same table - sort of like a spreadsheet.  You can order or select by any row or any combinations of rows.  In practice, however, it is best to have an index setup for the more popular types of queries you will be doing.

SQLite keeps each database in a single Windows or Unix file.  PostgreSQL has a complex internal data representation made up of many Windows/Unix files.  Both of these  facts are totally abstracted away from you, and in fact shouldn't matter to you at all.

So, in implementing a component file system, it makes most sense to equate a single APL component file to a single SQL table.  Each component would be a record in an SQL table.  Ordering and access are handled automatically by the database.  Your implementation should not care much at all whether it is SQLite or PostgreSQL.

I hope this is helpful and not disrespectful.

Sincerely,

Blake

--

"The secret to creativity is knowing how to hide your sources."

   Albert Einstein

http://soundcloud.com/davidlamkins
http://reverbnation.com/lamkins
http://reverbnation.com/lcw
http://lamkins-guitar.com/
http://lamkins.net/
http://successful-lisp.com/