Re: [Bug-apl] ISO Component File API preview

[David B. Lamkins]

I'm certainly willing to give GDBM a try as a replacement for SQLite.

Does SQLite bog down with multiple databases? Any idea why? A global
lock, perhaps...?

On Fri, 2014-07-11 at 12:24 +0800, Elias Mårtenson wrote:
> Would you be willing to move the CF stuff on top of GDBM? It's a much
> better fit for it, and will give you everything you do with SQLite
> today, without the extra overhead (SQLite is also not very efficient
> if you open multiple databases).
> 
> 
> Regards,
> Elias
> 
> 
> On 11 July 2014 12:21, David B. Lamkins <address@hidden> wrote:
>         On Thu, 2014-07-10 at 18:52 -0500, Blake McBride wrote:
>         
>         >
>         >
>         > The component file system is for APL-only applications.  The
>         databse
>         > API is to communicate with the rest of the world.  It
>         depends on what
>         > you need.
>         >
>         
>         
>         and...
>         
>         >
>         > The problem is that you are creating a component file system
>         that
>         > _depends_ on features of SQLite and is _needlessly_
>         dependent on those
>         > features.  I understand it works on other databases, but it
>         is an
>         > unreasonable burden on those systems.  You could just as
>         easily create
>         > the same system that doesn't depend on particulars of
>         SQLite.  That
>         > would work everywhere in a way that is reasonable to any of
>         the
>         > systems.
>         >
>         
>         
>         Ahhh... OK. Perhaps I've had an "aha!" moment. :)
>         
>         If I understand you correctly, generalizing the current
>         component file
>         implementation to work with databases other than SQLite is
>         problematic
>         for all of the reasons you've discussed: poor generality, poor
>         use of
>         resources, and - in general - a poor fit to SQL.
>         
>         Therefore, here's my proposal:
>         
>         I'll drop all further work on generalizing the current
>         component file
>         API to work with SQL databases. IOW, the component file system
>         will
>         remain - literally - a component *file* system. The quick
>         patch that I
>         worked up last night will never become part of my code.
>         
>         Programmers who want a way to persist data for an APL-only
>         single-user
>         application and a shallow learning curve and no sysadmin or
>         dbadmin
>         concerns will (I hope) appreciate and use the simple
>         component-file API.
>         Programmers who need the functionality of a real database -
>         and all that
>         it offers in terms of access control, concurrency and remote
>         access -
>         can use lib_sql with a PostgreSQL backend.
>         
>         The presence of SQLite will remain an incidental
>         implementation detail
>         of the component file library. All further concerns specific
>         to the use
>         of SQLite as an implementation vehicle will be treated as
>         out-of-scope
>         for the project.
>         
>         The current component file code can serve as a reference
>         implementation
>         to validate the design choices not implicitly required by the
>         ISO APL
>         Annex. There are a number of such choices that I was forced to
>         make in
>         the absence of detailed guidance from the spec. For example:
>         the choice
>         to ensure that component IDs assigned by CF_APPEND are always
>         nondecreasing for a given component file. There are many other
>         detailed
>         choices that had to be made. If you have a concern with how
>         I've
>         implemented the API to meet the spec, we can work out the
>         details
>         together.
>         
>         Meanwhile, if someone feels that it's important to layer a
>         component
>         file API on top of lib_sql *and* allow a choice of database
>         backends,
>         they may either start from scratch or fork the existing code.
>         Personally, I don't see the need for this particular layering
>         of
>         abstractions; for anything more complex than an
>         integer-indexed file of
>         APL objects, I'd prefer to work directly with lib_sql. But if
>         someone
>         else sees value in the layered approach, by all means pursue
>         it as a
>         separate project.
>         
>         Meanwhile, I'll continue working on design sketches for a
>         component file
>         system that doesn't build upon SQLite. The primary reason for
>         doing so
>         will be to increase performance while not requiring use of
>         explicit
>         transactions to reduce single-record insert times below the
>         100ms mark.
>         Of course the content of the component file will need to
>         remain reliable
>         in the face of software and hardware failures; that's one of
>         the
>         implementation attributes that I got "for free" by choosing
>         SQLite as a
>         short-term implementation tactic. An alternative
>         implementation will be
>         challenging, no doubt, but worth the effort in the long run.
>         Whether
>         such an implementation should be done as a native library or
>         as a hybrid
>         is not clear to me at this time.
>         
>         The take-away from all of this should be the following: if you
>         use the
>         existing component file library (and I certainly hope that you
>         will do
>         so in cases where it best serves your needs), you can be
>         assured that
>         any future work that I do with the component file library will
>         preserve
>         your data. Any change in the file format will be accompanied
>         by a
>         utility workspace to migrate your existing component files.
>         
>         >
>         > This is the most I've been involved with a community
>         project.  Fun!
>         >
>         
>         
>         Indeed. :)
>         
>         Also, it's worth once again acknowledging Jüergen's work in
>         making all
>         of this possible. All of these little libraries and utilities
>         are
>         penny-ante stuff compared to the core interpreter. I just want
>         to say,
>         again, that I am grateful for the ability to work with a
>         full-featured
>         open-source APL interpreter for the first time ever. If you
>         look at the
>         history of free APL systems, they've traditionally been thin
>         on the
>         ground, undermaintained and not very interesting.
>         
>         A lot of technologies have developed or matured in the past 30
>         years to
>         get to the point where we are today with free APL: Unicode,
>         Linux,
>         Emacs, gcc, fast hires video and the x86 hardware platform
>         being
>         foremost among a plethora of candidates. None of us would be
>         here on
>         this mailing list today, however, had Jüergen not put in an
>         immense
>         amount of design and implementation effort to create GNU APL.
>         I
>         sincerely appreciate his continuing support and involvement.
>         
> 
>