Re: [Chicken-users] Some questions about loading libraries

[Matt Gushee]

Hi, Felix--

Thanks for your reply. I have some clarifications and responses below ...

On Tue, Jan 15, 2013 at 12:42 PM, Felix <address@hidden> wrote:

> I am developing a project which I expect will involve a number of extension
> libraries, or plugins (a large number, many of them provided by third
> parties, if my project ever becomes popular). For several reasons (which I
> will explain on request if anyone is curious), I feel it is best *not* to
> implement these libraries as eggs. So I am trying to work out a reasonable
> method for deploying the libraries and loading them at runtime.

Sure. What does "plugin" mean, though? A dynamically loadable library?
I assume it is compiled code? Written in Scheme?

Actually, I have realized that my question conflates two similar but distinct issues. So here's what's going on, specifically: I am developing a web CMS that is intended to be somewhat like Drupal, only better. Yes, I'm probably quite insane, but at least you can't say I lack ambition ;-)

The immediate problem I'm working on is a database layer. And although I am currently supporting just one database, that one is Redis, and I am anticipating that some users will be more comfortable with a traditional RDBS, say PostgresQL or SQLite (just don't say the M word to me, 'kay?). So I would like to be able to build in the capability to choose different backends. And since it is often not possible for people to compile code in a web hosting environment, it should be possible to make the choice at runtime. In any case, the database code will be compiled Chicken libraries installed in a known location. So there shouldn't be any problem finding the libraries; the issue with this component is how to load one or another implementation of a unified interface on demand, and how to manage the namespace. The ideal (which I know doesn't work and is not supposed to work), would be something like:

  (module sappari-lib       ; main library for the framework

                    ....                        ; various exports

                    (let ((backend

                               (string->symbol

                                  (or

                                     (get-environment-variable "SAPPARI_DB")

                                     "redis")))

                        (case backend

                            ((redis) (import (prefix sappari-db-redis db:)))

                            ((pgsql) (import (prefix sappari-db-pgsql db:)))

                            ...

   

[BTW .... my understanding of the module system is a bit vague, but I've seen that the docs say something like "modules are purely a syntactic construct" ... so am I right in thinking that means that modules really don't exist at runtime ... and thus, if conditional imports were possible at all, they would only be possible at compile time?]

The other problem is loading the arbitrary extension libraries that users of my CMS can install at will from a huge online collection (there's my megalomania talking again ;). Although there would be a potentially infinite number of different extensions, [I think] they would really all be fancy HTTP request handlers. Here the main problem is just having a reliable way to find the appropriate library in the filesystem. Regarding compilation or lack thereof, my plan is to recommend compiling all libraries, but as mentioned above, I expect there are use cases where users would not be able to compile any code, so I would like to give them the option of installing a binary that could load the extensions as interpreted code--if it's not unreasonably difficult to do so.

                    

> 1) Is there a way to set an arbitrary search path, such that REQUIRE (or
> LOAD, or some such thing) will work when the library is in a non-standard
> location?

You could implement your own routine (probably based on "load"), which
scans a number of directories. "repository-path" is for eggs, and "require"
respects it. You can also add directories to "##sys#include-pathnames".
I think "require" will try this too.

I tried this, and on my system it only works for 'include', not for 'require'. BTW, I have Chicken 4.8.0 on Arch Linux. I compiled the package myself, but only in order to have the latest version; i.e. I didn't hack anything. There were no patches, and the build() function in the PKGBUILD script is:

build() {

  cd ${pkgname}-${pkgver}

  # parallel builds are not supported..

  export MAKEFLAGS="${MAKEFLAGS/-j?/}"

  # some yummy make options there. check out README in the tarball.

  make PLATFORM=linux PREFIX=/usr

}

Nothing unusual there, right?

 

> 2) Is it possible to load a library selected at runtime (via an environment
> variable, config file, command-line argument ... the exact method isn't
> that important) AND have the symbols defined in that library included in a
> module?

You can compute arbitrary file-paths and pass them to "load". To make
the symbols available, you have to make sure the import-libraries for
the modules are available. How do you use the symbols in the
libraries?  Do you evaluate code at runtime?

See above re: compilation/evaluation. As far as using symbols goes, I've haven't gotten to a detailed plan yet, but I think each plugin would register itself/its functions in a data structure established by the main library. With an open-ended collection of arbitrary components, I can't see any other sane way to do it.

Hm. You can copy the "C_path_to_executable(argv[ 0 ])" in "chicken.h",
and use it via the foreign-function interface" to get the pathname
of an executable.

Ah, thank you! I was planning to look for a function something like that, but had no idea what its name might be. I'll try it. Are there any non-obvious pitfalls I should be aware of in using this?

 

> 4) Is LOAD-RELATIVE broken? I wrote some test code to try to use that
> procedure, but as far as I can tell it behaves just like LOAD, i.e. any
> relative path I give it is determined relative to the current directory
> from which the program is invoked. Though I would note that the
> documentation isn't entirely clear to me: "loads FILE relative to the path
> of the currently loaded file." ...? What does "the currently loaded file"
> mean? I will be happy to post my code if this is not a known issue.

It means that if loading is chained, nested "load-relative" calls with
load files relative to the outer "load"/"load-relative" invocations. e.g. 

file a.scm:

...
(load "foo/b.scm")
...

file foo/b.scm:

...
(load-relative "c.scm")         ; <- will load "foo/c.scm"
...

Ohhhh, I see. This works for interpreted but not compiled code. At least that's how it is for me. Is that expected behavior? I didn't understand that from the docs.

 

Perhaps, if you describe the implementation of your plugins and the
way the code therein is invoked, we can give more detailed advice.

Okay, I'll be glad to describe it once I get to that stage, which should be within the next couple of months, if death or the short-term need for income don't get in the way.

Thanks for all the info!

--

Matt Gushee