Re: FFI in Emacs

[Aurélien Aptel]

Hi all,

I've been doing some research in the last few weeks since I wasn't
familiar with all the details that goes into implementing a FFI.

First, you need to be able make arbitrary call to any function in a
shared library that your OS supports (.so, .dll, etc), so:

a) load the shared library dynamically
b) get the address of any symbol in it (be it a global var or a function).

Then, to call a function:
c) load the argument according to user-provided type and ABI
d) make the call
e) retrieve eventual return value (also depends on ABI)

Steps a and b can be done portably using libltdl [1]; c,d and e with
libffi [2] or libffcall [3].

So far so good. Once you have that, you have to expose the feature to
Elisp interpreter. You have to take into account that:
a) a function can allocate memory that has to be freed
b) a function can write in its arguments e.g. a buffer address and its
size (like read(2))
c) Lisp_Object have to be converted to native type (and vice versa)
d) a type can also be structs or unions

Still do-able but as you can see it gets tricky. Also:
- native int type can't be stored as a Lisp int because of the type
bits (problem c)
- any mistake using the FFI can lead to a crash (it is C after all)
- performance cost due to the dynamic call and all the conversion

Finally, we have to agree on:
a) the API
b) making it Stallman-compatible (GPL issues, etc) :-)

I'm sure everyone agree that we should implement an existing API, but which one?
Are we sure Stefan's gpl_compatible symbol solution sufficient for problem b?

Writing a complete wrapper for a slightly complex library can be
tedious and probably easier using directly C. I think emacs needs both
a FFI and a way to load dynamically compiled modules.
Although it can be redundant, I see it as complementary. The FFI can
be simple and not handle complex cases, use it if you need to call a
small set of simple functions, one-off jobs, etc. Write a module for
the rest and use tools like SWIG [4] made to generate binding/wrapper
for a library if you think it fits the job.

Dave Love already wrote a patch to write/load compiled modules for
Emacs 22 [5] which I have not been able to apply (or test, obviously)
on the official Emacs 22 source release. I've tried to fix it but I'm
stuck in autoconf hell which I'm not very familiar with. If someone
managed to try it, please post the steps you used. Anyway it can be a
good starting point.

I'm interested in working seriously on this, possibly for the next
Google Summer of Code.

I'm waiting for your suggestions and comments.

1: http://www.gnu.org/software/libtool/manual/html_node/Using-libltdl.html
2: http://sourceware.org/libffi/
3: http://www.gnu.org/software/libffcall/
4: http://www.swig.org/
5: http://www.loveshack.ukfsn.org/emacs/dynamic-loading/