[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-apl] using c libs in apl?

From: Elias Mårtenson
Subject: Re: [Bug-apl] using c libs in apl?
Date: Fri, 10 Feb 2017 10:57:57 +0800

The main thing LIBFFI does is to provide a mechanism by which you can call functions and pass parameters in the correct way without having to hard-code information about the calling conventions of the platform you're running on.

Suppose you have a pointer to a function with the following signature:

    typedef struct {
        int a;
        char *b;
    } Foo;

    Foo bar(Foo arg1);

How would you call the function? Remember that inside GNU APL you will not have access to the signature. Every platform has a different idea how a struct is to be passed to and returned from a function. This is what LIBFFI provides.

Adding FFI support for GNU APL is a matter of exposing the LIBFFI API to the APL level. My example is one way of doing that. Xiao-Yong provided a different proposal based on the J API.

No matter what API we settle on, we can be certain of one this: It's not going to be beautiful. Making a single C function call is still going to require several lines of code. Not because LIBFFI is bad, but because the actual problem being solved is complex.

However, even if it's complex I much prefer to be able to write integration code in APL than C, since the former can be more easily distributed.

However, there is one snag: Take the Foo struct as an example. If you were to allocate memory for that struct, how many bytes do you need?

The question was a trick question. It depends on the hardware, and not just the bitness of the platform. On the x64 platform, “a” will be 4 bytes and “b” will be 8 bytes for a total of 12 bytes. On SPARC the sizes will be the same, but “b” requires 64-bit alignment, meaning that the entire struct will take up 16 bytes, with “b” having an offset of 16, not 8 as was the case on x64.

The problem here is that the only place where the information about this is encoded is in the C header files. There are two ways of dealing with this:
  1. Hard-code the offsets and accept the fact that we're stuck with some very struct dependencies on not just the architecture but also assuming binary compatibility with the version of the library that you link to.
  2. Create a tool that automatically computes the struct offsets and makes this information available to the APL code. In CFFI this tool is called the groveller and it's implemented by automatically calling the C compiler behind the scenes. For more information how it works there, you can check the documentation on the CFFI groveller: https://common-lisp.net/project/cffi/manual/html_node/The-Groveller.html

On 9 February 2017 at 22:30, Juergen Sauermann <address@hidden> wrote:
Hi Elias,

I understand. I recently had a look at how Erlang is building their API to other libraries. Apparently they use an
opaque context in which all memory allocations they need are recorded and the memory is freed when the context
is destroyed. The context is passed as first argument with every API function call.

I would also assume that memory which is malloc'ed by the library for their own purpose is also freed by the library.
So the only memory allocations to take care of are the function arguments and results of the library functions that
are called. Actually I was hoping that libffi would solve this kind of issues, because this sounds very much like a
standard problem?

/// Jürgen

On 02/09/2017 10:05 AM, Elias Mårtenson wrote:
I wasn't referring to the management of APL memory, but rather native memory used when calling functions through the FFI.

As an example, I've been recently working on integrating GSSAPI in Emacs (I've previously integrated the same library in Common Lisp), and as an example, let's take a look at a typical C function call in GSSAPI:

A simple function is gss_display_name() which is used to retrieve the name of a principal as a string, given the principal object (returned from a previous function call). Here is the signature:

OM_uint32 gss_display_name(
    OM_uint32       *minor_status,
    name_t           input_name,
    gss_buffer_desc *output_name_buffer,
    gss_OID         *output_name_type);

Here's how you use the function, assuming the name is in the variable ‘name’:

    gss_buffer_desc out;
    gss_OID out_type;

    int minor;
    int major = gss_display_name(&minor, name, &out, &out_type);
    if(GSS_ERROR(major)) {
        // there was an error, and the details about the error can be found in major and minor

    // The name is now available in a string located at out.value with the length out.length
    // Making this extra complicated is that out.value is not nul-terminated.
    char *name_as_string = malloc(out.length + 1);
    strncpy(name_as_string, out.value, out.length);
    name_as_string[out.length] = 0;

    // We now have the name as a string in the variable name_as_string.
    // There are some other API calls needed to release the memory allocated by the call to gss_display_name
    // but I'm ignoring that for the purpose of the example.

All right, with this in mind, we'll have to figure out an APL API that allows me to do this, and even more complex stuff. It's possible, but not easy. Here's an attempt at doing so that I'm just typing out as I see it just to have something discuss around:

  ⍝ The size of a gss_buffer_desc consists of 2 pointers,
  ⍝ which makes it 16 bytes on 64-bit platforms and 8 bytes
  ⍝ on 32-bit platforms.
  gss_buffer_desc_size ← 16
  out ← ⎕FFI_Alloc gss_buffer_desc_size

  ⍝ The gss_OID type is just a pointer, so 8 bytes on 64-bit platforms
  gss_OID_size ← 8
  out_type ← ⎕FFI_Alloc gss_OID_size

  minor ← ⎕FFI_Alloc 4    ⍝ 32-bit number

  ⍝ We need to specify the datatype of the return value, so we'll use an
  ⍝ axis argument for that.
  major ← 'gss_display_name' ⎕FFI_Call minor[Type_Int32] name out out_type

  ⍝ The C macro GSS_ERROR expands to some bit-fiddling,
  ⍝ but it's nothing we can't deal with in APL. There is
  ⍝ an error if any of the most-significant 16 bits are set.
  is_error ← 0 ≠ +/((32⍴2)⊤4294901760) ∧ (32⍴2)⊤major

  ⍝ Extract a pointer from 8 bytes after the top of the struct that out points to
  out_value ← 8 ⎕FFI_Dereference_Pointer out

  ⍝ Extract a 64-bit number from the top of the struct
  out_length ← 0 ⎕FFI_Dereference_Int64 out

  ⍝ Construct an APL string from an array of UTF-8 characters.
  ⍝ The idea here is that the left argument specifies the number of bytes to
  ⍝ copy, and if the function is called monadically it will simply copy
  ⍝ until a terminating NUL byte.
  name_as_string ← out_length ⎕FFI_MakeString out_value

  ⍝ Finally, free the memory we allocated previously
  ⎕FFI_Free out
  ⎕FFI_Free out_type

I don't think this can be made much simpler, and this is a reasonably simple real-world example of C API's that one needs to call. I've adopted this example from my Common Lisp code, and if you want to look at how it's done there you're welcome to look at that code: https://github.com/lokedhs/cl-gss/blob/master/src/cl-gss.lisp#L136


On 9 February 2017 at 00:50, Juergen Sauermann <address@hidden> wrote:
Hi Elias,

the latest libapl API (libapl.h) may give some ideas. It uses a 2-step approach like GNU APL internally: first
create a value with a given shape/rank and then set the elements of its ravel. The value must be released
explicitly when no longer needed. This is because libapl is a C interface not a C++ interface. Therefore the
Value_P magic cannot be used in a C library. In C++ things are much simpler because you could use
Value_P objects, which release the underlying APL value automatically.

/// Jürgen

On 02/08/2017 05:33 PM, Elias Mårtenson wrote:
This is something I might want to take a look at. I think the most difficult part of implementing this is to decide on a nice way to map the libffi API to APL in a natural way.

I'm thinking of providing a quad-function that allows you to declare a C function and their arguments (and associated types). That way you don't have to mess with datatypes when it comes to actually calling the native functions.

Still, you need to have constructs that allows you to allocate memory, as well as functions to access the content of said memory. I have no idea how such an API should look in APL.

I might take a look at this, but right now I'm working on some other projects so I don't have time. 


On 8 Feb 2017 23:05, "Juergen Sauermann" <address@hidden> wrote:

I had a quick look at both the C code from the www.jsoftware.com <http://www.jsoftware.com/help/user/call_procedure.htm> and fromhttps://github.com/libffi/libffi <https://github.com/libffi/libffi>
My first impression is that the former is quite hack-ish.

But I haven't worked with libffi myself, so I cant really say if it lives up to its promises.
If it does then my vote would definitely be for *libffi*.

Another plus for *libffi* is that it is available as debian package.

/// Jürgen

On 02/08/2017 01:38 AM, Elias Mårtenson wrote:
This would be really neat to have, but as someone who has written a lot of FFI (foreign function interface) code in Common Lisp which has a very powerful such interface, there are a lot of nuances that needs to be covered in order to have a decent FFI.

For example, what if you need to call a function which accepts a struct as its first argument which contains a pointer to another struct which in turn has a list of unsigned integers of size Foo (defined with a typedef in a .h file of course). The second argument being a pointer to a callback function.

That just gives a small idea of the issues one would come across.

Thankfully there is a C library, libffi, which can help here. It's designed to assist when creating an FFI for something like GNU APL. I recommend anyone who considers taking up this project to investigate it.

libffi can be found here: https://github.com/libffi/libffi

I certainly would really like it if this was implemented.


On 8 Feb 2017 03:01, "Juergen Sauermann" <address@hidden <mailto:address@hiddenline.de>> wrote:

    Hi Xiao-Yong,

    I believe this could be achieved by a single "master"-native
    function which then loads the
    actual DLL as specified by the arguments of the master function.
    Referring to the example in
    link you gave below:

    *a=: 'kernel32 GetProfileStringA s *c *c *c *c s' b=:
    'windows';'device'; 'default'; (32$'z');32 a cd b
    |31|windows|device|default|HP LaserJet 4P/4MP,HPPCL5MS,LPT |32|

    This would become in GNU APL:

    *a← 'kernel32 GetProfileStringA s *c *c *c *c s' b← 'windows'
    'device' 'default' (32⍴'z') 32******'universal-dll-loader' ⎕FX 'cd'****** a cd b ⍝ dlopen("kernel32.dll"), dlsym("GetProfileStringA") on
    first access,****⍝ and call GetProfileStringA with argument b*

    The *universal-dll-loader.so* needs to be written only once and
    contains mainly the code from github below. That code somehow goes
    into the *native/template_F12.cc* code of GNU APL and thats it. If
    you need help doing this then please let me know. /// Jürgen

    On 02/07/2017 06:30 PM, Xiao-Yong Jin wrote:
    It would be nice if one doesn't need to write wrappers and the APL system can do the
    structure conversions within the APL interpreter.  In J, you can dlopen a library
    and pass J values directly without writing and compiling C, see


    and the relevant code is at


    It would simplify using external libraries a lot.

    On Feb 4, 2017, at 7:38 AM, Juergen Sauermann<address@hiddenonline.de>
    <mailto:address@hiddenline.de>  wrote:


    yes there is: native functions. You can load shared libraries and ⎕FX functions in
    them to be called from APL code. The src/native directory contains a few templates
    that you can use as a starting point and to call your favourite library from them.

    Of course you need to provide wrappers from/to APL values to/from the data
    structures expected or produced by the libraries.

    Coming back to your other problems, if you do not like the terminal I/O of GNU APL, then
    you can write your own one and call libapl from it. I have extended libapl recently, giving
    you the full functionality of GNU APL without the specific ways how it handles terminal IO.

    /// Jürgen

    On 02/04/2017 02:52 AM,address@hidden <mailto:address@hidden>  wrote:
    is there method for loading a c lib and using it in apl ?  cdecl?   like this in fpc?


reply via email to

[Prev in Thread] Current Thread [Next in Thread]