Re: [Chicken-users] [Swig-dev] Future of SWIG chicken module

[felix]

John Lenz wrote:
> > One more question: does SWIG support callbacks?
>
>
> Lastly, the reason swig does not allow just any scripting language  
> function to be passed to a C/C++ function that takes a function pointer  
> is that a callback wrapper function would need to be exported.  But  
> since chicken already supports generating this wrapper function with  
> (define-callback-wrapper ...) it would be easy to make the swig chicken  
> module completly support callbacks.  Again, swig itself does not  
> support exporting the callback wrapper function but we wouldn't need it  
> to :)

Well, it's not quite that easy, I'm afraid. To perform callbacks the
_caller_ has to be treated specially, since Chicken does some weird
stuff with the C-stack. So for example:

(define-external (callback) void ...)

((foreign-lambda* void () "callback();"))

would probably crash. It would have to be

((foreign-callback-lambda* void () "callback();"))

The latter form is much less efficient, unfortunately. The reason for
this is how Chicken implements tail-call optimization and first class
continuations.

> Every scripting language module in swig (including how the current  
> chicken module is implemented) generates all the wrapping functions.   
> This is neccissary for a lot of languages because to register a  
> function with the interpreter it must be in the form (for python)
> PyObject *_wrapWhatever(PyObject *self, PyObject *args) {...}
> SWIG then sets up this function to extract the C variables from the  
> args and self pointers and wrap up the return value into a PyObject *
>
> The current swig chicken module does the same thing... a function like
> void _wrapWhatever(int argc, C_word closure, C_word continuation,  
> C_word arg1) {...}
> and then the C interface is used (like C_fix C_unfix and such).  In the  
> current swig chicken module, chicken is viewed exactly as another  
> scripting language.
>
> So in any case, I notice that a lot of these wrapper functions can be  
> generated by chicken with say the (define-foreign-lambda ) or whatnot.   
> So my idea was to not have SWIG generate any wrapper functions at  
> all...  After thinking about it this past week I am having doubts as to  
> if this will work, but here is the idea anyway.
>
> Swig would take as input the header files and such and would generate a 
> . scm file that contained instructions to chicken about the interface.   
> This way we can completly avoid generating any wrapper functions at  all.

In fact, the wrapper functions will be generated anyway - only by Chicken
not by SWIG. `foreign-lambda' and friends generate wrappers themselves,
for various technical reasons.

> You see, the problem really comes down to 1) How types will be  
> converted and 2) how linkage will occur.  Since chicken itself is  
> exporting to C code, we have a lot more flexibility than a normal  
> scripting language.  By giving chicken a description of the interface,  
> the chicken generated code can make calls directly into the functions  
> being wrapped.
>
> The only problem is support for more complicated types, like say std:: 
> vector<int>.  SWIG has the ability to automaticly convert a function  
> that say returns a std::vector<int> into a scheme list (the guile swig  
> module already does).  This requires generating some wrapper  
> function... or at least some wrapper code after the call returns to  
> create the scheme list.  SWIG has support for what are called typemaps,  
> which allow arbitrary code to be inserted into the wrapper function to  
> transform the types.  So SWIG has a typemap that converts a std:: 
> vector<int> to a guile scheme list.
>
> Another problem involved with linkage is the runtime type handeling.  
> Usually, a struct or a class is stored in say the PyObject * or a  
> C_word as a void pointer and some type information.  Thus, in the  
> generated _wrap function we need to check is the input argument a valid  
> type (since we will eventually cast the pointer from a void * to  
> whatever type the function takes).  This is currently where both the  
> SWIG chicken module and the (foreign*) functions in chicken are BOTH  
> broken.  There is no type checking going on at all.  I can pass a  
> pointer of any type to any function which accepts a pointer. 

Well, not quite - the type-checking for `foreign-...' is done on the Scheme 
side.
But usually pointers are untyped, as you correctly point out. But IIRC Jonah 
has used
a special undocumented feature of Chicken ("tagged pointers") for that stuff.
But, as I said before, I haven't looked at the SWIG code too hard.

> SWIG has  
> a bunch of code to do run time type checking... is the input type the  
> same as the output type, or can the input type be converted to the  
> output type, etc. which would also be lost.  Lastly, even if the user  
> is careful and only passes correct types to correct functions, type  
> conversion is neccissary with multiple inheritance since the offset of  
> the base class inside the derived class might not always be at the  
> beginning, so a direct cast from a void * causes problems.
>
> For both of these two things we could obviously add some support for in  
> chicken, but we would in a sense be wasting all the time and  
> duplicating the support already in SWIG.  SWIG has an extensive amount  
> of code dealing with run time type checking... you can have typedefs of  
> derived classes and more typedefs and on and on and SWIG will figure it  
> all out.  SWIG has a lot of code for some of the more compilcated C++  
> types like std::vector and std::list as well.
>
> So maybe a combination of the two?
> Say chicken could have a directive or function or something that would  
> allow arbitrary code to be insterted before the actuall foreign call is  
> made?  This way, SWIG would parse the interface file and SWIG would  
> generate all the type conversion code (like the C_fix call on the  
> return value or the C_c_pointer_or_null() call on the input.  Thus,  
> SWIG would in essense do all the Chicken<->C interface stuff but  
> instead of dumping it all in wrapper functions, it would just tell  
> chicken about it... That is, SWIG would give chicken the exact C code  
> that needed to go before and after the call to the foreign function.   
> This way we still avoid generating wrapper functions... all the code  
> would in some sense be "inlined" right into the generated chicken c  code.
> Optionally as well, SWIG would only need to export that code for  
> "problem" types like pointers to foreign types, c++ types std::vector,  
> etc.  The base case like a function that takes integers and returns a  
> float or something could stay exactly how it is.
>
> Of course, the other option is to keep the current method of type  
> conversion and linkage, that is, SWIG generates a bunch of wrapper  
> functions and marshals every type from chicken->c and from c->chicken  
> without chicken knowing anything about it.

I think the current method is fine. The code generated by SWIG is not
necessarily less efficient, than what Chicken generates. There are some
points where the current SWIG Chicken module could be improved, by supporting
specific features (like constants, etc.). Also the type-checking of numeric
arguments is not fully optimal (fixnums should be accepted for flonum arguments,
for example). And multiple values are (AFAIC) not handled.
But otherwise it works fine.


cheers,
felix