Re: [Bug-apl] using c libs in apl?

[Xiao-Yong Jin]

I believe passing/returning struct is actually compiler dependent (man gcc and 
/struct-return).

What you are talking about implementation dependent type and alignment is a 
real problem.
Actually without fully preprocessing the headerfiles, we can never be sure 
about system/architecture dependent types.

In <sys/time.h>, we have
           struct timeval {
               time_t      tv_sec;     /* seconds */
               suseconds_t tv_usec;    /* microseconds */
           };
the size of time_t and suseconds_t are architecture dependent.  The only 
reliable way that I
know of is,
      gcc -x c -include stdio.h -include sys/time.h -o /tmp/tt - <<<'int main()
      {printf("%d %d %d\n",sizeof(struct 
timeval),sizeof(time_t),sizeof(suseconds_t));}' && /tmp/tt && rm /tmp/tt
On Linux, I got "16 8 8".
On MacOS, I got "16 8 4".
It's not easy to call a simple gettimeofday in a system independent way.
For a concrete example, for the sake of discussion, here are my straightforward 
way of doing it in J,
gtod=:3 :0                                   NB.for Linux
v=.mema 16                                   NB.hardcoded sizeof(struct timeval)
'libc.so.6 gettimeofday i * *'cd(<v);<<0     NB.hardcoded library name
z=.memr v,0 2 4                              NB.hardcoded type
memf v
z
)
gtod=:3 :0                                   NB.for MacOS
v=.mema 16
'libc.dylib gettimeofday i * *'cd(<v);<<0    NB.different library name
z=.({.,16bffffffff(17 b.){:)memr v,0 2 4     NB.bit and (17 b.) to only use the 
lower 4 bytes
memf v
)
So any workspace depending on these crude ffi api would be 
architecture/system/kernel/library dependent.
I guess the difference in the library name can be mitigated.
The difference in the size of the type of the implementation, however, needs to 
be hard coded.
I don't have a solution for this except reaching out to compilers.

My copy of libffi seems to support structure, so it might work across 
architectures, but you still need to know the size of each type in a struct 
(see info libffi 2.3.3 type example.

> On Feb 9, 2017, at 8:57 PM, Elias Mårtenson <address@hidden> wrote:
> 
> The main thing LIBFFI does is to provide a mechanism by which you can call 
> functions and pass parameters in the correct way without having to hard-code 
> information about the calling conventions of the platform you're running on.
> 
> Suppose you have a pointer to a function with the following signature:
> 
>     typedef struct {
>         int a;
>         char *b;
>     } Foo;
> 
>     Foo bar(Foo arg1);
> 
> How would you call the function? Remember that inside GNU APL you will not 
> have access to the signature. Every platform has a different idea how a 
> struct is to be passed to and returned from a function. This is what LIBFFI 
> provides.
> 
> Adding FFI support for GNU APL is a matter of exposing the LIBFFI API to the 
> APL level. My example is one way of doing that. Xiao-Yong provided a 
> different proposal based on the J API.
> 
> No matter what API we settle on, we can be certain of one this: It's not 
> going to be beautiful. Making a single C function call is still going to 
> require several lines of code. Not because LIBFFI is bad, but because the 
> actual problem being solved is complex.
> 
> However, even if it's complex I much prefer to be able to write integration 
> code in APL than C, since the former can be more easily distributed.
> 
> However, there is one snag: Take the Foo struct as an example. If you were to 
> allocate memory for that struct, how many bytes do you need?
> 
> The question was a trick question. It depends on the hardware, and not just 
> the bitness of the platform. On the x64 platform, “a” will be 4 bytes and “b” 
> will be 8 bytes for a total of 12 bytes. On SPARC the sizes will be the same, 
> but “b” requires 64-bit alignment, meaning that the entire struct will take 
> up 16 bytes, with “b” having an offset of 16, not 8 as was the case on x64.
> 
> The problem here is that the only place where the information about this is 
> encoded is in the C header files. There are two ways of dealing with this:
>       • Hard-code the offsets and accept the fact that we're stuck with some 
> very struct dependencies on not just the architecture but also assuming 
> binary compatibility with the version of the library that you link to.
>       • Create a tool that automatically computes the struct offsets and 
> makes this information available to the APL code. In CFFI this tool is called 
> the groveller and it's implemented by automatically calling the C compiler 
> behind the scenes. For more information how it works there, you can check the 
> documentation on the CFFI groveller: 
> https://common-lisp.net/project/cffi/manual/html_node/The-Groveller.html
> Regards,
> Elias
> 
> On 9 February 2017 at 22:30, Juergen Sauermann <address@hidden> wrote:
> Hi Elias,
> 
> I understand. I recently had a look at how Erlang is building their API to 
> other libraries. Apparently they use an
> opaque context in which all memory allocations they need are recorded and the 
> memory is freed when the context
> is destroyed. The context is passed as first argument with every API function 
> call.
> 
> I would also assume that memory which is malloc'ed by the library for their 
> own purpose is also freed by the library.
> So the only memory allocations to take care of are the function arguments and 
> results of the library functions that
> are called. Actually I was hoping that libffi would solve this kind of 
> issues, because this sounds very much like a
> standard problem?
> 
> /// Jürgen
> 
> 
> On 02/09/2017 10:05 AM, Elias Mårtenson wrote:
>> I wasn't referring to the management of APL memory, but rather native memory 
>> used when calling functions through the FFI.
>> 
>> As an example, I've been recently working on integrating GSSAPI in Emacs 
>> (I've previously integrated the same library in Common Lisp), and as an 
>> example, let's take a look at a typical C function call in GSSAPI:
>> 
>> A simple function is gss_display_name() which is used to retrieve the name 
>> of a principal as a string, given the principal object (returned from a 
>> previous function call). Here is the signature:
>> 
>> OM_uint32 gss_display_name(
>>     OM_uint32       *minor_status,
>>     name_t           input_name,
>>     gss_buffer_desc *output_name_buffer,
>>     gss_OID         *output_name_type);
>> 
>> Here's how you use the function, assuming the name is in the variable ‘name’:
>> 
>>     gss_buffer_desc out;
>>     gss_OID out_type;
>> 
>>     int minor;
>>     int major = gss_display_name(&minor, name, &out, &out_type);
>>     if(GSS_ERROR(major)) {
>>         // there was an error, and the details about the error can be found 
>> in major and minor
>>     }
>> 
>>     // The name is now available in a string located at out.value with the 
>> length out.length
>>     // Making this extra complicated is that out.value is not nul-terminated.
>>     char *name_as_string = malloc(out.length + 1);
>>     strncpy(name_as_string, out.value, out.length);
>>     name_as_string[out.length] = 0;
>> 
>>     // We now have the name as a string in the variable name_as_string.
>>     // There are some other API calls needed to release the memory allocated 
>> by the call to gss_display_name
>>     // but I'm ignoring that for the purpose of the example.
>> 
>> All right, with this in mind, we'll have to figure out an APL API that 
>> allows me to do this, and even more complex stuff. It's possible, but not 
>> easy. Here's an attempt at doing so that I'm just typing out as I see it 
>> just to have something discuss around:
>> 
>>   ⍝ The size of a gss_buffer_desc consists of 2 pointers,
>>   ⍝ which makes it 16 bytes on 64-bit platforms and 8 bytes
>>   ⍝ on 32-bit platforms.
>>   gss_buffer_desc_size ← 16
>>   out ← ⎕FFI_Alloc gss_buffer_desc_size
>> 
>>   ⍝ The gss_OID type is just a pointer, so 8 bytes on 64-bit platforms
>>   gss_OID_size ← 8
>>   out_type ← ⎕FFI_Alloc gss_OID_size
>> 
>>   minor ← ⎕FFI_Alloc 4    ⍝ 32-bit number
>> 
>>   ⍝ We need to specify the datatype of the return value, so we'll use an
>>   ⍝ axis argument for that.
>>   major ← 'gss_display_name' ⎕FFI_Call minor[Type_Int32] name out out_type
>> 
>>   ⍝ The C macro GSS_ERROR expands to some bit-fiddling,
>>   ⍝ but it's nothing we can't deal with in APL. There is
>>   ⍝ an error if any of the most-significant 16 bits are set.
>>   is_error ← 0 ≠ +/((32⍴2)⊤4294901760) ∧ (32⍴2)⊤major
>> 
>>   ⍝ Extract a pointer from 8 bytes after the top of the struct that out 
>> points to
>>   out_value ← 8 ⎕FFI_Dereference_Pointer out
>> 
>>   ⍝ Extract a 64-bit number from the top of the struct
>>   out_length ← 0 ⎕FFI_Dereference_Int64 out
>> 
>>   ⍝ Construct an APL string from an array of UTF-8 characters.
>>   ⍝ The idea here is that the left argument specifies the number of bytes to
>>   ⍝ copy, and if the function is called monadically it will simply copy
>>   ⍝ until a terminating NUL byte.
>>   name_as_string ← out_length ⎕FFI_MakeString out_value
>> 
>>   ⍝ Finally, free the memory we allocated previously
>>   ⎕FFI_Free out
>>   ⎕FFI_Free out_type
>> 
>> I don't think this can be made much simpler, and this is a reasonably simple 
>> real-world example of C API's that one needs to call. I've adopted this 
>> example from my Common Lisp code, and if you want to look at how it's done 
>> there you're welcome to look at that code: 
>> https://github.com/lokedhs/cl-gss/blob/master/src/cl-gss.lisp#L136
>> 
>> Regards,
>> Elias
>> 
>> On 9 February 2017 at 00:50, Juergen Sauermann <address@hidden> wrote:
>> Hi Elias,
>> 
>> the latest libapl API (libapl.h) may give some ideas. It uses a 2-step 
>> approach like GNU APL internally: first
>> create a value with a given shape/rank and then set the elements of its 
>> ravel. The value must be released
>> explicitly when no longer needed. This is because libapl is a C interface 
>> not a C++ interface. Therefore the
>> Value_P magic cannot be used in a C library. In C++ things are much simpler 
>> because you could use
>> Value_P objects, which release the underlying APL value automatically.
>> 
>> /// Jürgen
>> 
>> 
>> On 02/08/2017 05:33 PM, Elias Mårtenson wrote:
>>> This is something I might want to take a look at. I think the most 
>>> difficult part of implementing this is to decide on a nice way to map the 
>>> libffi API to APL in a natural way.
>>> 
>>> I'm thinking of providing a quad-function that allows you to declare a C 
>>> function and their arguments (and associated types). That way you don't 
>>> have to mess with datatypes when it comes to actually calling the native 
>>> functions.
>>> 
>>> Still, you need to have constructs that allows you to allocate memory, as 
>>> well as functions to access the content of said memory. I have no idea how 
>>> such an API should look in APL.
>>> 
>>> I might take a look at this, but right now I'm working on some other 
>>> projects so I don't have time. 
>>> 
>>> Regards, 
>>> Elias  
>>> 
>>> On 8 Feb 2017 23:05, "Juergen Sauermann" <address@hidden> wrote:
>>> Hi,
>>> 
>>> I had a quick look at both the C code from the www.jsoftware.com 
>>> <http://www.jsoftware.com/help/user/call_procedure.htm> and 
>>> fromhttps://github.com/libffi/libffi <https://github.com/libffi/libffi>
>>> My first impression is that the former is quite hack-ish.
>>> 
>>> But I haven't worked with libffi myself, so I cant really say if it lives 
>>> up to its promises.
>>> If it does then my vote would definitely be for *libffi*.
>>> 
>>> Another plus for *libffi* is that it is available as debian package.
>>> 
>>> /// Jürgen
>>> 
>>> 
>>> On 02/08/2017 01:38 AM, Elias Mårtenson wrote:
>>> This would be really neat to have, but as someone who has written a lot of 
>>> FFI (foreign function interface) code in Common Lisp which has a very 
>>> powerful such interface, there are a lot of nuances that needs to be 
>>> covered in order to have a decent FFI.
>>> 
>>> For example, what if you need to call a function which accepts a struct as 
>>> its first argument which contains a pointer to another struct which in turn 
>>> has a list of unsigned integers of size Foo (defined with a typedef in a .h 
>>> file of course). The second argument being a pointer to a callback function.
>>> 
>>> That just gives a small idea of the issues one would come across.
>>> 
>>> Thankfully there is a C library, libffi, which can help here. It's designed 
>>> to assist when creating an FFI for something like GNU APL. I recommend 
>>> anyone who considers taking up this project to investigate it.
>>> 
>>> libffi can be found here: https://github.com/libffi/libffi
>>> 
>>> I certainly would really like it if this was implemented.
>>> 
>>> Regards,
>>> Elias
>>> 
>>> On 8 Feb 2017 03:01, "Juergen Sauermann" <address@hidden 
>>> <mailto:address@hidden>> wrote:
>>> 
>>>     Hi Xiao-Yong,
>>> 
>>>     I believe this could be achieved by a single "master"-native
>>>     function which then loads the
>>>     actual DLL as specified by the arguments of the master function.
>>>     Referring to the example in
>>>     link you gave below:
>>> 
>>>     *a=: 'kernel32 GetProfileStringA s *c *c *c *c s' b=:
>>>     'windows';'device'; 'default'; (32$'z');32 a cd b
>>>     +--+-------+------+-------+--------------------------------+--+
>>>     |31|windows|device|default|HP LaserJet 4P/4MP,HPPCL5MS,LPT |32|
>>>     +--+-------+------+-------+--------------------------------+--+***
>>> 
>>>     This would become in GNU APL:
>>> 
>>> 
>>>     *a← 'kernel32 GetProfileStringA s *c *c *c *c s' b← 'windows'
>>>     'device' 'default' (32⍴'z') 32******'universal-dll-loader' ⎕FX 
>>> 'cd'****** a cd b ⍝ dlopen("kernel32.dll"), dlsym("GetProfileStringA") on
>>>     first access,****⍝ and call GetProfileStringA with argument b*
>>> 
>>>     The *universal-dll-loader.so* needs to be written only once and
>>>     contains mainly the code from github below. That code somehow goes
>>>     into the *native/template_F12.cc* code of GNU APL and thats it. If
>>>     you need help doing this then please let me know. /// Jürgen
>>> 
>>>     On 02/07/2017 06:30 PM, Xiao-Yong Jin wrote:
>>>     It would be nice if one doesn't need to write wrappers and the APL 
>>> system can do the
>>>     structure conversions within the APL interpreter.  In J, you can dlopen 
>>> a library
>>>     and pass J values directly without writing and compiling C, see
>>> 
>>>        http://www.jsoftware.com/help/user/call_procedure.htm
>>>     <http://www.jsoftware.com/help/user/call_procedure.htm>
>>> 
>>>     and the relevant code is at
>>> 
>>>        https://github.com/jsoftware/jsource/blob/master/jsrc/x15.c
>>>     <https://github.com/jsoftware/jsource/blob/master/jsrc/x15.c>
>>> 
>>>     It would simplify using external libraries a lot.
>>> 
>>>     On Feb 4, 2017, at 7:38 AM, Juergen Sauermann<address@hidden>
>>>     <mailto:address@hidden>  wrote:
>>> 
>>>     Hi,
>>> 
>>>     yes there is: native functions. You can load shared libraries and ⎕FX 
>>> functions in
>>>     them to be called from APL code. The src/native directory contains a 
>>> few templates
>>>     that you can use as a starting point and to call your favourite library 
>>> from them.
>>> 
>>>     Of course you need to provide wrappers from/to APL values to/from the 
>>> data
>>>     structures expected or produced by the libraries.
>>> 
>>>     Coming back to your other problems, if you do not like the terminal I/O 
>>> of GNU APL, then
>>>     you can write your own one and call libapl from it. I have extended 
>>> libapl recently, giving
>>>     you the full functionality of GNU APL without the specific ways how it 
>>> handles terminal IO.
>>> 
>>>     /// Jürgen
>>> 
>>> 
>>>     On 02/04/2017 02:52 AM,address@hidden <mailto:address@hidden>  wrote:
>>>     is there method for loading a c lib and using it in apl ?  cdecl?   
>>> like this in fpc?
>>> 
>>> 
>>>     ftp://ftp.freepascal.org/fpc/docs-pdf/CinFreePascal.pdf
>>>     <ftp://ftp.freepascal.org/fpc/docs-pdf/CinFreePascal.pdf>
>>>       
>>> 
>>> 
>>> 
>> 
>> 
> 
>