Re: [Bug-apl] using c libs in apl?

[Xiao-Yong Jin]

Whether we use libffi or some hackish code, we need at least read, write, 
malloc, and free.
Type can certainly be an argument to some of those functions.
I don't think it can be any simpler.

J supplies the foreign conjunction, 
http://www.jsoftware.com/help/dictionary/dx015.htm
or in APL lingo, a system function.  The description of J's functions is at
http://www.jsoftware.com/help/user/memory_management.htm
The memr/memw/mema/memf is simply a rename of those 15!:N.
There is also error reporting, 
http://www.jsoftware.com/help/user/cd_domain_error.htm

If I were to replicate J's API, one ⎕FFI (an ambivalent operator) in GNU APL 
could do it.
One would package it in a separate workspace similar to ⎕FIO of course.
Here is a crude replication of a part of J API.

RESULT ← 'FILENAME PROCEDURE [>][+][%] DECLARATION' (0 ⎕FFI) PARAMETERS
DATA ← 1 ⎕FFI ADDRESS BYTE_OFFSET COUNT [TYPE]
RETURNCODE ← DATA (2 ⎕FFI) ADDRESS BYTE_OFFSET COUNT [TYPE]
ADDRESS ← 3 ⎕FFI LENGTH
RETURNCODE ← 4 ⎕FFI ADDRESS
ERRORCODE ← 10 ⎕FFI ''
ERRORMESSAGE ← 11 ⎕FFI ''

The description of those parameters follows J, see
http://www.jsoftware.com/help/user/dlls.htm

> On Feb 9, 2017, at 3:05 AM, Elias Mårtenson <address@hidden> wrote:
> 
> I wasn't referring to the management of APL memory, but rather native memory 
> used when calling functions through the FFI.
> 
> As an example, I've been recently working on integrating GSSAPI in Emacs 
> (I've previously integrated the same library in Common Lisp), and as an 
> example, let's take a look at a typical C function call in GSSAPI:
> 
> A simple function is gss_display_name() which is used to retrieve the name of 
> a principal as a string, given the principal object (returned from a previous 
> function call). Here is the signature:
> 
> OM_uint32 gss_display_name(
>     OM_uint32       *minor_status,
>     name_t           input_name,
>     gss_buffer_desc *output_name_buffer,
>     gss_OID         *output_name_type);
> 
> Here's how you use the function, assuming the name is in the variable ‘name’:
> 
>     gss_buffer_desc out;
>     gss_OID out_type;
> 
>     int minor;
>     int major = gss_display_name(&minor, name, &out, &out_type);
>     if(GSS_ERROR(major)) {
>         // there was an error, and the details about the error can be found 
> in major and minor
>     }
> 
>     // The name is now available in a string located at out.value with the 
> length out.length
>     // Making this extra complicated is that out.value is not nul-terminated.
>     char *name_as_string = malloc(out.length + 1);
>     strncpy(name_as_string, out.value, out.length);
>     name_as_string[out.length] = 0;
> 
>     // We now have the name as a string in the variable name_as_string.
>     // There are some other API calls needed to release the memory allocated 
> by the call to gss_display_name
>     // but I'm ignoring that for the purpose of the example.
> 
> All right, with this in mind, we'll have to figure out an APL API that allows 
> me to do this, and even more complex stuff. It's possible, but not easy. 
> Here's an attempt at doing so that I'm just typing out as I see it just to 
> have something discuss around:
> 
>   ⍝ The size of a gss_buffer_desc consists of 2 pointers,
>   ⍝ which makes it 16 bytes on 64-bit platforms and 8 bytes
>   ⍝ on 32-bit platforms.
>   gss_buffer_desc_size ← 16
>   out ← ⎕FFI_Alloc gss_buffer_desc_size
> 
>   ⍝ The gss_OID type is just a pointer, so 8 bytes on 64-bit platforms
>   gss_OID_size ← 8
>   out_type ← ⎕FFI_Alloc gss_OID_size
> 
>   minor ← ⎕FFI_Alloc 4    ⍝ 32-bit number
> 
>   ⍝ We need to specify the datatype of the return value, so we'll use an
>   ⍝ axis argument for that.
>   major ← 'gss_display_name' ⎕FFI_Call minor[Type_Int32] name out out_type
> 
>   ⍝ The C macro GSS_ERROR expands to some bit-fiddling,
>   ⍝ but it's nothing we can't deal with in APL. There is
>   ⍝ an error if any of the most-significant 16 bits are set.
>   is_error ← 0 ≠ +/((32⍴2)⊤4294901760) ∧ (32⍴2)⊤major
> 
>   ⍝ Extract a pointer from 8 bytes after the top of the struct that out 
> points to
>   out_value ← 8 ⎕FFI_Dereference_Pointer out
> 
>   ⍝ Extract a 64-bit number from the top of the struct
>   out_length ← 0 ⎕FFI_Dereference_Int64 out
> 
>   ⍝ Construct an APL string from an array of UTF-8 characters.
>   ⍝ The idea here is that the left argument specifies the number of bytes to
>   ⍝ copy, and if the function is called monadically it will simply copy
>   ⍝ until a terminating NUL byte.
>   name_as_string ← out_length ⎕FFI_MakeString out_value
> 
>   ⍝ Finally, free the memory we allocated previously
>   ⎕FFI_Free out
>   ⎕FFI_Free out_type
> 
> I don't think this can be made much simpler, and this is a reasonably simple 
> real-world example of C API's that one needs to call. I've adopted this 
> example from my Common Lisp code, and if you want to look at how it's done 
> there you're welcome to look at that code: 
> https://github.com/lokedhs/cl-gss/blob/master/src/cl-gss.lisp#L136
> 
> Regards,
> Elias
> 
> On 9 February 2017 at 00:50, Juergen Sauermann <address@hidden> wrote:
> Hi Elias,
> 
> the latest libapl API (libapl.h) may give some ideas. It uses a 2-step 
> approach like GNU APL internally: first
> create a value with a given shape/rank and then set the elements of its 
> ravel. The value must be released
> explicitly when no longer needed. This is because libapl is a C interface not 
> a C++ interface. Therefore the
> Value_P magic cannot be used in a C library. In C++ things are much simpler 
> because you could use
> Value_P objects, which release the underlying APL value automatically.
> 
> /// Jürgen
> 
> 
> On 02/08/2017 05:33 PM, Elias Mårtenson wrote:
>> This is something I might want to take a look at. I think the most difficult 
>> part of implementing this is to decide on a nice way to map the libffi API 
>> to APL in a natural way.
>> 
>> I'm thinking of providing a quad-function that allows you to declare a C 
>> function and their arguments (and associated types). That way you don't have 
>> to mess with datatypes when it comes to actually calling the native 
>> functions.
>> 
>> Still, you need to have constructs that allows you to allocate memory, as 
>> well as functions to access the content of said memory. I have no idea how 
>> such an API should look in APL.
>> 
>> I might take a look at this, but right now I'm working on some other 
>> projects so I don't have time. 
>> 
>> Regards, 
>> Elias  
>> 
>> On 8 Feb 2017 23:05, "Juergen Sauermann" <address@hidden> wrote:
>> Hi,
>> 
>> I had a quick look at both the C code from the www.jsoftware.com 
>> <http://www.jsoftware.com/help/user/call_procedure.htm> and 
>> fromhttps://github.com/libffi/libffi <https://github.com/libffi/libffi>
>> My first impression is that the former is quite hack-ish.
>> 
>> But I haven't worked with libffi myself, so I cant really say if it lives up 
>> to its promises.
>> If it does then my vote would definitely be for *libffi*.
>> 
>> Another plus for *libffi* is that it is available as debian package.
>> 
>> /// Jürgen
>> 
>> 
>> On 02/08/2017 01:38 AM, Elias Mårtenson wrote:
>> This would be really neat to have, but as someone who has written a lot of 
>> FFI (foreign function interface) code in Common Lisp which has a very 
>> powerful such interface, there are a lot of nuances that needs to be covered 
>> in order to have a decent FFI.
>> 
>> For example, what if you need to call a function which accepts a struct as 
>> its first argument which contains a pointer to another struct which in turn 
>> has a list of unsigned integers of size Foo (defined with a typedef in a .h 
>> file of course). The second argument being a pointer to a callback function.
>> 
>> That just gives a small idea of the issues one would come across.
>> 
>> Thankfully there is a C library, libffi, which can help here. It's designed 
>> to assist when creating an FFI for something like GNU APL. I recommend 
>> anyone who considers taking up this project to investigate it.
>> 
>> libffi can be found here: https://github.com/libffi/libffi
>> 
>> I certainly would really like it if this was implemented.
>> 
>> Regards,
>> Elias
>> 
>> On 8 Feb 2017 03:01, "Juergen Sauermann" <address@hidden 
>> <mailto:address@hidden>> wrote:
>> 
>>     Hi Xiao-Yong,
>> 
>>     I believe this could be achieved by a single "master"-native
>>     function which then loads the
>>     actual DLL as specified by the arguments of the master function.
>>     Referring to the example in
>>     link you gave below:
>> 
>>     *a=: 'kernel32 GetProfileStringA s *c *c *c *c s' b=:
>>     'windows';'device'; 'default'; (32$'z');32 a cd b
>>     +--+-------+------+-------+--------------------------------+--+
>>     |31|windows|device|default|HP LaserJet 4P/4MP,HPPCL5MS,LPT |32|
>>     +--+-------+------+-------+--------------------------------+--+***
>> 
>>     This would become in GNU APL:
>> 
>> 
>>     *a← 'kernel32 GetProfileStringA s *c *c *c *c s' b← 'windows'
>>     'device' 'default' (32⍴'z') 32******'universal-dll-loader' ⎕FX 
>> 'cd'****** a cd b ⍝ dlopen("kernel32.dll"), dlsym("GetProfileStringA") on
>>     first access,****⍝ and call GetProfileStringA with argument b*
>> 
>>     The *universal-dll-loader.so* needs to be written only once and
>>     contains mainly the code from github below. That code somehow goes
>>     into the *native/template_F12.cc* code of GNU APL and thats it. If
>>     you need help doing this then please let me know. /// Jürgen
>> 
>>     On 02/07/2017 06:30 PM, Xiao-Yong Jin wrote:
>>     It would be nice if one doesn't need to write wrappers and the APL 
>> system can do the
>>     structure conversions within the APL interpreter.  In J, you can dlopen 
>> a library
>>     and pass J values directly without writing and compiling C, see
>> 
>>        http://www.jsoftware.com/help/user/call_procedure.htm
>>     <http://www.jsoftware.com/help/user/call_procedure.htm>
>> 
>>     and the relevant code is at
>> 
>>        https://github.com/jsoftware/jsource/blob/master/jsrc/x15.c
>>     <https://github.com/jsoftware/jsource/blob/master/jsrc/x15.c>
>> 
>>     It would simplify using external libraries a lot.
>> 
>>     On Feb 4, 2017, at 7:38 AM, Juergen Sauermann<address@hidden>
>>     <mailto:address@hidden>  wrote:
>> 
>>     Hi,
>> 
>>     yes there is: native functions. You can load shared libraries and ⎕FX 
>> functions in
>>     them to be called from APL code. The src/native directory contains a few 
>> templates
>>     that you can use as a starting point and to call your favourite library 
>> from them.
>> 
>>     Of course you need to provide wrappers from/to APL values to/from the 
>> data
>>     structures expected or produced by the libraries.
>> 
>>     Coming back to your other problems, if you do not like the terminal I/O 
>> of GNU APL, then
>>     you can write your own one and call libapl from it. I have extended 
>> libapl recently, giving
>>     you the full functionality of GNU APL without the specific ways how it 
>> handles terminal IO.
>> 
>>     /// Jürgen
>> 
>> 
>>     On 02/04/2017 02:52 AM,address@hidden <mailto:address@hidden>  wrote:
>>     is there method for loading a c lib and using it in apl ?  cdecl?   like 
>> this in fpc?
>> 
>> 
>>     ftp://ftp.freepascal.org/fpc/docs-pdf/CinFreePascal.pdf
>>     <ftp://ftp.freepascal.org/fpc/docs-pdf/CinFreePascal.pdf>
>>       
>> 
>> 
>> 
> 
>