Hi,
I have found the following remark by Marc in one of the posts,
On some platforms (processor and operating system dependent) that can execute dynamically generated machine code (32 bit intel, power-pc, sparc), you can pass any procedure including closures. This feature is enabled if USE_dynamic_code_gen is defined in lib/os_dyn.h and this is the default in recent versions of Gambit. Give it a try... it may work to pass a closure on your platform... but don't count on it to be portable!
Could somebody explain how it works? michele
Afficher les réponses par date
On 5-Oct-09, at 7:20 AM, Michele Zaffalon wrote:
Hi,
I have found the following remark by Marc in one of the posts,
On some platforms (processor and operating system dependent) that can execute dynamically generated machine code (32 bit intel, power-pc, sparc), you can pass any procedure including closures. This feature is enabled if USE_dynamic_code_gen is defined in lib/os_dyn.h and this is the default in recent versions of Gambit. Give it a try... it may work to pass a closure on your platform... but don't count on it to be portable!
Could somebody explain how it works?
It used to work! I noticed a stupid bug in lib/os_dyn.c . You have to replace all the ___WORD_SIZE by ___WORD_WIDTH and then recompile the system with "make".
Then, on supported platforms such as 32 bit intel, you'll be able to run code like this which passes a closure to a foreign function:
(c-declare "
int f( int g( int x ) ) { return g(1) + g(2); }
")
(define f (c-lambda ((function (int) int)) int "f"))
(c-define (square n) (int) int "square" "" (* n n))
(pp (square 10)) ;; prints 100
(pp (f square)) ;; prints 5 (i.e. 1^2 + 2^2)
(pp (f (lambda (x) (expt 2 x)))) ;; prints 6 (i.e. 2^1 + 2^2)
It is the last expression, i.e. the call (f (lambda (x) (expt 2 x))), which is enabled by the dynamic code generation. The call (f square) is always supported because it does not require dynamic code generation (because square is defined with a c-define form).
Marc
It used to work! I noticed a stupid bug in lib/os_dyn.c . You have to replace all the ___WORD_SIZE by ___WORD_WIDTH and then recompile the system with "make".
Then, on supported platforms such as 32 bit intel, you'll be able to run code like this which passes a closure to a foreign function:
(c-declare "
int f( int g( int x ) ) { return g(1) + g(2); }
")
(define f (c-lambda ((function (int) int)) int "f"))
(c-define (square n) (int) int "square" "" (* n n))
(pp (square 10)) ;; prints 100
(pp (f square)) ;; prints 5 (i.e. 1^2 + 2^2)
(pp (f (lambda (x) (expt 2 x)))) ;; prints 6 (i.e. 2^1 + 2^2)
It is the last expression, i.e. the call (f (lambda (x) (expt 2 x))), which is enabled by the dynamic code generation. The call (f square) is always supported because it does not require dynamic code generation (because square is defined with a c-define form).
Marc
I made clean ./configure CFLAGS="-g" and make after replacing ___WORD_SIZE by ___WORD_WIDTH in os_dyn.c
This is what I then do: bind.scm contains (c-declare " int f(int g(int x)) {return g(1)+g(2);} ")
(define f (c-lambda ((function (int) int)) int "f"))
after doing gsc bind.scm, I invoke gsi
(load "bind")
"/home/michele/src/scheme/gambit/ffi/4/bind.o1"
(f (lambda (x) (expt 2 x)))
I get a segmentation fault
bt full is provided.
System is OpenBSD 4.5 (GENERIC) #1749: Sat Feb 28 14:51:18 MST 2009 deraadt@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Mobile Intel(R) Pentium(R) 4 CPU 2.30GHz ("GenuineIntel" 686-class) 2.16 GHz
michele
On 5-Oct-09, at 9:02 PM, Michele Zaffalon wrote:
(load "bind")
"/home/michele/src/scheme/gambit/ffi/4/bind.o1"
(f (lambda (x) (expt 2 x)))
I get a segmentation fault
bt full is provided.
System is OpenBSD 4.5 (GENERIC) #1749: Sat Feb 28 14:51:18 MST 2009 deraadt@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Mobile Intel(R) Pentium(R) 4 CPU 2.30GHz ("GenuineIntel" 686-class) 2.16 GHz
It is quite probable that OpenBSD forbids executing code in the C heap, for security reasons I guess. Usually there is a way to tell the operating system that a certain range of addresses contains executable code. I have no clue what this is on OpenBSD. If you do find out or someone on the list knows, please let me know.
Marc
It is quite probable that OpenBSD forbids executing code in the C heap, for security reasons I guess. Usually there is a way to tell the operating system that a certain range of addresses contains executable code. I have no clue what this is on OpenBSD. If you do find out or someone on the list knows, please let me know.
Is this the only way to have a Scheme defined function passed on the fly to C? Short of using global variables as in here https://mercure.iro.umontreal.ca/pipermail/gambit-list/2009-January/002939.h...
By the way, towards the end of this post, you point out the correct option to configure.
michele
On 6-Oct-09, at 8:37 AM, Michele Zaffalon wrote:
It is quite probable that OpenBSD forbids executing code in the C heap, for security reasons I guess. Usually there is a way to tell the operating system that a certain range of addresses contains executable code. I have no clue what this is on OpenBSD. If you do find out or someone on the list knows, please let me know.
Is this the only way to have a Scheme defined function passed on the fly to C? Short of using global variables as in here https://mercure.iro.umontreal.ca/pipermail/gambit-list/2009-January/002939.h...
Well, the only solutions I find acceptable. There might be some (more or less portable) library to implement C closures that could be used (libffi?). You have to understand what the fundamental problem is. In C, functions pointers are implemented (usually) as the address of the code which implements the function. The C compiler generates a "call" instruction to that address. Scheme closures carry with them an environment... that is what makes them more useful than plain functions. So to convert a Scheme closure to a C function pointer, in general it is necessary to allocate a fresh code address that will correspond to the Scheme closure. The technique shown in the message linked above is based on the idea of allocating at compile time a pool of C functions, and then to assign them on demand to Scheme closures. It is 100% portable, but it places a static limit on the number of Scheme closures that can be converted (the code could be improved to un-assign the Scheme closure when it is no longer live using wills for example). The other solution, based on dynamic code generation, does not place a limit on the number of converted Scheme closures, but it is not portable (it is CPU and OS dependent). There is no portable solution I know of that solves both issues (portable here means it respects the C standard and only relies on the C standard).
Marc
Is this the only way to have a Scheme defined function passed on the fly to C? Short of using global variables as in here
https://mercure.iro.umontreal.ca/pipermail/gambit-list/2009-January/002939.h...
Well, the only solutions I find acceptable. There might be some (more or less portable) library to implement C closures that could be used (libffi?). You have to understand what the fundamental problem is. In C, functions pointers are implemented (usually) as the address of the code which implements the function. The C compiler generates a "call" instruction to that address. Scheme closures carry with them an environment... that is what makes them more useful than plain functions. So to convert a Scheme closure to a C function pointer, in general it is necessary to allocate a fresh code address that will correspond to the Scheme closure. The technique shown in the message linked above is based on the idea of allocating at compile time a pool of C functions, and then to assign them on demand to Scheme closures. It is 100% portable, but it places a static limit on the number of Scheme closures that can be converted (the code could be improved to un-assign the Scheme closure when it is no longer live using wills for example). The other solution, based on dynamic code generation, does not place a limit on the number of converted Scheme closures, but it is not portable (it is CPU and OS dependent). There is no portable solution I know of that solves both issues (portable here means it respects the C standard and only relies on the C standard).
Marc
I don't understand: in
(c-define (helper-func x) (int) int "helper_func" "" (*f* x))
(define gsl-func (c-lambda (int) int "gsl_func"))
and helper_func is called by the gsl_func routine, *f* can be any Scheme closure. Then why can't the C interface accept
(define gsl-func2 (c-lambda ( (function (int) int) int) int "gsl_func"))
and generate the correct helper-func with *f* replaced by whatever function is passed to gsl-func2?
I am sure this is a silly question so apologies in advance.
michele
On 2009-10-08, at 9:13 AM, Michele Zaffalon wrote:
Is this the only way to have a Scheme defined function passed on the fly to C? Short of using global variables as in here
https://mercure.iro.umontreal.ca/pipermail/gambit-list/2009-January/002939.h...
Well, the only solutions I find acceptable. There might be some (more or less portable) library to implement C closures that could be used (libffi?). You have to understand what the fundamental problem is. In C, functions pointers are implemented (usually) as the address of the code which implements the function. The C compiler generates a "call" instruction to that address. Scheme closures carry with them an environment... that is what makes them more useful than plain functions. So to convert a Scheme closure to a C function pointer, in general it is necessary to allocate a fresh code address that will correspond to the Scheme closure. The technique shown in the message linked above is based on the idea of allocating at compile time a pool of C functions, and then to assign them on demand to Scheme closures. It is 100% portable, but it places a static limit on the number of Scheme closures that can be converted (the code could be improved to un-assign the Scheme closure when it is no longer live using wills for example). The other solution, based on dynamic code generation, does not place a limit on the number of converted Scheme closures, but it is not portable (it is CPU and OS dependent). There is no portable solution I know of that solves both issues (portable here means it respects the C standard and only relies on the C standard).
Marc
I don't understand: in
(c-define (helper-func x) (int) int "helper_func" "" (*f* x))
(define gsl-func (c-lambda (int) int "gsl_func"))
and helper_func is called by the gsl_func routine, *f* can be any Scheme closure. Then why can't the C interface accept
(define gsl-func2 (c-lambda ( (function (int) int) int) int "gsl_func"))
and generate the correct helper-func with *f* replaced by whatever function is passed to gsl-func2?
The user is free to do that in his/her code (that is what the code at the above link does). So I assume you mean "why doesn't the C interface do this automatically for the user?".
Here are a few problems that come to mind:
1) This approach is not thread-safe. If two threads concurrently call gsl-func2 with different closures, then they will clobber the value of *f* that the other thread is using.
2) Even if threads are not used, imagine the case where the C code is receiving the function and storing it away for later use in a table (for example the function is a callback attached to a particular event, say the mouse is moved, or a key is typed on the keyboard, etc). With your approach there is a single *f* which is shared by all the callbacks you install. Which means that they are not independent. In fact, whichever callback is called, the most recently installed callback will be called (because that is what *f* will contain).
In other words, the fundamental problem here is that all closures must be independent, so they cannot share state. That's why the decision to use this approach must be in the hands of the user. The system can't do this automatically.
The two solutions I explained previously solve the sharing problem, with different caveats.
Marc
I don't understand: in
(c-define (helper-func x) (int) int "helper_func" "" (*f* x))
(define gsl-func (c-lambda (int) int "gsl_func"))
and helper_func is called by the gsl_func routine, *f* can be any Scheme closure. Then why can't the C interface accept
(define gsl-func2 (c-lambda ( (function (int) int) int) int "gsl_func"))
and generate the correct helper-func with *f* replaced by whatever function is passed to gsl-func2?
The user is free to do that in his/her code (that is what the code at the above link does). So I assume you mean "why doesn't the C interface do this automatically for the user?".
Here are a few problems that come to mind:
- This approach is not thread-safe. If two threads concurrently call
gsl-func2 with different closures, then they will clobber the value of *f* that the other thread is using.
- Even if threads are not used, imagine the case where the C code is
receiving the function and storing it away for later use in a table (for example the function is a callback attached to a particular event, say the mouse is moved, or a key is typed on the keyboard, etc). With your approach there is a single *f* which is shared by all the callbacks you install. Which means that they are not independent. In fact, whichever callback is called, the most recently installed callback will be called (because that is what *f* will contain).
In other words, the fundamental problem here is that all closures must be independent, so they cannot share state. That's why the decision to use this approach must be in the hands of the user. The system can't do this automatically.
The two solutions I explained previously solve the sharing problem, with different caveats.
Marc
Sorry, I didn't express myself clearly. What can I do: I am just a physicist...
Given
(define gsl-func2 (c-lambda ( (function (int) int) int) int "gsl_func"))
the first argument of gsl-func2 is a function f defined on the Scheme side.
The C interface now produces a C function helper_func that is passed to gsl_func: this helper_func does nothing else that calling the Scheme function f on the parameter, just in the same way *f* was used.
What I want to say in my broken way is: if it works with the global *f*, which is not yet defined, why can't it work with f? Or in other words, the Scheme f with signature (funtion (int) int) is magically converted to the C function supplied gsl_func and this C function returns f(x).
I think I have repeated the same concept a few times already :-)
michele
On 2009-10-08, at 1:11 PM, Michele Zaffalon wrote:
Sorry, I didn't express myself clearly. What can I do: I am just a physicist...
Given
(define gsl-func2 (c-lambda ( (function (int) int) int) int "gsl_func"))
the first argument of gsl-func2 is a function f defined on the Scheme side.
The C interface now produces a C function helper_func that is passed to gsl_func: this helper_func does nothing else that calling the Scheme function f on the parameter, just in the same way *f* was used.
What I want to say in my broken way is: if it works with the global *f*, which is not yet defined, why can't it work with f? Or in other words, the Scheme f with signature (funtion (int) int) is magically converted to the C function supplied gsl_func and this C function returns f(x).
I'm not sure this will help but, here goes...
The representation of a "function" is different for Gambit an C. In C, a function "value" is normally represented as the address of the machine code that implements this function. The C compiler generates the "call" machine instruction to call functions represented this way. On the Gambit side, a function value is represented as the address of the "closure object" which contains the environment of the closure and the code pointer. You can't use the "call" machine instruction to call such functions. Even in a Scheme system where closures are represented as a piece of code (this is possible, in fact the first versions of Gambit used this representation) you also have the problem that the parameter passing protocol may be different in C and Scheme, and it probably has to be to support "rest parameters".
So... when a Scheme function is passed to the C world, as a C function, there is a real conversion that must occur because the underlying representations and parameter passing protocol are different in the two worlds.
Does that clarify things?
Marc
I'm not sure this will help but, here goes...
The representation of a "function" is different for Gambit an C. In C, a function "value" is normally represented as the address of the machine code that implements this function. The C compiler generates the "call" machine instruction to call functions represented this way. On the Gambit side, a function value is represented as the address of the "closure object" which contains the environment of the closure and the code pointer. You can't use the "call" machine instruction to call such functions. Even in a Scheme system where closures are represented as a piece of code (this is possible, in fact the first versions of Gambit used this representation) you also have the problem that the parameter passing protocol may be different in C and Scheme, and it probably has to be to support "rest parameters".
So... when a Scheme function is passed to the C world, as a C function, there is a real conversion that must occur because the underlying representations and parameter passing protocol are different in the two worlds.
Does that clarify things?
Marc
It was already clear enough in your previous reply, thanks for the patience.
So the bottom line is that
(c-define (callback-wrapper x) (int) int "callback_wrapper" "" (f x))
won't work with
(letrec ((f (lambda (x) (* 4 x)))) (callback-wrapper 4))
but it will in the top level only with
(define (f p) (* 3 p)) (callback-wrapper 4) ; -> 12
michele