In the email below, I address the follow-up questions you passed to the
answers I passed in the previous email, mention reasons for why we recognize
those rules I described, and I also pass another three practices regarding
FFI development, and two questions that neither the manual nor the mailing
list afaik have addressed, that are vital to how FFI:s are best designed.
2008/9/5 Joel J. Adamson <adamsonj(a)email.unc.edu> <adamsonj(a)email.unc.edu>
> Thanks so much for the tips; I have some follow-up questions:
>
> >>>>> "MM" == Mikael More <mikael.more(a)gmail.com> writes:
>
> MM> - Throwing exceptions with no handler/catcher may sigsegv your
> MM> app.
>
> Good to know, I was unaware of this.
We saw this in threads, i.e. we got sigsegv when the thunk of a threads
throws an exception and there was no exception handler.
>
> MM> - You should presuppose that ___argXX variables inside
> MM> Scheme-to-C calls are trashed after C-to-Scheme calls. I.e.: If
> MM> you have declared a c-define or c-lambda that takes parameters,
> MM> these will appear as ___arg0, ___arg1 etc. in the C code of the
> MM> c-define / c-lambda. These will have correct contents all until
> MM> you invoke Scheme functions from the C code. not afterwards.
>
> Okay, take the following functions, that may be the source of the
> problem:
>
> /* C function for transferring vectors */
> int f64vector_length (___SCMOBJ);
> double f64vector_ref (___SCMOBJ, int);
> double * scm_vector_to_C (___SCMOBJ f64vec)
> {
> /* initialize the length of the vector */
> int len = f64vector_length (f64vec);
> /* initializes the new vector */
> double * newCvec = malloc (len*sizeof (double));
> /* declare a counter */
> int i;
> /* iterate over the length of the vector copying each object into
> the new vector */
> for (i = 0; i<len; i++)
> {
> /* could be re-written using pointer arithmetic */
> newCvec[i] = f64vector_ref (f64vec, i);
> }
> return newCvec;
> }
Can you please provide the complete c-lambda for this one?
>
>
> and then the corresponding "c-define"s:
>
> ;; Scheme function for getting data out of f64vectors
> (c-define (proc1 f64vec i)
> (scheme-object int)
> double
> "f64vector_ref" ""
> (f64vector-ref f64vec i))
>
> (c-define (proc2 f64vec)
> (scheme-object)
> int
> "f64vector_length" ""
> (f64vector-length f64vec))
>
> MM> So back them up to the stack if you do.
>
> Forgive my ignorance of C, but how do I do that? (where the heck is
> this stack that everyone speaks of?)
In C, "the stack" == "local variables". Back up to stack == make local
variables and store there.
>
>
> Now, I also noticed this passage in the C-interface chapter of the
> manual:
>
> ,----
> | Within the C code the variables `___arg1', `___arg2', etc. can be
> | referenced to access the converted arguments. Similarly, the result to
> | be returned from the call should be assigned to the variable `___result'
> | except when the result is of type `struct', `union', `type', `pointer',
> | `nonnull-pointer', `function' or `nonnull-function' in which case a
> | pointer must be assigned to the variable `___result_voidstar' which is
> | of type `void*'. For results of type `pointer', `nonnull-pointer',
> | `function' and `nonnull-function', the value assigned to the variable
> | `___result_voidstar' must be the pointer or function cast to `void*'.
> `----
>
> However, I have functions like this:
>
> (define make-gsl-matrix
> (c-lambda (int int scheme-object)
> gsl-matrix*
> "gsl_matrix * ___matrix = gsl_matrix_alloc (___arg1, ___arg2);
> ___matrix->data = scm_vector_to_C (___arg3);
> ___matrix->size1 = ___arg1;
> ___matrix->size2 = ___arg2;
> ___result_voidstar = ___matrix;"))
The rule about backing up arguments to the stack to save argument value
contents from being lost before calling Scheme functions from C, when in a
call to C from Scheme, would in the case of the c-lambda above translate
into:
(define make-gsl-matrix
(c-lambda (int int scheme-object)
gsl-matrix*
"int a1 = ___arg1; int a2 = ___arg2; int a3 = ___arg3;
gsl_matrix * ___matrix = gsl_matrix_alloc (___arg1, ___arg2);
___matrix->data = scm_vector_to_C (a3);
___matrix->size1 = a1;
___matrix->size2 = a2;
___result_voidstar = ___matrix;"))
( I do not know if the rule applies to c-lambdas with particular argument
counts. The case we tracked down was a c-lambda taking over ten arguments.
Also I have no idea if the rule applies to very basic procedures such as
car, cdr, +, etc. )
>
> It seems that this is seriously wrong, according to the paragraph above:
> the last line should be
>
> ___result_voidstar = (void*)___matrix;
>
> and the return type should be void* and not gsl-matrix*, defined as
> follows:
>
> (c-define-type gsl-matrix*
> (pointer "gsl_matrix"
> gsl-matrix*))
>
> But then the question becomes how do I access the result I need? I
> would much rather have a function that returns a matrix as a result
> (functional style) rather than having it imperatively modify an existing
> variable, a la C.
Returning a gsl-matrix* from a c-lambda is perfectly fine.
>
> Thanks for your help!
> Joel
>
>
## You may want to hook a release function to it also, that is a function
that will perform the garbage collection for variables of this type, when
the garbage collector invokes it to.
Example:
(c-declare #<<c-declare-end
___SCMOBJ release_cell_star(void* p) {
if (p == 0)
fprintf(stderr,"release_cell_star: Called with null parameter!
Internal inconsistency !! Ignoring.\n");
else {
delete (Cell*) p;
}
return ___FIX(___NO_ERR);
}
c-declare-end
)
(c-define-type Cell* (pointer "Cell" Cell* "release_cell_star"))
The example above is for freeing a C++ object. In C, you would typically
have free(p); instead of delete (Cell*) p; .
## Be careful never to reintroduce a pointer to a structure from the C
world into the Scheme world twice, so that the garbage collector would trig
twice. Examples:
OK:
(c-declare #<<c-declare-end
___SCMOBJ release_mystruct_star(void* p) {
if (p == 0)
fprintf(stderr,"release_mystruct_star: Called with null parameter!
Internal inconsistency !! Ignoring.\n");
else {
delete (Cell*) p;
}
return ___FIX(___NO_ERR);
}
c-declare-end
)
(c-define-type MyStruct* (pointer "MyStruct" MyStruct*
"release_mystruct_star"))
OK:
(define create-mystruct (c-lambda () MyStruct* #<<c-lambda-end
___result_voidstar = (void*) [ some code that allocates a new MyStruct ];
c-lambda-end
))
*NOT* OK. The reason it's not OK is because the MyStruct* is reintroduced
from the C world to the Scheme world when create-mystruct returns.
(define create-mystruct (c-lambda () MyStruct* #<<c-lambda-end
___result_voidstar = (void*) construct_my_returnvalue([ some code that
allocates a new MyStruct ]);
c-lambda-end
))
(c-define (construct-my-returnvalue struct)
(MyStruct*)
MyStruct*
"construct_my_returnvalue" "static"
[ code ] )
OK:
(define create-mystruct (c-lambda () scheme-object #<<c-lambda-end
___result = construct_my_returnvalue([ some code that allocates a new
MyStruct ]);
c-lambda-end
))
(c-define (construct-my-returnvalue struct)
(MyStruct*)
scheme-object
"construct_my_returnvalue" "static"
[ code ] )
Function that reintroduces a MyStruct* from the C world to the Scheme world.
*NOT* OK:
(define do-something-with-mystruct (c-lambda (MyStruct*) MyStruct*
#<<c-lambda-end
[code that does something with ___arg1]
___result_voidstar = ___arg1;
c-lambda-end
))
Function that reintroduces a MyStruct* from the C world to the Scheme world.
OK:
(define (do-something-with-mystruct MyStruct*)
(do-something-with-mystruct MyStruct* MyStruct*))
(define do-something-with-mystruct-private (c-lambda (MyStruct*
scheme-object) scheme-object #<<c-lambda-end
[code that typecasts ___arg1 to a MyStruct and does something with ___arg1]
___result = ___arg2;
c-lambda-end
))
Function that reintroduces a MyStruct* from the C world to the Scheme world.
*NOT* OK, What makes it not OK is two things: first, it reintroduces
MyStruct* from the C world to the Scheme world when invoking
construct_my_returnvalue, then it reintroduces it yet another time when
do-something-with-mystruct returns.
(define do-something-with-mystruct (c-lambda (MyStruct*) MyStruct*
#<<c-lambda-end
[code that does something with ___arg1]
___result_voidstar = construct_my_returnvalue(___arg1);
c-lambda-end
))
(c-define (construct-my-returnvalue struct)
(MyStruct*)
MyStruct*
"construct_my_returnvalue" "static"
[ code ] )
Function that reintroduces a MyStruct* from the C world to the Scheme world.
OK:
(define (do-something-with-mystruct MyStruct*)
(do-something-with-mystruct MyStruct* MyStruct*))
(define do-something-with-mystruct-private (c-lambda (MyStruct*
scheme-object) scheme-object #<<c-lambda-end
[code that typecasts ___arg1 to a MyStruct and does something with ___arg1]
___result = construct_my_returnvalue(___arg2);
c-lambda-end
))
(c-define (construct-my-returnvalue struct)
(scheme-object)
scheme-object
"construct_my_returnvalue" "static"
[ code ] )
## Note that there is no handling mechanism, such as garbage collection, to
free C frames.
Background:
In Scheme code, the garbage collector always has perfect handling of your
variables in all situations: you may have whatever complex patterns of
execution, throw exceptions anywhere, and invoke thread-terminate! on any
thread, and the garbage collector will always ensure that allocated memory
that will never more be relevant will be garbage collected.
This does not apply to c-lambdas: C code has a static stack structure. If C
function A invokes C function B that invokes C function C, these are always
returned in the same order, i.e. C returns to B that returns to A. (The same
applies to C++.) Since there are no continuations or closures, there
is *NO*way for B to, for instance,
- call some code from within A during its execution
- save its state to variable D, return into A, and then continue at D.
The only thing B can do about A's future execution and state, is to
- modify variables A has access to, if B has references to them
- return to A, possibly returning a return value
- in the case of C++, throw an exception.
Consequences:
When combining C and Scheme code, you must ensure that you call c-lambda:s
in a way that is in alignment with the way C works.
The rules here are:
* If you call Scheme code from within C code, always ensure that you return
back from the Scheme code to the C code.
I.e., ensure the thread will not be terminated, and that the Scheme code
called from the C code will not throw exceptions that will not be continued.
Applied to the examples above, do *not* do (thread-terminate!
(current-thread)) or (raise) from within construct-my-returnvalue.
As regards thread termination:
As Gambit is a multitasking environment, you must also ensure that no other
part of your application hinders the return of a C-to-Scheme call.
* A way to ensure this partially **could** be to compile the Scheme
functions you call from Scheme code with (declare (not interrupts-enabled)),
but beware, perhaps your Scheme code invokes other Scheme code that has
(declare (interrupts-enabled)). (Whether routines such as + and car are
relevant to this, I have no idea.)
* If your application uses multiple threads, and there are instances when
these perform thread-terminate!, another way to ensure this partially could
be to have a separate thread for execution of c-lambda:s. For example:
(define (worker-thread-thunk)
((thread-receive))
(worker-thread-thunk))
(define worker-thread ; CPU cheap. Start on load is ok.
(thread-start! (make-thread worker-thread-thunk)))
(define (perform-in-worker-thread thunk)
(let ((mailbox (make-empty-mailbox)))
(thread-send worker-thread
(lambda ()
(mailbox-put! mailbox
(with-exception-catcher
(lambda (e) (lambda () (raise e)))
thunk))))
(mailbox-get! mailbox)))
The mailbox code is found in the Gambit manual.
As regards exceptions:
Never throw an exception from Scheme code that is invoked by C code, that
you will not be continued.
If you would happen to have a Scheme function generate a return value for a
c-lambda ( Usually, I suppose, that this should be completely unnecessary,
and should not be adviced, though I have not seen examples of how to
construct return value structures such as lists in C. Could someone who
knows perhaps provide us with some examples that do this, and are
thread-safe as well, Marc ? ), one way to get around the risk of it
throwing exceptions would be to stuff the return value generating code in a
closure, i.e., rather than:
(define a (c-lambda () scheme-object #<<c-lambda-end
..code..
MyStruct* mystruct = ..code..;
___result_voidstar = construct_my_returnvalue(mystruct etc. );
c-lambda-end
))
(c-define (construct-my-returnvalue struct etc.)
(MyStruct* etc.)
scheme-object
"construct_my_returnvalue" "static"
(if conditions
(error " AN ERROR ")
struct))
Do:
(define (a)
((a-private)))
(define a-private (c-lambda () scheme-object #<<c-lambda-end
..code..
MyStruct* mystruct = ..code..;
___result_voidstar = construct_my_returnvalue(mystruct etc. );
c-lambda-end
))
(c-define (construct-my-returnvalue struct etc.)
(MyStruct* etc.)
scheme-object
"construct_my_returnvalue" "static"
(lambda ()
(if conditions
(error " AN ERROR ")
struct)))
* Do not return multiple times into C code
That is, C code is not compliant with continuations.
(I have no idea what happens if you would do this.)
Then, there is one more relevant design aspect to Gambit here, that neither
the manual nor the mailing list afaik has addressed:
Must c-lambda:s be returned in the exact inverse order as they were invoked,
or can they be returned in other orders also?
Whether this is so or not has great impact on how code that combines use of
threads and/or closures and/or continuations with c-code that calls Scheme.
An example where it would have impact: Start two threads that invoke the
example procedure a above an unlimited number of times, i.e.,
(define (spawn-a) (thread-start! (make-thread (lambda () (let loop () (a)
(loop))))))
(define first-thread (spawn-a))
(define second-thread (spawn-a))
At some point, this will happen:
1. a is invoked in the first thread. construct_my_returnvalue is entered in
the first thread.
2. The scheduler switches current thread of execution from the first thread
to the second thread.
3. a is invoked in the second thread. construct_my_returnvalue is entered
in the second thread.
4. The scheduler switches current thread of execution from the second
thread to the first thread.
5. construct_my_returnvalue returns in the first thread. a returns in the
first thread.
Is Gambit designed in a way that allows this? If not, what do you consider
the best strategies to safeguard against this ever happening?
If it is, how are C frames rewinded? In the example above, right after a
returns at 5., will its C frame be freed, or will this happen first when the
C frame allocated in 3. is freed also?
Marc or anyone who knows, how does Gambit work in this respect?
Mikael
