In the email below, I address the follow-up questions you passed to the answers I passed in the previous email, mention reasons for why we recognize those rules I described, and I also pass another three practices regarding FFI development, and two questions that neither the manual nor the mailing list afaik have addressed, that are vital to how FFI:s are best designed.

2008/9/5 Joel J. Adamson adamsonj@email.unc.edu adamsonj@email.unc.edu

Thanks so much for the tips; I have some follow-up questions:

...

...

...

...

...

"MM" == Mikael More mikael.more@gmail.com writes:

MM> - Throwing exceptions with no handler/catcher may sigsegv your MM> app.

Good to know, I was unaware of this.

We saw this in threads, i.e. we got sigsegv when the thunk of a threads throws an exception and there was no exception handler.

MM> - You should presuppose that ___argXX variables inside MM> Scheme-to-C calls are trashed after C-to-Scheme calls. I.e.: If MM> you have declared a c-define or c-lambda that takes parameters, MM> these will appear as ___arg0, ___arg1 etc. in the C code of the MM> c-define / c-lambda. These will have correct contents all until MM> you invoke Scheme functions from the C code. not afterwards.

Okay, take the following functions, that may be the source of the problem:

/* C function for transferring vectors */ int f64vector_length (___SCMOBJ); double f64vector_ref (___SCMOBJ, int); double * scm_vector_to_C (___SCMOBJ f64vec) { /* initialize the length of the vector */ int len = f64vector_length (f64vec); /* initializes the new vector */ double * newCvec = malloc (len*sizeof (double)); /* declare a counter */ int i; /* iterate over the length of the vector copying each object into the new vector */ for (i = 0; i<len; i++) { /* could be re-written using pointer arithmetic */ newCvec[i] = f64vector_ref (f64vec, i); } return newCvec; }

Can you please provide the complete c-lambda for this one?

and then the corresponding "c-define"s:

;; Scheme function for getting data out of f64vectors (c-define (proc1 f64vec i) (scheme-object int) double "f64vector_ref" "" (f64vector-ref f64vec i))

(c-define (proc2 f64vec) (scheme-object) int "f64vector_length" "" (f64vector-length f64vec))

MM> So back them up to the stack if you do.

Forgive my ignorance of C, but how do I do that? (where the heck is this stack that everyone speaks of?)

In C, "the stack" == "local variables". Back up to stack == make local variables and store there.

Now, I also noticed this passage in the C-interface chapter of the manual:

,---- | Within the C code the variables `___arg1', `___arg2', etc. can be | referenced to access the converted arguments. Similarly, the result to | be returned from the call should be assigned to the variable `___result' | except when the result is of type `struct', `union', `type', `pointer', | `nonnull-pointer', `function' or `nonnull-function' in which case a | pointer must be assigned to the variable `___result_voidstar' which is | of type `void*'. For results of type `pointer', `nonnull-pointer', | `function' and `nonnull-function', the value assigned to the variable | `___result_voidstar' must be the pointer or function cast to `void*'. `----

However, I have functions like this:

(define make-gsl-matrix (c-lambda (int int scheme-object) gsl-matrix* "gsl_matrix * ___matrix = gsl_matrix_alloc (___arg1, ___arg2); ___matrix->data = scm_vector_to_C (___arg3); ___matrix->size1 = ___arg1; ___matrix->size2 = ___arg2; ___result_voidstar = ___matrix;"))

The rule about backing up arguments to the stack to save argument value contents from being lost before calling Scheme functions from C, when in a call to C from Scheme, would in the case of the c-lambda above translate into:

(define make-gsl-matrix (c-lambda (int int scheme-object) gsl-matrix* "int a1 = ___arg1; int a2 = ___arg2; int a3 = ___arg3; gsl_matrix * ___matrix = gsl_matrix_alloc (___arg1, ___arg2); ___matrix->data = scm_vector_to_C (a3); ___matrix->size1 = a1; ___matrix->size2 = a2; ___result_voidstar = ___matrix;"))

( I do not know if the rule applies to c-lambdas with particular argument counts. The case we tracked down was a c-lambda taking over ten arguments. Also I have no idea if the rule applies to very basic procedures such as car, cdr, +, etc. )

It seems that this is seriously wrong, according to the paragraph above: the last line should be

___result_voidstar = (void*)___matrix;

and the return type should be void* and not gsl-matrix*, defined as follows:

(c-define-type gsl-matrix* (pointer "gsl_matrix" gsl-matrix*))

But then the question becomes how do I access the result I need? I would much rather have a function that returns a matrix as a result (functional style) rather than having it imperatively modify an existing variable, a la C.

Returning a gsl-matrix* from a c-lambda is perfectly fine.

Thanks for your help! Joel

## You may want to hook a release function to it also, that is a function that will perform the garbage collection for variables of this type, when the garbage collector invokes it to.

Example:

(c-declare #<<c-declare-end

___SCMOBJ release_cell_star(void* p) { if (p == 0) fprintf(stderr,"release_cell_star: Called with null parameter! Internal inconsistency !! Ignoring.\n"); else { delete (Cell*) p; }

return ___FIX(___NO_ERR); }

c-declare-end )

(c-define-type Cell* (pointer "Cell" Cell* "release_cell_star"))

The example above is for freeing a C++ object. In C, you would typically have free(p); instead of delete (Cell*) p; .

## Be careful never to reintroduce a pointer to a structure from the C world into the Scheme world twice, so that the garbage collector would trig twice. Examples:

OK:

(c-declare #<<c-declare-end

___SCMOBJ release_mystruct_star(void* p) { if (p == 0) fprintf(stderr,"release_mystruct_star: Called with null parameter! Internal inconsistency !! Ignoring.\n"); else { delete (Cell*) p; }

return ___FIX(___NO_ERR); }

c-declare-end )

(c-define-type MyStruct* (pointer "MyStruct" MyStruct* "release_mystruct_star"))

OK:

(define create-mystruct (c-lambda () MyStruct* #<<c-lambda-end

___result_voidstar = (void*) [ some code that allocates a new MyStruct ];

c-lambda-end ))

*NOT* OK. The reason it's not OK is because the MyStruct* is reintroduced from the C world to the Scheme world when create-mystruct returns.

(define create-mystruct (c-lambda () MyStruct* #<<c-lambda-end

___result_voidstar = (void*) construct_my_returnvalue([ some code that allocates a new MyStruct ]);

c-lambda-end ))

(c-define (construct-my-returnvalue struct) (MyStruct*) MyStruct* "construct_my_returnvalue" "static" [ code ] )

OK:

(define create-mystruct (c-lambda () scheme-object #<<c-lambda-end

___result = construct_my_returnvalue([ some code that allocates a new MyStruct ]);

c-lambda-end ))

(c-define (construct-my-returnvalue struct) (MyStruct*) scheme-object "construct_my_returnvalue" "static" [ code ] )

Function that reintroduces a MyStruct* from the C world to the Scheme world. *NOT* OK:

(define do-something-with-mystruct (c-lambda (MyStruct*) MyStruct* #<<c-lambda-end

[code that does something with ___arg1]

___result_voidstar = ___arg1;

c-lambda-end ))

Function that reintroduces a MyStruct* from the C world to the Scheme world. OK:

(define (do-something-with-mystruct MyStruct*) (do-something-with-mystruct MyStruct* MyStruct*))

(define do-something-with-mystruct-private (c-lambda (MyStruct* scheme-object) scheme-object #<<c-lambda-end

[code that typecasts ___arg1 to a MyStruct and does something with ___arg1]

___result = ___arg2;

c-lambda-end ))

Function that reintroduces a MyStruct* from the C world to the Scheme world. *NOT* OK, What makes it not OK is two things: first, it reintroduces MyStruct* from the C world to the Scheme world when invoking construct_my_returnvalue, then it reintroduces it yet another time when do-something-with-mystruct returns.

(define do-something-with-mystruct (c-lambda (MyStruct*) MyStruct* #<<c-lambda-end

[code that does something with ___arg1]

___result_voidstar = construct_my_returnvalue(___arg1);

c-lambda-end ))

(c-define (construct-my-returnvalue struct) (MyStruct*) MyStruct* "construct_my_returnvalue" "static" [ code ] )

Function that reintroduces a MyStruct* from the C world to the Scheme world. OK:

(define (do-something-with-mystruct MyStruct*) (do-something-with-mystruct MyStruct* MyStruct*))

(define do-something-with-mystruct-private (c-lambda (MyStruct* scheme-object) scheme-object #<<c-lambda-end

[code that typecasts ___arg1 to a MyStruct and does something with ___arg1]

___result = construct_my_returnvalue(___arg2);

c-lambda-end ))

(c-define (construct-my-returnvalue struct) (scheme-object) scheme-object "construct_my_returnvalue" "static" [ code ] )

## Note that there is no handling mechanism, such as garbage collection, to free C frames.

Background:

In Scheme code, the garbage collector always has perfect handling of your variables in all situations: you may have whatever complex patterns of execution, throw exceptions anywhere, and invoke thread-terminate! on any thread, and the garbage collector will always ensure that allocated memory that will never more be relevant will be garbage collected.

This does not apply to c-lambdas: C code has a static stack structure. If C function A invokes C function B that invokes C function C, these are always returned in the same order, i.e. C returns to B that returns to A. (The same applies to C++.) Since there are no continuations or closures, there is *NO*way for B to, for instance,

- call some code from within A during its execution - save its state to variable D, return into A, and then continue at D.

The only thing B can do about A's future execution and state, is to

- modify variables A has access to, if B has references to them - return to A, possibly returning a return value - in the case of C++, throw an exception.

Consequences:

When combining C and Scheme code, you must ensure that you call c-lambda:s in a way that is in alignment with the way C works.

The rules here are:

* If you call Scheme code from within C code, always ensure that you return back from the Scheme code to the C code.

I.e., ensure the thread will not be terminated, and that the Scheme code called from the C code will not throw exceptions that will not be continued.

Applied to the examples above, do *not* do (thread-terminate! (current-thread)) or (raise) from within construct-my-returnvalue.

As regards thread termination:

As Gambit is a multitasking environment, you must also ensure that no other part of your application hinders the return of a C-to-Scheme call.

* A way to ensure this partially **could** be to compile the Scheme functions you call from Scheme code with (declare (not interrupts-enabled)), but beware, perhaps your Scheme code invokes other Scheme code that has (declare (interrupts-enabled)). (Whether routines such as + and car are relevant to this, I have no idea.)

* If your application uses multiple threads, and there are instances when these perform thread-terminate!, another way to ensure this partially could be to have a separate thread for execution of c-lambda:s. For example:

(define (worker-thread-thunk) ((thread-receive)) (worker-thread-thunk))

(define worker-thread ; CPU cheap. Start on load is ok. (thread-start! (make-thread worker-thread-thunk)))

(define (perform-in-worker-thread thunk) (let ((mailbox (make-empty-mailbox))) (thread-send worker-thread (lambda () (mailbox-put! mailbox (with-exception-catcher (lambda (e) (lambda () (raise e))) thunk)))) (mailbox-get! mailbox)))

The mailbox code is found in the Gambit manual.

As regards exceptions:

Never throw an exception from Scheme code that is invoked by C code, that you will not be continued.

If you would happen to have a Scheme function generate a return value for a c-lambda ( Usually, I suppose, that this should be completely unnecessary, and should not be adviced, though I have not seen examples of how to construct return value structures such as lists in C. Could someone who knows perhaps provide us with some examples that do this, and are thread-safe as well, Marc ? ), one way to get around the risk of it throwing exceptions would be to stuff the return value generating code in a closure, i.e., rather than:

(define a (c-lambda () scheme-object #<<c-lambda-end

..code..

MyStruct* mystruct = ..code..;

___result_voidstar = construct_my_returnvalue(mystruct etc. );

c-lambda-end ))

(c-define (construct-my-returnvalue struct etc.) (MyStruct* etc.) scheme-object "construct_my_returnvalue" "static" (if conditions (error " AN ERROR ") struct))

Do:

(define (a) ((a-private)))

(define a-private (c-lambda () scheme-object #<<c-lambda-end

..code..

MyStruct* mystruct = ..code..;

___result_voidstar = construct_my_returnvalue(mystruct etc. );

c-lambda-end ))

(c-define (construct-my-returnvalue struct etc.) (MyStruct* etc.) scheme-object "construct_my_returnvalue" "static" (lambda () (if conditions (error " AN ERROR ") struct)))

* Do not return multiple times into C code

That is, C code is not compliant with continuations.

(I have no idea what happens if you would do this.)

Then, there is one more relevant design aspect to Gambit here, that neither the manual nor the mailing list afaik has addressed:

Must c-lambda:s be returned in the exact inverse order as they were invoked, or can they be returned in other orders also?

Whether this is so or not has great impact on how code that combines use of threads and/or closures and/or continuations with c-code that calls Scheme.

An example where it would have impact: Start two threads that invoke the example procedure a above an unlimited number of times, i.e.,

(define (spawn-a) (thread-start! (make-thread (lambda () (let loop () (a) (loop)))))) (define first-thread (spawn-a)) (define second-thread (spawn-a))

At some point, this will happen:

1. a is invoked in the first thread. construct_my_returnvalue is entered in the first thread. 2. The scheduler switches current thread of execution from the first thread to the second thread. 3. a is invoked in the second thread. construct_my_returnvalue is entered in the second thread. 4. The scheduler switches current thread of execution from the second thread to the first thread. 5. construct_my_returnvalue returns in the first thread. a returns in the first thread.

Is Gambit designed in a way that allows this? If not, what do you consider the best strategies to safeguard against this ever happening? If it is, how are C frames rewinded? In the example above, right after a returns at 5., will its C frame be freed, or will this happen first when the C frame allocated in 3. is freed also?

Marc or anyone who knows, how does Gambit work in this respect?

Mikael