I am writing a compiler for a coroutine-based language and have the exact same limitation with regards to the FFI.
I am curious, Dimitris, whether you know of any other language with first-class coroutines or coroutines that have your desired feature. I myself do not. Perhaps it's still an open research problem.
-Patrick
On Sun, Feb 24, 2013 at 1:51 PM, Dimitris Vyzovitis vyzo@hackzen.org wrote:
On Sun, Feb 24, 2013 at 8:23 AM, Mikael mikael.rcv@gmail.com wrote:
Indeed it is a kind of limitation, though limitation is an aspect of the nature of any technology solution.
Find below some reflections about the design area your example code regards and a suggestion for best practice.
Sorry, but I wasn't waxing philosophically here. This is a limitation in the ffi that makes certain kinds of programs integrating with C impossible to write without hairy workarounds (more below). The issue should at least be in the radar -- a fix is even better.
Two immediately affected classes of programs:
- Progressive I/O through a C library. This is the case my example
code was intended to illustrate. The general scenario is a C library that provides a mechanism for pulling data through custom I/O objects. Note that the streams may be infinite or have message boundaries, so doing full port reads at scheme level can't work. This is common when you are doing systems programming.
- Event loops in the C library. Same issue with the upcalls, they
basically can't run threads or do nested ffi calls. This is common for gui systems etc.
I checked your code closer now; basically the c-lambda is a part of the application logics that could as well have been implemented in Scheme.
This code was simplified to illustrate the problem. Scheme implementation as you suggest is impossible (the streams are infinite, did you catch the /dev/zero?)
Whenever you can implement application logics in Scheme, do it, because the C stack model is as you have found out limited to "direct style" i.e. each step up must be balanced with one step down and there are *no* shortcuts. This is the C stack, and how it's delivered to you in the FFI.
The implementation of ___call already captures frame markers and conveys the intent of being re-entrant. It is possible that frames get overwritten by different returns, but this could be addressed by reseving enough stack space for the interleaved C frames from different threads. Also, some context would be nice when aborting (even better, just crash instead of the nasty exit. so that the problem is immediately obvious and can be debugged with gdb)
Of course it follows that if you attempt to do maneuvers on the C stack that are not possible from Scheme, something breaks.
Using extraordinarily fancy stack manipulation techniques, this could be worked around, though that would not be based on completely standard features found in any computing environment and thus not end up as crossplatform as Gambit is today, and, it'd take a bit of performance.
I am aware of these techniques - this is how I worked around the problem. Make a separate (mmap allocated) frame for the C stack, and trampoline through a scheme procedure that handles the i/o pumping and then ffi re-enters. It is not generally portable (some assembly -- setjmp/longjmp/getcontext/setcontext can't return pointer-sized values and don't provide any usable mechanism for storing the context for multiple threads, and there is also the issue of redzoning the stack to catch overflows), but it works for now.
When Gambit gets SMP support in some months from now, you'll have one C stack per OS thread you launched, so you could do a bit more of this then.
Not really.
Now back to the problem -
I think you missed the point of the example code. This was abstracted from real code that triggered the exit and simplified to illustrate the issue while preserving sufficient structure from the original code.
So your c-lambda is:
(define do-read (c-lambda (scheme-object) int #<<END-C char buf[4096]; ___SCMOBJ ctx = ___EXT(___make_vector)(2, ___FAL, ___STILL); ___VECTORSET (ctx, ___FIX(0), ___arg1);
for (;;) { if (do_read ((void*)ctx, buf, sizeof(buf)) < 0) break; } END-C ))
To solve your issue, rewrite it to Scheme i.e.
(lambda (v) (let ((buf (make-u8vector 4096)) (ctx (make-still-vector 2 #f))) (vector-set! ctx 0 1) (let loop () (if (< 0 (do-read ctx buf (u8vector-length buf)) (loop)))
Nope, this doesn't solve anything as mentioned above. The streams are infinite.
It also doesn't work when there are message boundaries for interactive protocols that need to be processed by a C-library. The scheme code has no knowledge of framing and required input sizes, the C library provides this information in the pump upcall as it discovers it.
Again, what you are wanting to get is an extremely high level functionality, namely using a IO multiplexer, in this case the one of Gambit's scheduler, to return to different places in code in essentially different stacks - this would generally never be done in C, and this remains the case also here.
Generally, never make a Scheme-to-C call with a C-to-Scheme call that may block in any way, except for if such use is done once in parallell within the OS thread. C's stack model does not deliver for other than exactly this. Instead, make your C call return back to Scheme with some kind of status value that instructs your Scheme code how to proceed. Such an approach will always end up being 'good code'.
Good code doesn't force you to stand on your head to work around limitations, it is written directly to do the task at hand (but that's just my opinion, I have no interest in flamewars about such generalities). Again, the issue is the limitation of the ffi.
If you need to pass complex result data back from the C call to Scheme, you can do this as C-to-Scheme calls though then keep the thread system off during those calls as to keep your code safe from the possibility of other code compromising the C stack in relation with the task you're performing and thus crashing your program.
If you would really need multiple C stacks, then, presuming they do C logics only and do not directly invoke any C code beyond your control (such as OS routines), you could use a C cooperate multithreading library to get the functionality you look for. That would be sufficiently crossplatform.
Gambit already has cross-platform user threads, you shouldn't have to shoehorn another threading system into the platform. It also doesn't quite solve the problem, what you need is a way to do coroutines that cross the language barriers - see the description of the work-around I used above.
-- vyzo _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list