[gambit-list] C stack frames need to be returned in exactly the same order they were created Re: FFI and threads: abrupt exit 71

Patrick Li patrickli.2001 at gmail.com
Sun Feb 24 17:25:00 EST 2013

I am writing a compiler for a coroutine-based language and have the
exact same limitation with regards to the FFI.

I am curious, Dimitris, whether you know of any other language with
first-class coroutines or coroutines that have your desired feature. I
myself do not. Perhaps it's still an open research problem.


On Sun, Feb 24, 2013 at 1:51 PM, Dimitris Vyzovitis <vyzo at hackzen.org> wrote:
> On Sun, Feb 24, 2013 at 8:23 AM, Mikael <mikael.rcv at gmail.com> wrote:
>> Indeed it is a kind of limitation, though limitation is an aspect of the
>> nature of any technology solution.
>> Find below some reflections about the design area your example code regards
>> and a suggestion for best practice.
> Sorry, but I wasn't waxing philosophically here.
> This is a limitation in the ffi that makes certain kinds of programs
> integrating with C impossible to write without hairy workarounds (more
> below).
> The issue should at least be in the radar -- a fix is even better.
> Two immediately affected classes of programs:
> 1. Progressive I/O through a C library. This is the case my example
> code was intended to illustrate.
> The general scenario is a C library that provides a mechanism for
> pulling data through custom I/O objects.
> Note that the streams may be infinite or have message boundaries, so
> doing full port reads at scheme level can't work.
> This is common when you are doing systems programming.
> 2. Event loops in the C library. Same issue with the upcalls, they
> basically can't run threads or do nested ffi calls.
> This is common for gui systems etc.
>> I checked your code closer now; basically the c-lambda is a part of the
>> application logics that could as well have been implemented in Scheme.
> This code was simplified to illustrate the problem. Scheme
> implementation as you suggest is impossible (the streams are infinite,
> did you catch the /dev/zero?)
>> Whenever you can implement application logics in Scheme, do it, because the
>> C stack model is as you have found out limited to "direct style" i.e. each
>> step up must be balanced with one step down and there are *no* shortcuts.
>> This is the C stack, and how it's delivered to you in the FFI.
> The implementation of ___call already captures frame markers and
> conveys the intent of being re-entrant. It is possible that frames get
> overwritten by different returns, but this could be addressed by
> reseving enough stack space for the interleaved C frames from
> different threads.
> Also, some context would be nice when aborting (even better, just
> crash instead of the nasty exit. so that the problem is immediately
> obvious and can be debugged with gdb)
>> Of course it follows that if you attempt to do maneuvers on the C stack that
>> are not possible from Scheme, something breaks.
>> Using extraordinarily fancy stack manipulation techniques, this could be
>> worked around, though that would not be based on completely standard
>> features found in any computing environment and thus not end up as
>> crossplatform as Gambit is today, and, it'd take a bit of performance.
> I am aware of these techniques - this is how I worked around the problem.
> Make a separate (mmap allocated) frame for the C stack, and trampoline
> through a scheme procedure that handles the i/o pumping and then ffi
> re-enters.
> It is not generally portable (some assembly  --
> setjmp/longjmp/getcontext/setcontext can't return pointer-sized values
> and don't provide any usable mechanism for storing the context for
> multiple threads, and there is also the issue of redzoning the stack
> to catch overflows), but it works for now.
>> When Gambit gets SMP support in some months from now, you'll have one C
>> stack per OS thread you launched, so you could do a bit more of this then.
> Not really.
>> Now back to the problem -
> I think you missed the point of the example code. This was abstracted
> from real code that triggered the exit and simplified to illustrate
> the issue while preserving sufficient structure from the original
> code.
>> So your c-lambda is:
>> (define do-read
>>   (c-lambda (scheme-object) int
>>        #<<END-C
>> char buf[4096];
>> ___SCMOBJ ctx = ___EXT(___make_vector)(2, ___FAL, ___STILL);
>> ___VECTORSET (ctx, ___FIX(0), ___arg1);
>> for (;;) {
>>   if (do_read ((void*)ctx, buf, sizeof(buf)) < 0) break;
>> }
>> END-C
>> ))
>> To solve your issue, rewrite it to Scheme i.e.
>> (lambda (v)
>>   (let ((buf (make-u8vector 4096))
>>         (ctx (make-still-vector 2 #f)))
>>     (vector-set! ctx 0 1)
>>     (let loop ()
>>       (if (< 0 (do-read ctx buf (u8vector-length buf)) (loop)))
> Nope, this doesn't solve anything as mentioned above. The streams are infinite.
> It also doesn't work when there are message boundaries for interactive
> protocols that need to be processed by a C-library.
> The scheme code has no knowledge of framing and required input sizes,
> the C library provides this information in the pump upcall as it
> discovers it.
>> Again, what you are wanting to get is an extremely high level functionality,
>> namely using a IO multiplexer, in this case the one of Gambit's scheduler,
>> to return to different places in code in essentially different stacks - this
>> would generally never be done in C, and this remains the case also here.
>> Generally, never make a Scheme-to-C call with a C-to-Scheme call that may
>> block in any way, except for if such use is done once in parallell within
>> the OS thread. C's stack model does not deliver for other than exactly this.
>> Instead, make your C call return back to Scheme with some kind of status
>> value that instructs your Scheme code how to proceed. Such an approach will
>> always end up being 'good code'.
> Good code doesn't force you to stand on your head to work around
> limitations, it is written directly to do the task at hand (but that's
> just my opinion, I have no interest in flamewars about such
> generalities).
> Again, the issue is the limitation of the ffi.
>> If you need to pass complex result data back from the C call to Scheme, you
>> can do this as C-to-Scheme calls though then keep the thread system off
>> during those calls as to keep your code safe from the possibility of other
>> code compromising the C stack in relation with the task you're performing
>> and thus crashing your program.
>> If you would really need multiple C stacks, then, presuming they do C logics
>> only and do not directly invoke any C code beyond your control (such as OS
>> routines), you could use a C cooperate multithreading library to get the
>> functionality you look for. That would be sufficiently crossplatform.
> Gambit already has cross-platform user threads, you shouldn't have to
> shoehorn another threading system into the platform.
> It also doesn't quite solve the problem, what you need is a way to do
> coroutines that cross the language barriers - see the description of
> the work-around I used above.
> -- vyzo
> _______________________________________________
> Gambit-list mailing list
> Gambit-list at iro.umontreal.ca
> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list

More information about the Gambit-list mailing list