Dimitris,
(From only looking very generally at your code,)
May this be a stack unwinding problem?
If you unwind a stack other than in the same order it was wound in the first place, your Gambit process will terminate or at least crash.
Example:
You have a lambda A that invokes C-lambda B that invokes lambda C.
Also, a lambda D that invokes C-lambda E that invokes lambda F.
If you invoke A and A invokes B and B invokes C,
and then say, Gambit switches thread, and that Gambit thread invokes D which invokes E which invokes F,
and then, say, Gambit switches thread again, and that Gambit thread returns from C to B.
Then your Gambit process is toast.
To handle this you can switch off Gambit's threading system when you're in B or E for the course of your C and F calls respectively. Note that switching off Gambit's threading system means that some or several of Gambit's built-in IO primitives would block forever.
See the "How to work with external libraries" Wiki page for how to switch it off.
Was this of help?
Mikael
2013/2/23 Dimitris Vyzovitis vyzo@hackzen.org
I am running into an abrupt exit with error code 71 (OSError) when multiple (scheme) threads are making ffi calls that upcall to scheme.
A simplified scenario (see attached file) is as follows: 2 or more threads make an ffi call. The ffi call is a longer running procedure, that makes upcalls into scheme procedures that do I/O. This results in the program exiting abruptly with error code 71, and no context information regarding the error (even when compiled and running with full debugging).
For an illustration of the problem, see the attached file.
The following should print dots indefinitely, but instead quickly terminates with 71.
$ gsc exit71.scm $ gsi -e "(begin (load "exit71") (main))" ....<plenty of dots>.... $ echo $? 71
This is with gambit-4.6.6, system is GNU/Linux x86-64.
-- vyzo
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Afficher les réponses par date
Switching threads off is not an option because of I/O.
So basically you are saying that ffi scheme upcalls (through c-define'd lambdas) are not thread safe (scheme threads always). I would expect each thread to maintain its own c stack, but apparently it is not so and C-stacks cannot be safely interleaved with multiple threads.
If this is the case, it is a pretty serious limitation in the ffi.
PS: wiki seems down again
On Sat, Feb 23, 2013 at 1:14 AM, Mikael mikael.rcv@gmail.com wrote:
Dimitris,
(From only looking very generally at your code,)
May this be a stack unwinding problem?
If you unwind a stack other than in the same order it was wound in the first place, your Gambit process will terminate or at least crash.
Example:
You have a lambda A that invokes C-lambda B that invokes lambda C.
Also, a lambda D that invokes C-lambda E that invokes lambda F.
If you invoke A and A invokes B and B invokes C,
and then say, Gambit switches thread, and that Gambit thread invokes D which invokes E which invokes F,
and then, say, Gambit switches thread again, and that Gambit thread returns from C to B.
Then your Gambit process is toast.
To handle this you can switch off Gambit's threading system when you're in B or E for the course of your C and F calls respectively. Note that switching off Gambit's threading system means that some or several of Gambit's built-in IO primitives would block forever.
See the "How to work with external libraries" Wiki page for how to switch it off.
Was this of help?
Mikael
2013/2/23 Dimitris Vyzovitis vyzo@hackzen.org
I am running into an abrupt exit with error code 71 (OSError) when multiple (scheme) threads are making ffi calls that upcall to scheme.
A simplified scenario (see attached file) is as follows: 2 or more threads make an ffi call. The ffi call is a longer running procedure, that makes upcalls into scheme procedures that do I/O. This results in the program exiting abruptly with error code 71, and no context information regarding the error (even when compiled and running with full debugging).
For an illustration of the problem, see the attached file.
The following should print dots indefinitely, but instead quickly terminates with 71.
$ gsc exit71.scm $ gsi -e "(begin (load "exit71") (main))" ....<plenty of dots>.... $ echo $? 71
This is with gambit-4.6.6, system is GNU/Linux x86-64.
-- vyzo
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Hi Dmitris,
Indeed it is a kind of limitation, though limitation is an aspect of the nature of any technology solution.
Find below some reflections about the design area your example code regards and a suggestion for best practice.
I checked your code closer now; basically the c-lambda is a part of the application logics that could as well have been implemented in Scheme.
Whenever you can implement application logics in Scheme, do it, because the C stack model is as you have found out limited to "direct style" i.e. each step up must be balanced with one step down and there are *no* shortcuts. This is the C stack, and how it's delivered to you in the FFI.
Of course it follows that if you attempt to do maneuvers on the C stack that are not possible from Scheme, something breaks.
Using extraordinarily fancy stack manipulation techniques, this could be worked around, though that would not be based on completely standard features found in any computing environment and thus not end up as crossplatform as Gambit is today, and, it'd take a bit of performance.
When Gambit gets SMP support in some months from now, you'll have one C stack per OS thread you launched, so you could do a bit more of this then.
Now back to the problem -
So your c-lambda is:
(define do-read (c-lambda (scheme-object) int #<<END-C char buf[4096]; ___SCMOBJ ctx = ___EXT(___make_vector)(2, ___FAL, ___STILL); ___VECTORSET (ctx, ___FIX(0), ___arg1);
for (;;) { if (do_read ((void*)ctx, buf, sizeof(buf)) < 0) break; } END-C ))
To solve your issue, rewrite it to Scheme i.e.
(lambda (v) (let ((buf (make-u8vector 4096)) (ctx (make-still-vector 2 #f))) (vector-set! ctx 0 1) (let loop () (if (< 0 (do-read ctx buf (u8vector-length buf)) (loop)))
Again, what you are wanting to get is an extremely high level functionality, namely using a IO multiplexer, in this case the one of Gambit's scheduler, to return to different places in code in essentially different stacks - this would generally never be done in C, and this remains the case also here.
Generally, never make a Scheme-to-C call with a C-to-Scheme call that may block in any way, except for if such use is done once in parallell within the OS thread. C's stack model does not deliver for other than exactly this. Instead, make your C call return back to Scheme with some kind of status value that instructs your Scheme code how to proceed. Such an approach will always end up being 'good code'.
If you need to pass complex result data back from the C call to Scheme, you can do this as C-to-Scheme calls though then keep the thread system off during those calls as to keep your code safe from the possibility of other code compromising the C stack in relation with the task you're performing and thus crashing your program.
If you would really need multiple C stacks, then, presuming they do C logics only and do not directly invoke any C code beyond your control (such as OS routines), you could use a C cooperate multithreading library to get the functionality you look for. That would be sufficiently crossplatform.
What are you trying to do?
Brgds, Mikael
2013/2/24 Dimitris Vyzovitis vyzo@hackzen.org
Switching threads off is not an option because of I/O.
So basically you are saying that ffi scheme upcalls (through c-define'd lambdas) are not thread safe (scheme threads always). I would expect each thread to maintain its own c stack, but apparently it is not so and C-stacks cannot be safely interleaved with multiple threads.
If this is the case, it is a pretty serious limitation in the ffi.
PS: wiki seems down again
On Sat, Feb 23, 2013 at 1:14 AM, Mikael mikael.rcv@gmail.com wrote:
Dimitris,
(From only looking very generally at your code,)
May this be a stack unwinding problem?
If you unwind a stack other than in the same order it was wound in the
first
place, your Gambit process will terminate or at least crash.
Example:
You have a lambda A that invokes C-lambda B that invokes lambda C.
Also, a lambda D that invokes C-lambda E that invokes lambda F.
If you invoke A and A invokes B and B invokes C,
and then say, Gambit switches thread, and that Gambit thread invokes D
which
invokes E which invokes F,
and then, say, Gambit switches thread again, and that Gambit thread
returns
from C to B.
Then your Gambit process is toast.
To handle this you can switch off Gambit's threading system when you're
in B
or E for the course of your C and F calls respectively. Note that
switching
off Gambit's threading system means that some or several of Gambit's built-in IO primitives would block forever.
See the "How to work with external libraries" Wiki page for how to
switch it
off.
Was this of help?
Mikael
2013/2/23 Dimitris Vyzovitis vyzo@hackzen.org
I am running into an abrupt exit with error code 71 (OSError) when multiple (scheme) threads are making ffi calls that upcall to scheme.
A simplified scenario (see attached file) is as follows: 2 or more threads make an ffi call. The ffi call is a longer running procedure, that makes upcalls into scheme procedures that do I/O. This results in the program exiting abruptly with error code 71, and no context information regarding the error (even when compiled and running with full debugging).
For an illustration of the problem, see the attached file.
The following should print dots indefinitely, but instead quickly terminates with 71.
$ gsc exit71.scm $ gsi -e "(begin (load "exit71") (main))" ....<plenty of dots>.... $ echo $? 71
This is with gambit-4.6.6, system is GNU/Linux x86-64.
-- vyzo
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Sun, Feb 24, 2013 at 8:23 AM, Mikael mikael.rcv@gmail.com wrote:
Indeed it is a kind of limitation, though limitation is an aspect of the nature of any technology solution.
Find below some reflections about the design area your example code regards and a suggestion for best practice.
Sorry, but I wasn't waxing philosophically here. This is a limitation in the ffi that makes certain kinds of programs integrating with C impossible to write without hairy workarounds (more below). The issue should at least be in the radar -- a fix is even better.
Two immediately affected classes of programs: 1. Progressive I/O through a C library. This is the case my example code was intended to illustrate. The general scenario is a C library that provides a mechanism for pulling data through custom I/O objects. Note that the streams may be infinite or have message boundaries, so doing full port reads at scheme level can't work. This is common when you are doing systems programming.
2. Event loops in the C library. Same issue with the upcalls, they basically can't run threads or do nested ffi calls. This is common for gui systems etc.
I checked your code closer now; basically the c-lambda is a part of the application logics that could as well have been implemented in Scheme.
This code was simplified to illustrate the problem. Scheme implementation as you suggest is impossible (the streams are infinite, did you catch the /dev/zero?)
Whenever you can implement application logics in Scheme, do it, because the C stack model is as you have found out limited to "direct style" i.e. each step up must be balanced with one step down and there are *no* shortcuts. This is the C stack, and how it's delivered to you in the FFI.
The implementation of ___call already captures frame markers and conveys the intent of being re-entrant. It is possible that frames get overwritten by different returns, but this could be addressed by reseving enough stack space for the interleaved C frames from different threads. Also, some context would be nice when aborting (even better, just crash instead of the nasty exit. so that the problem is immediately obvious and can be debugged with gdb)
Of course it follows that if you attempt to do maneuvers on the C stack that are not possible from Scheme, something breaks.
Using extraordinarily fancy stack manipulation techniques, this could be worked around, though that would not be based on completely standard features found in any computing environment and thus not end up as crossplatform as Gambit is today, and, it'd take a bit of performance.
I am aware of these techniques - this is how I worked around the problem. Make a separate (mmap allocated) frame for the C stack, and trampoline through a scheme procedure that handles the i/o pumping and then ffi re-enters. It is not generally portable (some assembly -- setjmp/longjmp/getcontext/setcontext can't return pointer-sized values and don't provide any usable mechanism for storing the context for multiple threads, and there is also the issue of redzoning the stack to catch overflows), but it works for now.
When Gambit gets SMP support in some months from now, you'll have one C stack per OS thread you launched, so you could do a bit more of this then.
Not really.
Now back to the problem -
I think you missed the point of the example code. This was abstracted from real code that triggered the exit and simplified to illustrate the issue while preserving sufficient structure from the original code.
So your c-lambda is:
(define do-read (c-lambda (scheme-object) int #<<END-C char buf[4096]; ___SCMOBJ ctx = ___EXT(___make_vector)(2, ___FAL, ___STILL); ___VECTORSET (ctx, ___FIX(0), ___arg1);
for (;;) { if (do_read ((void*)ctx, buf, sizeof(buf)) < 0) break; } END-C ))
To solve your issue, rewrite it to Scheme i.e.
(lambda (v) (let ((buf (make-u8vector 4096)) (ctx (make-still-vector 2 #f))) (vector-set! ctx 0 1) (let loop () (if (< 0 (do-read ctx buf (u8vector-length buf)) (loop)))
Nope, this doesn't solve anything as mentioned above. The streams are infinite.
It also doesn't work when there are message boundaries for interactive protocols that need to be processed by a C-library. The scheme code has no knowledge of framing and required input sizes, the C library provides this information in the pump upcall as it discovers it.
Again, what you are wanting to get is an extremely high level functionality, namely using a IO multiplexer, in this case the one of Gambit's scheduler, to return to different places in code in essentially different stacks - this would generally never be done in C, and this remains the case also here.
Generally, never make a Scheme-to-C call with a C-to-Scheme call that may block in any way, except for if such use is done once in parallell within the OS thread. C's stack model does not deliver for other than exactly this. Instead, make your C call return back to Scheme with some kind of status value that instructs your Scheme code how to proceed. Such an approach will always end up being 'good code'.
Good code doesn't force you to stand on your head to work around limitations, it is written directly to do the task at hand (but that's just my opinion, I have no interest in flamewars about such generalities). Again, the issue is the limitation of the ffi.
If you need to pass complex result data back from the C call to Scheme, you can do this as C-to-Scheme calls though then keep the thread system off during those calls as to keep your code safe from the possibility of other code compromising the C stack in relation with the task you're performing and thus crashing your program.
If you would really need multiple C stacks, then, presuming they do C logics only and do not directly invoke any C code beyond your control (such as OS routines), you could use a C cooperate multithreading library to get the functionality you look for. That would be sufficiently crossplatform.
Gambit already has cross-platform user threads, you shouldn't have to shoehorn another threading system into the platform. It also doesn't quite solve the problem, what you need is a way to do coroutines that cross the language barriers - see the description of the work-around I used above.
-- vyzo
I am writing a compiler for a coroutine-based language and have the exact same limitation with regards to the FFI.
I am curious, Dimitris, whether you know of any other language with first-class coroutines or coroutines that have your desired feature. I myself do not. Perhaps it's still an open research problem.
-Patrick
On Sun, Feb 24, 2013 at 1:51 PM, Dimitris Vyzovitis vyzo@hackzen.org wrote:
On Sun, Feb 24, 2013 at 8:23 AM, Mikael mikael.rcv@gmail.com wrote:
Indeed it is a kind of limitation, though limitation is an aspect of the nature of any technology solution.
Find below some reflections about the design area your example code regards and a suggestion for best practice.
Sorry, but I wasn't waxing philosophically here. This is a limitation in the ffi that makes certain kinds of programs integrating with C impossible to write without hairy workarounds (more below). The issue should at least be in the radar -- a fix is even better.
Two immediately affected classes of programs:
- Progressive I/O through a C library. This is the case my example
code was intended to illustrate. The general scenario is a C library that provides a mechanism for pulling data through custom I/O objects. Note that the streams may be infinite or have message boundaries, so doing full port reads at scheme level can't work. This is common when you are doing systems programming.
- Event loops in the C library. Same issue with the upcalls, they
basically can't run threads or do nested ffi calls. This is common for gui systems etc.
I checked your code closer now; basically the c-lambda is a part of the application logics that could as well have been implemented in Scheme.
This code was simplified to illustrate the problem. Scheme implementation as you suggest is impossible (the streams are infinite, did you catch the /dev/zero?)
Whenever you can implement application logics in Scheme, do it, because the C stack model is as you have found out limited to "direct style" i.e. each step up must be balanced with one step down and there are *no* shortcuts. This is the C stack, and how it's delivered to you in the FFI.
The implementation of ___call already captures frame markers and conveys the intent of being re-entrant. It is possible that frames get overwritten by different returns, but this could be addressed by reseving enough stack space for the interleaved C frames from different threads. Also, some context would be nice when aborting (even better, just crash instead of the nasty exit. so that the problem is immediately obvious and can be debugged with gdb)
Of course it follows that if you attempt to do maneuvers on the C stack that are not possible from Scheme, something breaks.
Using extraordinarily fancy stack manipulation techniques, this could be worked around, though that would not be based on completely standard features found in any computing environment and thus not end up as crossplatform as Gambit is today, and, it'd take a bit of performance.
I am aware of these techniques - this is how I worked around the problem. Make a separate (mmap allocated) frame for the C stack, and trampoline through a scheme procedure that handles the i/o pumping and then ffi re-enters. It is not generally portable (some assembly -- setjmp/longjmp/getcontext/setcontext can't return pointer-sized values and don't provide any usable mechanism for storing the context for multiple threads, and there is also the issue of redzoning the stack to catch overflows), but it works for now.
When Gambit gets SMP support in some months from now, you'll have one C stack per OS thread you launched, so you could do a bit more of this then.
Not really.
Now back to the problem -
I think you missed the point of the example code. This was abstracted from real code that triggered the exit and simplified to illustrate the issue while preserving sufficient structure from the original code.
So your c-lambda is:
(define do-read (c-lambda (scheme-object) int #<<END-C char buf[4096]; ___SCMOBJ ctx = ___EXT(___make_vector)(2, ___FAL, ___STILL); ___VECTORSET (ctx, ___FIX(0), ___arg1);
for (;;) { if (do_read ((void*)ctx, buf, sizeof(buf)) < 0) break; } END-C ))
To solve your issue, rewrite it to Scheme i.e.
(lambda (v) (let ((buf (make-u8vector 4096)) (ctx (make-still-vector 2 #f))) (vector-set! ctx 0 1) (let loop () (if (< 0 (do-read ctx buf (u8vector-length buf)) (loop)))
Nope, this doesn't solve anything as mentioned above. The streams are infinite.
It also doesn't work when there are message boundaries for interactive protocols that need to be processed by a C-library. The scheme code has no knowledge of framing and required input sizes, the C library provides this information in the pump upcall as it discovers it.
Again, what you are wanting to get is an extremely high level functionality, namely using a IO multiplexer, in this case the one of Gambit's scheduler, to return to different places in code in essentially different stacks - this would generally never be done in C, and this remains the case also here.
Generally, never make a Scheme-to-C call with a C-to-Scheme call that may block in any way, except for if such use is done once in parallell within the OS thread. C's stack model does not deliver for other than exactly this. Instead, make your C call return back to Scheme with some kind of status value that instructs your Scheme code how to proceed. Such an approach will always end up being 'good code'.
Good code doesn't force you to stand on your head to work around limitations, it is written directly to do the task at hand (but that's just my opinion, I have no interest in flamewars about such generalities). Again, the issue is the limitation of the ffi.
If you need to pass complex result data back from the C call to Scheme, you can do this as C-to-Scheme calls though then keep the thread system off during those calls as to keep your code safe from the possibility of other code compromising the C stack in relation with the task you're performing and thus crashing your program.
If you would really need multiple C stacks, then, presuming they do C logics only and do not directly invoke any C code beyond your control (such as OS routines), you could use a C cooperate multithreading library to get the functionality you look for. That would be sufficiently crossplatform.
Gambit already has cross-platform user threads, you shouldn't have to shoehorn another threading system into the platform. It also doesn't quite solve the problem, what you need is a way to do coroutines that cross the language barriers - see the description of the work-around I used above.
-- vyzo _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Sun, Feb 24, 2013 at 2:25 PM, Patrick Li patrickli.2001@gmail.com wrote:
I am writing a compiler for a coroutine-based language and have the exact same limitation with regards to the FFI.
I am curious, Dimitris, whether you know of any other language with first-class coroutines or coroutines that have your desired feature. I myself do not. Perhaps it's still an open research problem.
I am not aware of any strictly coroutine-based solution, but it is not such a severe limitation if you are maintaining separate stacks for each thread.
The main problem there is dealing with stack growth, since C stacks are not generally movable -- if your allocated stack is too small you will overflow and if your stack is too large you are wasting too much memory. Still, the kernel doesn't back it by physical memory until it gets used, so you can jumbo-size them with the real limitation being your address space size (not too big of an issue in 64bit architectures).
It is also possible in some situations to grow the stacks as needed. When allocating with MAP_GROWSDOWN, the kernel may leave some space for growth, so when you catch the segfault in the redzone you can remap it. But it is not guaranteed how much room you will have (if any), and the address space quickly gets tight in 32bit.
Perhaps the best way around it is to use segmented (split) stacks. With recent gcc you can get compiler support at C-level (still not trivial to integrate with your language kernel, but doable). You may have to compile your entire C-library with segmented stack support for this to work transparently however.
You can also try to implement dynamic segmented stacks by stepping at your redzone handler until you catch a call instruction, at which point you can make a new stack and stitch the execution contexts. Not exactly a walk in the park though.
-- vyzo
Hi Dimitris,
I am aware of these techniques - this is how I worked around the problem. Make a separate (mmap allocated) frame for the C stack, and trampoline through a scheme procedure that handles the i/o pumping and then ffi re-enters. It is not generally portable (some assembly -- setjmp/longjmp/getcontext/setcontext can't return pointer-sized values and don't provide any usable mechanism for storing the context for multiple threads, and there is also the issue of redzoning the stack to catch overflows), but it works for now.
Neat! Are you aware of any good example code for this anywhere?
2013/2/25 Dimitris Vyzovitis vyzo@hackzen.org
On Sun, Feb 24, 2013 at 2:25 PM, Patrick Li patrickli.2001@gmail.com wrote:
I am writing a compiler for a coroutine-based language and have the exact same limitation with regards to the FFI.
I am curious, Dimitris, whether you know of any other language with first-class coroutines or coroutines that have your desired feature. I myself do not. Perhaps it's still an open research problem.
I am not aware of any strictly coroutine-based solution, but it is not such a severe limitation if you are maintaining separate stacks for each thread.
The main problem there is dealing with stack growth, since C stacks are not generally movable -- if your allocated stack is too small you will overflow and if your stack is too large you are wasting too much memory. Still, the kernel doesn't back it by physical memory until it gets used, so you can jumbo-size them with the real limitation being your address space size (not too big of an issue in 64bit architectures).
It is also possible in some situations to grow the stacks as needed. When allocating with MAP_GROWSDOWN, the kernel may leave some space for growth, so when you catch the segfault in the redzone you can remap it. But it is not guaranteed how much room you will have (if any), and the address space quickly gets tight in 32bit.
Perhaps the best way around it is to use segmented (split) stacks. With recent gcc you can get compiler support at C-level (still not trivial to integrate with your language kernel, but doable). You may have to compile your entire C-library with segmented stack support for this to work transparently however.
You can also try to implement dynamic segmented stacks by stepping at your redzone handler until you catch a call instruction, at which point you can make a new stack and stitch the execution contexts. Not exactly a walk in the park though.
-- vyzo
Same here - are you aware of particularly good documentation or use examples on it?
Regards, Mikael
On Sun, Feb 24, 2013 at 4:25 PM, Mikael mikael.rcv@gmail.com wrote:
...
Neat! Are you aware of any good example code for this anywhere?
You can read the glibc sources for some general techniques in implementing userland context switching -- check the code for setjmp/longjmp and the *context family (makecontext/swapcontext/getcontext/setcontext). If you want to see a full threading system using these techniques, check the source for gnu pth.
If I get around polishing my hack enough to be more generally useful, I can post it to the list.
2013/2/25 Dimitris Vyzovitis vyzo@hackzen.org
On Sun, Feb 24, 2013 at 2:25 PM, Patrick Li patrickli.2001@gmail.com wrote:
...
Same here - are you aware of particularly good documentation or use examples on it?
For handling stack overflows, man mmap, mremap and mprotect.
The basic idea is mmap your stack, mprotect your bottom pages and install a segfault handler with SA_SIGINFO (man sigaction) so that you get the context at the handler. You can find the exact address of the fault in the siginfo struct (si_addr), verify that it is indeed a fault in your redzone and then you can remap (or throw your hands in the air and abort :).
You can also jump out of the signal handler and manipulate execution context using the ucontext_t you get as third argument. Note that the context is live -- it is stored by the kernel just under the signal handler frame. So any changes you make to the registers there will be live in the process once you exit the handler.
You will also want to use sigaltstack (man sigaltstack) to setup an alternate stack for your signal handler, so that it doesn't double fault.
For segmented stacks check the gcc implementation; keywords: gcc split stacks
Hieb/Dybvig's classic cactus stack paper may also be of interest (Representing control in the presence of first-class continuations.)
-- vyzo