Hi Dimitris,
 
I am aware of these techniques - this is how I worked around the problem.
Make a separate (mmap allocated) frame for the C stack, and trampoline
through a scheme procedure that handles the i/o pumping and then ffi
re-enters.
It is not generally portable (some assembly  --
setjmp/longjmp/getcontext/setcontext can't return pointer-sized values
and don't provide any usable mechanism for storing the context for
multiple threads, and there is also the issue of redzoning the stack
to catch overflows), but it works for now.

Neat! Are you aware of any good example code for this anywhere?

2013/2/25 Dimitris Vyzovitis <vyzo@hackzen.org>
On Sun, Feb 24, 2013 at 2:25 PM, Patrick Li <patrickli.2001@gmail.com> wrote:
> I am writing a compiler for a coroutine-based language and have the
> exact same limitation with regards to the FFI.
>
> I am curious, Dimitris, whether you know of any other language with
> first-class coroutines or coroutines that have your desired feature. I
> myself do not. Perhaps it's still an open research problem.
>

I am not aware of any strictly coroutine-based solution, but it is not
such a severe limitation if you are maintaining separate stacks for
each thread.

The main problem there is dealing with stack growth, since C stacks
are not generally movable -- if your allocated stack is too small you
will overflow and if your stack is too large you are wasting too much
memory.  Still, the kernel doesn't back it by physical memory until it
gets used, so you can jumbo-size them with the real limitation being
your address space size (not too big of an issue in 64bit
architectures).

It is also possible in some situations to grow the stacks as needed.
When allocating with MAP_GROWSDOWN, the kernel may leave some space
for growth, so when you catch the segfault in the redzone you can
remap it. But it is not guaranteed how much room you will have (if
any), and the address space quickly gets tight in 32bit.

 
Perhaps the best way around it is to use segmented (split) stacks.
With recent gcc you can get compiler support at C-level (still not
trivial to integrate with your language kernel, but doable). You may
have to compile your entire C-library with segmented stack support for
this to work transparently however.

You can also try to implement dynamic segmented stacks by stepping at
your redzone handler until you catch a call instruction, at which
point you can make a new stack and stitch the execution contexts. Not
exactly a walk in the park though.

-- vyzo

Same here - are you aware of particularly good documentation or use examples on it?

Regards,
Mikael