Gambit and event-based systems
Hi, Specially in the mobile world, there are many platforms that force the developer to think and code its applications around events. Sometimes this is done with virtual functions or interface implementation, overriding methods, or plain C-style callbacks. The problem is, in any of these cases, generally the main loop is controlled by the application. Since Gambit runs within the ___setup(params) function, and then quits (expecting ___cleanup() to be called), there is no way to easily adapt to this architecture except for running in a separate thread and communicating through messages or shared data. Is there a better way to do this? Best regards, Álvaro Castro-Castilla Digital Experiences. -- visit The Blog <http://blog.fourthbit.com/> <http://fourthbit.com>
Hallo, On Thu, Aug 7, 2014 at 1:46 PM, Álvaro Castro-Castilla <alvaro.castro.castilla@gmail.com> wrote:
Hi,
Specially in the mobile world, there are many platforms that force the developer to think and code its applications around events. Sometimes this is done with virtual functions or interface implementation, overriding methods, or plain C-style callbacks.
The problem is, in any of these cases, generally the main loop is controlled by the application. Since Gambit runs within the ___setup(params) function, and then quits (expecting ___cleanup() to be called), there is no way to easily adapt to this architecture except for running in a separate thread and communicating through messages or shared data.
Is there a better way to do this?
Why can't you package your program as a C library that has an API for the events? Cheers, -- -alex http://unendli.ch/
On Thu, Aug 7, 2014 at 1:56 PM, Alex Queiroz <asandroq@gmail.com> wrote:
Hallo,
On Thu, Aug 7, 2014 at 1:46 PM, Álvaro Castro-Castilla <alvaro.castro.castilla@gmail.com> wrote:
Hi,
Specially in the mobile world, there are many platforms that force the
developer to think and code its applications around events. Sometimes this is done with virtual functions or interface implementation, overriding methods, or plain C-style callbacks.
The problem is, in any of these cases, generally the main loop is
controlled by the application. Since Gambit runs within the ___setup(params) function, and then quits (expecting ___cleanup() to be called), there is no way to easily adapt to this architecture except for running in a separate thread and communicating through messages or shared data.
Is there a better way to do this?
Why can't you package your program as a C library that has an API for the events?
That works for implementing the events, but where is the Gambit ___setup function running? You can't return from this function without breaking Gambit's system. If there is just one thread, how can you run Gambit, then leave the ___setup function and let the system call whatever event it requires? For instance, in SDL2, you are required to define a draw/update callback in iOS and then leave the main. If you don't return from Gambit's ___setup() in main, the draw/update callback is never run.
Hallo, On Thu, Aug 7, 2014 at 2:13 PM, Álvaro Castro-Castilla <alvaro.castro.castilla@gmail.com> wrote:
That works for implementing the events, but where is the Gambit ___setup function running? You can't return from this function without breaking Gambit's system.
`___setup` doesn't need to be "running". You call it and after it is finished, you can call your functions. If your code needs to do work in the background, you can install a timer in the host system and keep calling into Gambit. Cheers, -- -alex http://unendli.ch/
On Thu, Aug 7, 2014 at 2:19 PM, Alex Queiroz <asandroq@gmail.com> wrote:
Hallo,
On Thu, Aug 7, 2014 at 2:13 PM, Álvaro Castro-Castilla <alvaro.castro.castilla@gmail.com> wrote:
That works for implementing the events, but where is the Gambit ___setup function running? You can't return from this function without breaking Gambit's system.
`___setup` doesn't need to be "running". You call it and after it is finished, you can call your functions. If your code needs to do work in the background, you can install a timer in the host system and keep calling into Gambit.
Yes, you are right. I've actually done it long time ago, now that I recall. Let me try the idea, and see if I can cleanly satisfy the library requirement. But yes, apparently I wasn't thinking, since if you don't have to keep running ___setup() then it's easy to implement callbacks. Actually, the name ___setup and the fact that ___cleanup is an independent function makes it pretty clear. Otherwise it would have been ___run or something similar. The confusion stemmed from the fact that I was running Scheme code directly without calling any (c-define)'d function after ___setup(). But as I said, I've done that before, so I wasn't thinking. Thank you!
On Thu, Aug 7, 2014 at 2:31 PM, Álvaro Castro-Castilla < alvaro.castro.castilla@gmail.com> wrote:
On Thu, Aug 7, 2014 at 2:19 PM, Alex Queiroz <asandroq@gmail.com> wrote:
Hallo,
On Thu, Aug 7, 2014 at 2:13 PM, Álvaro Castro-Castilla <alvaro.castro.castilla@gmail.com> wrote:
That works for implementing the events, but where is the Gambit ___setup function running? You can't return from this function without breaking Gambit's system.
`___setup` doesn't need to be "running". You call it and after it is finished, you can call your functions. If your code needs to do work in the background, you can install a timer in the host system and keep calling into Gambit.
Yes, you are right. I've actually done it long time ago, now that I recall. Let me try the idea, and see if I can cleanly satisfy the library requirement.
But yes, apparently I wasn't thinking, since if you don't have to keep running ___setup() then it's easy to implement callbacks. Actually, the name ___setup and the fact that ___cleanup is an independent function makes it pretty clear. Otherwise it would have been ___run or something similar. The confusion stemmed from the fact that I was running Scheme code directly without calling any (c-define)'d function after ___setup(). But as I said, I've done that before, so I wasn't thinking.
Thank you!
Okay, it wasn't that easy. I *do* need to have a Gambit thread running a loop in the background as well. How can I keep that loop running? When you said "install a timer in the host system and keep calling" you mean that I need to break up that loop into a callback which is called with a repeating timer? (I think this is not a good solution). And then, where would you create and start that thread?
Okay, it wasn't that easy. I *do* need to have a Gambit thread running a loop in the background as well. How can I keep that loop running?
What do you need to do in this thread exactly? I imagine that loop will keep running as long as you are in the Scheme world, that is, handling any type of event. Unfortunately, in an event-based system your app will by default spend much of the time in the C world, waiting for things to happen. A way to work around that is suggested below.
When you said "install a timer in the host system and keep calling" you mean that I need to break up that loop into a callback which is called with a repeating timer? (I think this is not a good solution).
If you really prefer to implement your background task as a loop in a gambit thread, you can communicate between the 'main' (event-system controlled) and 'worker' gambit threads via gambit thread mailboxes. So your timer handler just sends a message to the worker thread and waits for a message in response. So the worker loop blocks waiting for a message at the beginning of each iteration and sends a message back at the end. The content of these messages can be bogus and ignored both ways. What this achieves is setting apart some time to give your logic a chance to run. Also, many event-based systems have an 'on idle' event type such that if you register to listen for it, your handler will get triggered whenever the application is doing nothing. So if you don't really need your background logic to happen in regularly timed ticks, using the idle handler instead might make better use of processing time left over from event handling and rendering. As a step further in this direction, you can have *all* your event handlers works like this, just delegating the real work to the "worker" gambit thread through mailboxes, and then you can make your code feel pretty much as though you controlled the main event loop. In this case, the messages sent to mailboxes would contain information about the events. So the worker thread can switch over event type, just as in plain SDL. Note you still need a timer or idle handler to have stuff happen outside of user input and redraw events. In this case your "worker" thread looks and feels pretty much like a "main" thread. The event handlers themselves only contain stubs.
And then, where would you create and start that thread?
I don't know the specifics of your platform. In some, starting the event loop is done explicitly, so you can create the thread before you do that. In others, there is an 'init' event type. Where would you, for example, initialize OpenGL state that will not change throughout your application? I bet somewhere around there you might be able to launch that thread too. As a last resort, you can just do it the first time that any event handler gets called.
On Thu, Aug 7, 2014 at 6:37 PM, Estevo <euccastro@gmail.com> wrote:
Okay, it wasn't that easy. I *do* need to have a Gambit thread running a
loop in the background as well. How can I keep that loop running?
What do you need to do in this thread exactly?
I imagine that loop will keep running as long as you are in the Scheme world, that is, handling any type of event. Unfortunately, in an event-based system your app will by default spend much of the time in the C world, waiting for things to happen. A way to work around that is suggested below.
When you said "install a timer in the host system and keep calling" you mean that I need to break up that loop into a callback which is called with a repeating timer? (I think this is not a good solution).
If you really prefer to implement your background task as a loop in a gambit thread, you can communicate between the 'main' (event-system controlled) and 'worker' gambit threads via gambit thread mailboxes. So your timer handler just sends a message to the worker thread and waits for a message in response. So the worker loop blocks waiting for a message at the beginning of each iteration and sends a message back at the end. The content of these messages can be bogus and ignored both ways. What this achieves is setting apart some time to give your logic a chance to run.
Also, many event-based systems have an 'on idle' event type such that if you register to listen for it, your handler will get triggered whenever the application is doing nothing. So if you don't really need your background logic to happen in regularly timed ticks, using the idle handler instead might make better use of processing time left over from event handling and rendering.
As a step further in this direction, you can have *all* your event handlers works like this, just delegating the real work to the "worker" gambit thread through mailboxes, and then you can make your code feel pretty much as though you controlled the main event loop. In this case, the messages sent to mailboxes would contain information about the events. So the worker thread can switch over event type, just as in plain SDL. Note you still need a timer or idle handler to have stuff happen outside of user input and redraw events.
In this case your "worker" thread looks and feels pretty much like a "main" thread. The event handlers themselves only contain stubs.
And then, where would you create and start that thread?
I don't know the specifics of your platform. In some, starting the event loop is done explicitly, so you can create the thread before you do that. In others, there is an 'init' event type. Where would you, for example, initialize OpenGL state that will not change throughout your application? I bet somewhere around there you might be able to launch that thread too.
As a last resort, you can just do it the first time that any event handler gets called.
Thank you for your replies. I'm working with SDL2, using SchemeSpheres' bindings. Everything is working smoothly, except for this requirement at the bottom of the README: http://hg.libsdl.org/SDL/file/704a0bfecf75/README-ios.txt It is easy to make it work, unless you want to run some continuous process in the background. Specifically, I'm running a remote REPL, which requires running (##repl-debug-main) after setting it up for the TCP ports. Once you call this function, you don't return from it until you exit the REPL, AFAIK. I see how you could use SDL timers to call at certain intervals some Scheme code "in the background", but this code will be run in a different thread, breaking Gambit. You can avoid running the function in a different thread as explained here http://wiki.libsdl.org/SDL_AddTimer?highlight=%28\bCategoryAPI\b%29|%28SDLFunctionTemplate%29 but that would block Gambit until (##repl-debug-main) exits, since is not running in a different Gambit thread.
One thing I'd try would be to launch a gambit thread running ##repl-debug-main and set up a timer or idle handler in the main gambit thread that just 'thread-sleep!'s for a small amount of time. Hopefully experimentation yields a delay that is small enough to be unnoticeable in your application, but large enough to give the REPL enough time to do its work. If you use a timer for this (or, I imagine, if you handle timers in Gambit at all), the trick to run it in the same OS thread would be required. I don't know, from the top of my head, whether and how SDL does idle events. A fancier approach would be to look up the implementation of ##repl-debug-main and try and see if there are any hooks by which the trick I mentioned in the previous message could be enabled. If there are not, you could try running your own modified version of ##repl-debug-main (factoring out the hooks that would enable the just-wait-for-as-long-as-the-REPL-is-doing-actual-work trick would be a nice contribution to gambit IMHO). On Thu, Aug 7, 2014 at 7:06 PM, Álvaro Castro-Castilla < alvaro.castro.castilla@gmail.com> wrote:
On Thu, Aug 7, 2014 at 6:37 PM, Estevo <euccastro@gmail.com> wrote:
Okay, it wasn't that easy. I *do* need to have a Gambit thread running a
loop in the background as well. How can I keep that loop running?
What do you need to do in this thread exactly?
I imagine that loop will keep running as long as you are in the Scheme world, that is, handling any type of event. Unfortunately, in an event-based system your app will by default spend much of the time in the C world, waiting for things to happen. A way to work around that is suggested below.
When you said "install a timer in the host system and keep calling" you mean that I need to break up that loop into a callback which is called with a repeating timer? (I think this is not a good solution).
If you really prefer to implement your background task as a loop in a gambit thread, you can communicate between the 'main' (event-system controlled) and 'worker' gambit threads via gambit thread mailboxes. So your timer handler just sends a message to the worker thread and waits for a message in response. So the worker loop blocks waiting for a message at the beginning of each iteration and sends a message back at the end. The content of these messages can be bogus and ignored both ways. What this achieves is setting apart some time to give your logic a chance to run.
Also, many event-based systems have an 'on idle' event type such that if you register to listen for it, your handler will get triggered whenever the application is doing nothing. So if you don't really need your background logic to happen in regularly timed ticks, using the idle handler instead might make better use of processing time left over from event handling and rendering.
As a step further in this direction, you can have *all* your event handlers works like this, just delegating the real work to the "worker" gambit thread through mailboxes, and then you can make your code feel pretty much as though you controlled the main event loop. In this case, the messages sent to mailboxes would contain information about the events. So the worker thread can switch over event type, just as in plain SDL. Note you still need a timer or idle handler to have stuff happen outside of user input and redraw events.
In this case your "worker" thread looks and feels pretty much like a "main" thread. The event handlers themselves only contain stubs.
And then, where would you create and start that thread?
I don't know the specifics of your platform. In some, starting the event loop is done explicitly, so you can create the thread before you do that. In others, there is an 'init' event type. Where would you, for example, initialize OpenGL state that will not change throughout your application? I bet somewhere around there you might be able to launch that thread too.
As a last resort, you can just do it the first time that any event handler gets called.
Thank you for your replies.
I'm working with SDL2, using SchemeSpheres' bindings. Everything is working smoothly, except for this requirement at the bottom of the README:
http://hg.libsdl.org/SDL/file/704a0bfecf75/README-ios.txt
It is easy to make it work, unless you want to run some continuous process in the background. Specifically, I'm running a remote REPL, which requires running (##repl-debug-main) after setting it up for the TCP ports. Once you call this function, you don't return from it until you exit the REPL, AFAIK.
I see how you could use SDL timers to call at certain intervals some Scheme code "in the background", but this code will be run in a different thread, breaking Gambit. You can avoid running the function in a different thread as explained here http://wiki.libsdl.org/SDL_AddTimer?highlight=%28\bCategoryAPI\b%29|%28SDLFunctionTemplate%29 <http://wiki.libsdl.org/SDL_AddTimer?highlight=%28%5CbCategoryAPI%5Cb%29%7C%28SDLFunctionTemplate%29> but that would block Gambit until (##repl-debug-main) exits, since is not running in a different Gambit thread.
On Fri, Aug 8, 2014 at 1:19 AM, Estevo <euccastro@gmail.com> wrote:
One thing I'd try would be to launch a gambit thread running ##repl-debug-main and set up a timer or idle handler in the main gambit thread that just 'thread-sleep!'s for a small amount of time. Hopefully experimentation yields a delay that is small enough to be unnoticeable in your application, but large enough to give the REPL enough time to do its work. If you use a timer for this (or, I imagine, if you handle timers in Gambit at all), the trick to run it in the same OS thread would be required. I don't know, from the top of my head, whether and how SDL does idle events.
A fancier approach would be to look up the implementation of ##repl-debug-main and try and see if there are any hooks by which the trick I mentioned in the previous message could be enabled. If there are not, you could try running your own modified version of ##repl-debug-main (factoring out the hooks that would enable the just-wait-for-as-long-as-the-REPL-is-doing-actual-work trick would be a nice contribution to gambit IMHO).
Thanks all for your insights and help :) I found a solution, which is very simple: just create the REPL thread, then create the SDL Cocoa draw loop callback, and finally let the main function return. Since the callback is going to be called repeatedly by iOS, the REPL thread will be resumed since it didn't terminate. The key insight is realizing that the current thread in Scheme returns to C, but the other threads will continue their execution as soon as you come back to Scheme. It works now, with both custom application loops and event-loop hooks models.
On Fri, Aug 8, 2014 at 11:56 AM, Álvaro Castro-Castilla < alvaro.castro.castilla@gmail.com> wrote:
On Fri, Aug 8, 2014 at 1:19 AM, Estevo <euccastro@gmail.com> wrote:
One thing I'd try would be to launch a gambit thread running ##repl-debug-main and set up a timer or idle handler in the main gambit thread that just 'thread-sleep!'s for a small amount of time. Hopefully experimentation yields a delay that is small enough to be unnoticeable in your application, but large enough to give the REPL enough time to do its work. If you use a timer for this (or, I imagine, if you handle timers in Gambit at all), the trick to run it in the same OS thread would be required. I don't know, from the top of my head, whether and how SDL does idle events.
A fancier approach would be to look up the implementation of ##repl-debug-main and try and see if there are any hooks by which the trick I mentioned in the previous message could be enabled. If there are not, you could try running your own modified version of ##repl-debug-main (factoring out the hooks that would enable the just-wait-for-as-long-as-the-REPL-is-doing-actual-work trick would be a nice contribution to gambit IMHO).
Thanks all for your insights and help :)
I found a solution, which is very simple: just create the REPL thread, then create the SDL Cocoa draw loop callback, and finally let the main function return. Since the callback is going to be called repeatedly by iOS, the REPL thread will be resumed since it didn't terminate. The key insight is realizing that the current thread in Scheme returns to C, but the other threads will continue their execution as soon as you come back to Scheme.
It works now, with both custom application loops and event-loop hooks models.
After all this, folks at SDL list just told me that this is actually a bug in the README, and you don't need the callback method any longer. You can take full control of the loop at the moment. The only reason this was necessary was for interaction with the Game Center, but apparently it isn't as of now. Well, at least I learnt something about Gambit threads - FFI interaction.
Yay! A caveat: I've never touched iOS, but some event-based systems will not call back unless an event for which you have registered is triggered. In games, where you redraw the whole screen many times per second this is not a problem, since the 'draw' callbacks should give plenty of opportunity for Scheme land code to run. But in a more static GUI application this might not be the case. It could be that the REPL freezes until the user interacts with the app window in some way. Hence the suggestion for timer or idle handlers. On Fri, Aug 8, 2014 at 11:56 AM, Álvaro Castro-Castilla < alvaro.castro.castilla@gmail.com> wrote:
On Fri, Aug 8, 2014 at 1:19 AM, Estevo <euccastro@gmail.com> wrote:
One thing I'd try would be to launch a gambit thread running ##repl-debug-main and set up a timer or idle handler in the main gambit thread that just 'thread-sleep!'s for a small amount of time. Hopefully experimentation yields a delay that is small enough to be unnoticeable in your application, but large enough to give the REPL enough time to do its work. If you use a timer for this (or, I imagine, if you handle timers in Gambit at all), the trick to run it in the same OS thread would be required. I don't know, from the top of my head, whether and how SDL does idle events.
A fancier approach would be to look up the implementation of ##repl-debug-main and try and see if there are any hooks by which the trick I mentioned in the previous message could be enabled. If there are not, you could try running your own modified version of ##repl-debug-main (factoring out the hooks that would enable the just-wait-for-as-long-as-the-REPL-is-doing-actual-work trick would be a nice contribution to gambit IMHO).
Thanks all for your insights and help :)
I found a solution, which is very simple: just create the REPL thread, then create the SDL Cocoa draw loop callback, and finally let the main function return. Since the callback is going to be called repeatedly by iOS, the REPL thread will be resumed since it didn't terminate. The key insight is realizing that the current thread in Scheme returns to C, but the other threads will continue their execution as soon as you come back to Scheme.
It works now, with both custom application loops and event-loop hooks models.
On Fri, Aug 8, 2014 at 1:08 PM, Estevo <euccastro@gmail.com> wrote:
Yay!
A caveat: I've never touched iOS, but some event-based systems will not call back unless an event for which you have registered is triggered. In games, where you redraw the whole screen many times per second this is not a problem, since the 'draw' callbacks should give plenty of opportunity for Scheme land code to run. But in a more static GUI application this might not be the case. It could be that the REPL freezes until the user interacts with the app window in some way. Hence the suggestion for timer or idle handlers.
Yes, that's a good observation :) In SDL, specially for the drawing callback, it will always be the case that you are constantly re-entering the Scheme code. But you are right, for static GUIs you'll need a callback. Anyway, all this was for nothing, since SDL seems to work now as normally, giving you full control of the loop. This is how it should be, as it makes the code way more portable between all supported platforms.
On Thu, Aug 07, 2014 at 01:46:42PM +0200, Álvaro Castro-Castilla wrote:
Hi,
Specially in the mobile world, there are many platforms that force the developer to think and code its applications around events. Sometimes this is done with virtual functions or interface implementation, overriding methods, or plain C-style callbacks.
The problem is, in any of these cases, generally the main loop is controlled by the application. Since Gambit runs within the ___setup(params) function, and then quits (expecting ___cleanup() to be called), there is no way to easily adapt to this architecture except for running in a separate thread and communicating through messages or shared data.
Is there a better way to do this?
This is a fundamental flaw in event-loop systems -- that they have to take over control. And they are usually not designed with the understanding that the whole application might need two such subsystems. Proper design would design an event-loop so that (a) the user can provide it with additional things to be done during each loop and/or (b) There's a function that can be called that performs one iteration of the event-loop, so that the user can have a loop that calls it frequently. Even so, there are probably things that are difficult, such as avoiding busy-waiting. Sometimes separate threads are the best answer, if the synchronisation mecchanisms are fast enough. -- hendrik
Yes, I'll add that my response was meant for *one* main-loop-hoarding framework. If you need to use more than one of these, things get hairy quickly and it's hard to make general suggestions without knowing the exact APIs you're given. On Thu, Aug 7, 2014 at 3:26 PM, Hendrik Boom <hendrik@topoi.pooq.com> wrote:
On Thu, Aug 07, 2014 at 01:46:42PM +0200, Álvaro Castro-Castilla wrote:
Hi,
Specially in the mobile world, there are many platforms that force the developer to think and code its applications around events. Sometimes this is done with virtual functions or interface implementation, overriding methods, or plain C-style callbacks.
The problem is, in any of these cases, generally the main loop is controlled by the application. Since Gambit runs within the ___setup(params) function, and then quits (expecting ___cleanup() to be called), there is no way to easily adapt to this architecture except for running in a separate thread and communicating through messages or shared data.
Is there a better way to do this?
This is a fundamental flaw in event-loop systems -- that they have to take over control. And they are usually not designed with the understanding that the whole application might need two such subsystems.
Proper design would design an event-loop so that (a) the user can provide it with additional things to be done during each loop and/or (b) There's a function that can be called that performs one iteration of the event-loop, so that the user can have a loop that calls it frequently.
Even so, there are probably things that are difficult, such as avoiding busy-waiting. Sometimes separate threads are the best answer, if the synchronisation mecchanisms are fast enough.
-- hendrik _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
participants (4)
-
Alex Queiroz -
Estevo -
Hendrik Boom -
Álvaro Castro-Castilla