Hello,
on Linux I observe gambit entering a tight loop.
Attached the result of "strace -r -p <pid> -o ot0.trace".
To me this looks like pattern of roughly 15/80/15/118... repeating the heartbeat operation. In this state the program takes almost 100% CPU and does not respond to anything. Normally it runs on around 40-60 open file descriptors (most from UDP and TCP).
The problem arises essentially for sure. Depending on the hardware sooner or later. (On 32bit armel after hours to few days, aarch64 takes sometimes almost week, AMD64 even longer.)
Anybody having an idea how to track this down?
Thanks so much
Jörg
Afficher les réponses par date
Hello Jorg. The heartbeat interrupts are not necessarily problematic. They will occur if your program is running, for example in a busy loop (in your program or the runtime library). Can you interrupt your program to get a REPL and see what it is doing?
A convenient way to do this is to start your program with the -:daR runtime option. When you hit ctrl-C a REPL will be started in the context of what the program was executing, so a “,b” command will give you a backtrace.
Another way is to start your program with the -:d$ runtime option that starts a REPL server listening on port 44555. Then you can start a REPL by connecting to that port (for example with the “nc” program) and use the “top” procedure to see what the threads are doing. For example, in one shell do
% gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
and in another shell do
% $ nc localhost 44555 Gambit v4.9.3-1201-g3b320533
(top 20 (thread-thread-group #1))
*** THREAD LIST: #<thread #1 primordial> SLEEPING 3.8s
The “top” procedure will monitor the threads for 20 seconds (the default is 10 seconds in the current thread’s thread group) and refresh the status every second. In the above the primordial thread was sleeping (and 3.8 seconds were left to sleep). If you remove the (thread-sleep! 5) to get an infinite loop, the status would be
#<thread #1 primordial> RUNNING P0
which indicates that the primordial thread is running (on processor #0). If a thread is blocked reading a port you will see something like
#<thread #1 primordial> WAITING #<input-port #2 (stdin)>
If on the other hand your program is totally unresponsive, it could be a bug in Gambit’s runtime system. This is much harder to debug, especially if it only happens after a long time. In that case you could compile Gambit with --enable-debug and not --enable-single-host and use gdb or lldb to attach to the process and interrupt it. If you also configure Gambit with --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program when it seems to be stuck and you will get a history of the 1024 last Scheme procedure calls, plus a C backtrace, on the gambit.log file. This should improve your understanding of the issue…
Marc
On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello,
on Linux I observe gambit entering a tight loop.
Attached the result of "strace -r -p <pid> -o ot0.trace".
To me this looks like pattern of roughly 15/80/15/118... repeating the heartbeat operation. In this state the program takes almost 100% CPU and does not respond to anything. Normally it runs on around 40-60 open file descriptors (most from UDP and TCP).
The problem arises essentially for sure. Depending on the hardware sooner or later. (On 32bit armel after hours to few days, aarch64 takes sometimes almost week, AMD64 even longer.)
Anybody having an idea how to track this down?
Thanks so much
Jörg <ot0.trace>_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
Hi all,
I have downloaded minimosa.img.gz from the gambit@30 directory at github. I have got it to work on my old Dell 9 mini. I have some questions: * How do I save the contents of the screen into a u8-vector? * How do I draw to the screen. Ideally I would take an u8-vector and bitblt it to the screen. Is this possible?
Thank you very much for your help.
Best Regards,
Alexander
You have zero privacy anyway. Get over it. Scott McNealy 1999.
Hello Marc,
rtl;dr: I dare to bet that this is either a bug or a to-be-documented shortcoming of Gambit Scheme. No proof, however, just a wager: Thread switches within a Scheme-to-C-to-Scheme callback (no matter what caused them, blocking operations or heartbeat) from one module which cause another Scheme-to-C-to-Scheme call (in another thread and another module/Scheme file) to return can/will confuse gambit runtime.
The Long Story:
The issue appears to be the than-expected regression of my question from May: https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html when I wrote:
But I'm afraid it might be a red herring and just mitigate the effect for the hell to come back under more load.
The lambdanative project disabled heartbeat interrupts (on Android, i.e. on slower hardware) a while back "because it caused issues". Which issues exactly is "lost in constitutional memory" I learned when I asked. Well lambdanative uses many external libraries.
What I did after I asked a week ago:
a) I extended the code attached to the posting cited above. Now, whenever calling into Scheme the heartbeat interrupts are disabled too. New version attached.
Towards the end of the file there is a c-safe-lambda macro definition I use instead of c-lambda whenever the C call might call back to Scheme. Additionally I use ##safe-lambda-post! from within C-to-Scheme callbacks to postpone thunks for execution after the Scheme-to-C call returned. All possibly blocking operations, `thread-start!` etc. are deferred that way.
This was rolled out onto 6 machines and works since without the issue coming up again.
b) I tried the old code on the slowest of those machine. After a short while the tight loop re-appeared. Then I tried your suggestions:
Am Sun, 2 Aug 2020 13:29:32 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
Hello Jorg. The heartbeat interrupts are not necessarily problematic. They will occur if your program is running, for example in a busy loop (in your program or the runtime library). Can you interrupt your program to get a REPL and see what it is doing?
No. Not when I locked up. When I leave a repl running right from the start it does no longer respond either.
A convenient way to do this is to start your program with the -:daR runtime option. When you hit ctrl-C a REPL will be started in the context of what the program was executing, so a “,b” command will give you a backtrace.
OK. This one I did not try. But i tried that version:
Another way is to start your program with the -:d$ runtime option that starts a REPL server listening on port 44555. Then you can start a REPL by connecting to that port (for example with the “nc” program) and use the “top” procedure to see what the threads are doing. For example, in one shell do
% gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
and in another shell do
% $ nc localhost 44555 Gambit v4.9.3-1201-g3b320533
No. nc will connect only as long as the issue did not come up. Afterwards if will hang in the connection attempt.
Note that it might be really tough to find a simple test case to reproduce the issue using heartbeats. I my case there are two libraries lwIP and onetierzero[1]. For each of them there is a C file/module with gambit ffi bindings. Both are driven from two sources: each needs a timer for housekeeping tasks and when gambit receives network traffic the corresponding gambit thread will call them too. So each network packet will cause roughly two Scheme-to-C-to-Scheme calls. Still it takes hours to enter the issue via heartbeat.
However the posting from may might make things easier: when two Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex) in each two modules and then the mutexs unlocked in the same sequence as they where locked, the issue *should* arise. (Speculating here: I did not yet try this one yet.)
Best Jörg
[1] https://github.com/0-8-15/onetierzero
------------------ for reference only -----------
(top 20 (thread-thread-group #1))
*** THREAD LIST: #<thread #1 primordial> SLEEPING 3.8s
The “top” procedure will monitor the threads for 20 seconds (the default is 10 seconds in the current thread’s thread group) and refresh the status every second. In the above the primordial thread was sleeping (and 3.8 seconds were left to sleep). If you remove the (thread-sleep! 5) to get an infinite loop, the status would be
#<thread #1 primordial> RUNNING P0
which indicates that the primordial thread is running (on processor #0). If a thread is blocked reading a port you will see something like
#<thread #1 primordial> WAITING #<input-port #2 (stdin)>
If on the other hand your program is totally unresponsive, it could be a bug in Gambit’s runtime system. This is much harder to debug, especially if it only happens after a long time. In that case you could compile Gambit with --enable-debug and not --enable-single-host and use gdb or lldb to attach to the process and interrupt it. If you also configure Gambit with --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program when it seems to be stuck and you will get a history of the 1024 last Scheme procedure calls, plus a C backtrace, on the gambit.log file. This should improve your understanding of the issue…
Marc
On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello,
on Linux I observe gambit entering a tight loop.
Attached the result of "strace -r -p <pid> -o ot0.trace".
To me this looks like pattern of roughly 15/80/15/118... repeating the heartbeat operation. In this state the program takes almost 100% CPU and does not respond to anything. Normally it runs on around 40-60 open file descriptors (most from UDP and TCP).
The problem arises essentially for sure. Depending on the hardware sooner or later. (On 32bit armel after hours to few days, aarch64 takes sometimes almost week, AMD64 even longer.)
Anybody having an idea how to track this down?
Thanks so much
Jörg <ot0.trace>_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
The situation you describe is not supported by Gambit. Any number of Scheme threads can call C functions, but only one Scheme thread at a time can do a Scheme to C call that calls back to Scheme. This is because the C stack is shared by all the Scheme to C calls… if the Scheme threads T1 and T2 call C (in the order T1 then T2) and if T1 returns to Scheme while T2’s call is still in progress, then T2’s C stack frame will be removed when T1’s C stack frame is removed (because T2’s frame was added after T1’s frame).
The Gambit manual describes a related case to avoid:
Gambit maintains the Scheme continuation separately from the C stack, thus allowing the Scheme continuation to be unwound independently from the C stack. The C stack frame created for the C function @samp{f} is only removed from the C stack when control returns from @samp{f} or when control returns to a C function ``above'' @samp{f}. Special care is required for programs which escape to Scheme (using first-class continuations) from a Scheme to C (to Scheme) call because the C stack frame will remain on the stack. The C stack may overflow if this happens in a loop with no intervening return to a C function. To avoid this problem make sure the C stack gets cleaned up by executing a normal return from a Scheme to C call.
Gambit Scheme thread context switching is implemented with continuations, so only one Scheme thread at a time can do Scheme to C to Scheme calls. One option is to disable context switching, but this is not easy to do reliably (both synchronous and preemptive context switching must be avoided).
Marc
On Aug 7, 2020, at 11:59 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello Marc,
rtl;dr: I dare to bet that this is either a bug or a to-be-documented shortcoming of Gambit Scheme. No proof, however, just a wager: Thread switches within a Scheme-to-C-to-Scheme callback (no matter what caused them, blocking operations or heartbeat) from one module which cause another Scheme-to-C-to-Scheme call (in another thread and another module/Scheme file) to return can/will confuse gambit runtime.
The Long Story:
The issue appears to be the than-expected regression of my question from May: https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html when I wrote:
But I'm afraid it might be a red herring and just mitigate the effect for the hell to come back under more load.
The lambdanative project disabled heartbeat interrupts (on Android, i.e. on slower hardware) a while back "because it caused issues". Which issues exactly is "lost in constitutional memory" I learned when I asked. Well lambdanative uses many external libraries.
What I did after I asked a week ago:
a) I extended the code attached to the posting cited above. Now, whenever calling into Scheme the heartbeat interrupts are disabled too. New version attached.
Towards the end of the file there is a c-safe-lambda macro definition I use instead of c-lambda whenever the C call might call back to Scheme. Additionally I use ##safe-lambda-post! from within C-to-Scheme callbacks to postpone thunks for execution after the Scheme-to-C call returned. All possibly blocking operations, `thread-start!` etc. are deferred that way.
This was rolled out onto 6 machines and works since without the issue coming up again.
b) I tried the old code on the slowest of those machine. After a short while the tight loop re-appeared. Then I tried your suggestions:
Am Sun, 2 Aug 2020 13:29:32 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
Hello Jorg. The heartbeat interrupts are not necessarily problematic. They will occur if your program is running, for example in a busy loop (in your program or the runtime library). Can you interrupt your program to get a REPL and see what it is doing?
No. Not when I locked up. When I leave a repl running right from the start it does no longer respond either.
A convenient way to do this is to start your program with the -:daR runtime option. When you hit ctrl-C a REPL will be started in the context of what the program was executing, so a “,b” command will give you a backtrace.
OK. This one I did not try. But i tried that version:
Another way is to start your program with the -:d$ runtime option that starts a REPL server listening on port 44555. Then you can start a REPL by connecting to that port (for example with the “nc” program) and use the “top” procedure to see what the threads are doing. For example, in one shell do
% gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
and in another shell do
% $ nc localhost 44555 Gambit v4.9.3-1201-g3b320533
No. nc will connect only as long as the issue did not come up. Afterwards if will hang in the connection attempt.
Note that it might be really tough to find a simple test case to reproduce the issue using heartbeats. I my case there are two libraries lwIP and onetierzero[1]. For each of them there is a C file/module with gambit ffi bindings. Both are driven from two sources: each needs a timer for housekeeping tasks and when gambit receives network traffic the corresponding gambit thread will call them too. So each network packet will cause roughly two Scheme-to-C-to-Scheme calls. Still it takes hours to enter the issue via heartbeat.
However the posting from may might make things easier: when two Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex) in each two modules and then the mutexs unlocked in the same sequence as they where locked, the issue *should* arise. (Speculating here: I did not yet try this one yet.)
Best Jörg
[1] https://github.com/0-8-15/onetierzero
------------------ for reference only -----------
(top 20 (thread-thread-group #1))
*** THREAD LIST: #<thread #1 primordial> SLEEPING 3.8s
The “top” procedure will monitor the threads for 20 seconds (the default is 10 seconds in the current thread’s thread group) and refresh the status every second. In the above the primordial thread was sleeping (and 3.8 seconds were left to sleep). If you remove the (thread-sleep! 5) to get an infinite loop, the status would be
#<thread #1 primordial> RUNNING P0
which indicates that the primordial thread is running (on processor #0). If a thread is blocked reading a port you will see something like
#<thread #1 primordial> WAITING #<input-port #2 (stdin)>
If on the other hand your program is totally unresponsive, it could be a bug in Gambit’s runtime system. This is much harder to debug, especially if it only happens after a long time. In that case you could compile Gambit with --enable-debug and not --enable-single-host and use gdb or lldb to attach to the process and interrupt it. If you also configure Gambit with --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program when it seems to be stuck and you will get a history of the 1024 last Scheme procedure calls, plus a C backtrace, on the gambit.log file. This should improve your understanding of the issue…
Marc
On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello,
on Linux I observe gambit entering a tight loop.
Attached the result of "strace -r -p <pid> -o ot0.trace".
To me this looks like pattern of roughly 15/80/15/118... repeating the heartbeat operation. In this state the program takes almost 100% CPU and does not respond to anything. Normally it runs on around 40-60 open file descriptors (most from UDP and TCP).
The problem arises essentially for sure. Depending on the hardware sooner or later. (On 32bit armel after hours to few days, aarch64 takes sometimes almost week, AMD64 even longer.)
Anybody having an idea how to track this down?
Thanks so much
Jörg <ot0.trace>_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
<0007-gambit-foreign.scm>
Thank you Marc,
this essentially confirms my hypothesis uttered in May, that gambit threading might be equivalent to continuation capture and thus fall under section 19.7 of the manual.
May I suggest to add a line there that "Scheme thread context switching is implemented with continuations, so only one Scheme thread at a time can do Scheme to C to Scheme calls" in order to not trap too many newbies like me with the same rope.
Best
Jörg
Am Fri, 7 Aug 2020 13:31:19 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
The situation you describe is not supported by Gambit. Any number of Scheme threads can call C functions, but only one Scheme thread at a time can do a Scheme to C call that calls back to Scheme. This is because the C stack is shared by all the Scheme to C calls… if the Scheme threads T1 and T2 call C (in the order T1 then T2) and if T1 returns to Scheme while T2’s call is still in progress, then T2’s C stack frame will be removed when T1’s C stack frame is removed (because T2’s frame was added after T1’s frame).
The Gambit manual describes a related case to avoid:
Gambit maintains the Scheme continuation separately from the C stack, thus allowing the Scheme continuation to be unwound independently from the C stack. The C stack frame created for the C function @samp{f} is only removed from the C stack when control returns from @samp{f} or when control returns to a C function ``above'' @samp{f}. Special care is required for programs which escape to Scheme (using first-class continuations) from a Scheme to C (to Scheme) call because the C stack frame will remain on the stack. The C stack may overflow if this happens in a loop with no intervening return to a C function. To avoid this problem make sure the C stack gets cleaned up by executing a normal return from a Scheme to C call.
Gambit Scheme thread context switching is implemented with continuations, so only one Scheme thread at a time can do Scheme to C to Scheme calls. One option is to disable context switching, but this is not easy to do reliably (both synchronous and preemptive context switching must be avoided).
Marc
On Aug 7, 2020, at 11:59 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello Marc,
rtl;dr: I dare to bet that this is either a bug or a to-be-documented shortcoming of Gambit Scheme. No proof, however, just a wager: Thread switches within a Scheme-to-C-to-Scheme callback (no matter what caused them, blocking operations or heartbeat) from one module which cause another Scheme-to-C-to-Scheme call (in another thread and another module/Scheme file) to return can/will confuse gambit runtime.
The Long Story:
The issue appears to be the than-expected regression of my question from May: https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html when I wrote:
But I'm afraid it might be a red herring and just mitigate the effect for the hell to come back under more load.
The lambdanative project disabled heartbeat interrupts (on Android, i.e. on slower hardware) a while back "because it caused issues". Which issues exactly is "lost in constitutional memory" I learned when I asked. Well lambdanative uses many external libraries.
What I did after I asked a week ago:
a) I extended the code attached to the posting cited above. Now, whenever calling into Scheme the heartbeat interrupts are disabled too. New version attached.
Towards the end of the file there is a c-safe-lambda macro definition I use instead of c-lambda whenever the C call might call back to Scheme. Additionally I use ##safe-lambda-post! from within C-to-Scheme callbacks to postpone thunks for execution after the Scheme-to-C call returned. All possibly blocking operations, `thread-start!` etc. are deferred that way.
This was rolled out onto 6 machines and works since without the issue coming up again.
b) I tried the old code on the slowest of those machine. After a short while the tight loop re-appeared. Then I tried your suggestions:
Am Sun, 2 Aug 2020 13:29:32 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
Hello Jorg. The heartbeat interrupts are not necessarily problematic. They will occur if your program is running, for example in a busy loop (in your program or the runtime library). Can you interrupt your program to get a REPL and see what it is doing?
No. Not when I locked up. When I leave a repl running right from the start it does no longer respond either.
A convenient way to do this is to start your program with the -:daR runtime option. When you hit ctrl-C a REPL will be started in the context of what the program was executing, so a “,b” command will give you a backtrace.
OK. This one I did not try. But i tried that version:
Another way is to start your program with the -:d$ runtime option that starts a REPL server listening on port 44555. Then you can start a REPL by connecting to that port (for example with the “nc” program) and use the “top” procedure to see what the threads are doing. For example, in one shell do
% gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
and in another shell do
% $ nc localhost 44555 Gambit v4.9.3-1201-g3b320533
No. nc will connect only as long as the issue did not come up. Afterwards if will hang in the connection attempt.
Note that it might be really tough to find a simple test case to reproduce the issue using heartbeats. I my case there are two libraries lwIP and onetierzero[1]. For each of them there is a C file/module with gambit ffi bindings. Both are driven from two sources: each needs a timer for housekeeping tasks and when gambit receives network traffic the corresponding gambit thread will call them too. So each network packet will cause roughly two Scheme-to-C-to-Scheme calls. Still it takes hours to enter the issue via heartbeat.
However the posting from may might make things easier: when two Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex) in each two modules and then the mutexs unlocked in the same sequence as they where locked, the issue *should* arise. (Speculating here: I did not yet try this one yet.)
Best Jörg
[1] https://github.com/0-8-15/onetierzero
------------------ for reference only -----------
(top 20 (thread-thread-group #1))
*** THREAD LIST: #<thread #1 primordial> SLEEPING 3.8s
The “top” procedure will monitor the threads for 20 seconds (the default is 10 seconds in the current thread’s thread group) and refresh the status every second. In the above the primordial thread was sleeping (and 3.8 seconds were left to sleep). If you remove the (thread-sleep! 5) to get an infinite loop, the status would be
#<thread #1 primordial> RUNNING P0
which indicates that the primordial thread is running (on processor #0). If a thread is blocked reading a port you will see something like
#<thread #1 primordial> WAITING #<input-port #2 (stdin)>
If on the other hand your program is totally unresponsive, it could be a bug in Gambit’s runtime system. This is much harder to debug, especially if it only happens after a long time. In that case you could compile Gambit with --enable-debug and not --enable-single-host and use gdb or lldb to attach to the process and interrupt it. If you also configure Gambit with --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program when it seems to be stuck and you will get a history of the 1024 last Scheme procedure calls, plus a C backtrace, on the gambit.log file. This should improve your understanding of the issue…
Marc
On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello,
on Linux I observe gambit entering a tight loop.
Attached the result of "strace -r -p <pid> -o ot0.trace".
To me this looks like pattern of roughly 15/80/15/118... repeating the heartbeat operation. In this state the program takes almost 100% CPU and does not respond to anything. Normally it runs on around 40-60 open file descriptors (most from UDP and TCP).
The problem arises essentially for sure. Depending on the hardware sooner or later. (On 32bit armel after hours to few days, aarch64 takes sometimes almost week, AMD64 even longer.)
Anybody having an idea how to track this down?
Thanks so much
Jörg <ot0.trace>_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
<0007-gambit-foreign.scm>
Hello,
Sorry to bother everyone with what I’m sure is a silly-noob question…
I’m trying to learn about Scheme macros, and I’m having trouble getting what I think is a relatively straight-forward macro to work in Gambit.
This formulation of my “while” macro works just fine:
(define-macro (while test . code) `(let loop () (if ,test (begin ,@code (loop)) )))
~ ~ ~ ~ script output STARTS ~ ~ ~ ~ Script started on Fri Aug 7 15:45:49 2020 bash-3.2$ gsi Gambit v4.9.3
(define-macro (while test . code)
`(let loop () (if ,test (begin ,@code (loop)) )))
(define i 0) (while (< i 10)
(println "i = " i) (set! i (+ i 1)) ) i = 0 i = 1 i = 2 i = 3 i = 4 i = 5 i = 6 i = 7 i = 8 i = 9
,q
bash-3.2$ ~ ~ ~ ~ script output ENDS ~ ~ ~ ~
Note that the gambit interpreter was started WITHOUT any command-line arguments, and in particular without “-:s”.
My formulations of what I believe to be this same macro using SYNTAX-RULES and SYNTAX-CASE, both fail regardless of whether I start the interpreter with “-:s” or not:
Here’s my code:
(define-syntax while (syntax-rules () ((_ test code ...) (let loop () (if test (begin code ... (loop)) )))))
(define-syntax while (lambda (x) (syntax-case x () ((_ test code ...) (syntax (let loop () (if test (begin code ... (loop)) )))))))
And here is what happens when I try to run these:
~ ~ ~ ~ script output STARTS ~ ~ ~ ~ bash-3.2$ bash-3.2$ gsi Gambit v4.9.3
(define-syntax while
(syntax-rules () ((_ test code ...) (let loop () (if test (begin code ... (loop)) ))))) *** ERROR IN (console)@2.18 -- Ill-formed expression
,q
bash-3.2$ bash-3.2$ bash-3.2$ gsi -:s Gambit v4.9.3
(define-syntax while
(syntax-rules () ((_ test code ...) (let loop () (if test (begin code ... (loop)) ))))) *** ERROR -- Ill-formed special form: syntax-error (##syntax-error %%%%tmp18)
,q
bash-3.2$ bash-3.2$ bash-3.2$ bash-3.2$ gsi Gambit v4.9.3
(define-syntax while
(lambda (x) (syntax-case x () ((_ test code ...) (syntax (let loop () (if test (begin code ... (loop)) ))))))) *** ERROR IN (console)@3.22 -- Ill-formed expression
,q
bash-3.2$ gsi -:s Gambit v4.9.3
(define-syntax while
(lambda (x) (syntax-case x () ((_ test code ...) (syntax (let loop () (if test (begin code ... (loop)) ))))))) *** ERROR -- Ill-formed special form: syntax-error (##syntax-error %%%%tmp18)
,q bash-3.2$ ~ ~ ~ ~ script output ENDS ~ ~ ~ ~
I’m using the most up-to-date version of Gambit from the MacPorts collection (v4.9.3), on a relatively up-to-date MacBook computer to run these.
What am I missing?
And lastly, does my problem have anything to do with this?
https://github.com/gambit/gambit/issues/384
And if so, is there a newer version of gambit that I can submit to MacPorts to make macros work again?
Thanks, - William
Your code works fine with a recent build of Gambit. Please see the build instructions here: https://github.com/gambit/gambit
Marc
On Aug 7, 2020, at 4:25 PM, William william@soukoreff.com wrote:
Hello,
Sorry to bother everyone with what I’m sure is a silly-noob question…
I’m trying to learn about Scheme macros, and I’m having trouble getting what I think is a relatively straight-forward macro to work in Gambit.
This formulation of my “while” macro works just fine:
(define-macro (while test . code) `(let loop () (if ,test (begin ,@code (loop)) )))
~ ~ ~ ~ script output STARTS ~ ~ ~ ~ Script started on Fri Aug 7 15:45:49 2020 bash-3.2$ gsi Gambit v4.9.3
(define-macro (while test . code)
`(let loop () (if ,test (begin ,@code (loop)) )))
(define i 0) (while (< i 10)
(println "i = " i) (set! i (+ i 1)) ) i = 0 i = 1 i = 2 i = 3 i = 4 i = 5 i = 6 i = 7 i = 8 i = 9
,q
bash-3.2$ ~ ~ ~ ~ script output ENDS ~ ~ ~ ~
Note that the gambit interpreter was started WITHOUT any command-line arguments, and in particular without “-:s”.
My formulations of what I believe to be this same macro using SYNTAX-RULES and SYNTAX-CASE, both fail regardless of whether I start the interpreter with “-:s” or not:
Here’s my code:
(define-syntax while (syntax-rules () ((_ test code ...) (let loop () (if test (begin code ... (loop)) )))))
(define-syntax while (lambda (x) (syntax-case x () ((_ test code ...) (syntax (let loop () (if test (begin code ... (loop)) )))))))
And here is what happens when I try to run these:
~ ~ ~ ~ script output STARTS ~ ~ ~ ~ bash-3.2$ bash-3.2$ gsi Gambit v4.9.3
(define-syntax while
(syntax-rules () ((_ test code ...) (let loop () (if test (begin code ... (loop)) ))))) *** ERROR IN (console)@2.18 -- Ill-formed expression
,q
bash-3.2$ bash-3.2$ bash-3.2$ gsi -:s Gambit v4.9.3
(define-syntax while
(syntax-rules () ((_ test code ...) (let loop () (if test (begin code ... (loop)) ))))) *** ERROR -- Ill-formed special form: syntax-error (##syntax-error %%%%tmp18)
,q
bash-3.2$ bash-3.2$ bash-3.2$ bash-3.2$ gsi Gambit v4.9.3
(define-syntax while
(lambda (x) (syntax-case x () ((_ test code ...) (syntax (let loop () (if test (begin code ... (loop)) ))))))) *** ERROR IN (console)@3.22 -- Ill-formed expression
,q
bash-3.2$ gsi -:s Gambit v4.9.3
(define-syntax while
(lambda (x) (syntax-case x () ((_ test code ...) (syntax (let loop () (if test (begin code ... (loop)) ))))))) *** ERROR -- Ill-formed special form: syntax-error (##syntax-error %%%%tmp18)
,q bash-3.2$ ~ ~ ~ ~ script output ENDS ~ ~ ~ ~
I’m using the most up-to-date version of Gambit from the MacPorts collection (v4.9.3), on a relatively up-to-date MacBook computer to run these.
What am I missing?
And lastly, does my problem have anything to do with this?
https://github.com/gambit/gambit/issues/384
And if so, is there a newer version of gambit that I can submit to MacPorts to make macros work again?
Thanks,
- William
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
Good idea to add that to the documentation. Will do.
Marc
On Aug 7, 2020, at 2:49 PM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Thank you Marc,
this essentially confirms my hypothesis uttered in May, that gambit threading might be equivalent to continuation capture and thus fall under section 19.7 of the manual.
May I suggest to add a line there that "Scheme thread context switching is implemented with continuations, so only one Scheme thread at a time can do Scheme to C to Scheme calls" in order to not trap too many newbies like me with the same rope.
Best
Jörg
Am Fri, 7 Aug 2020 13:31:19 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
The situation you describe is not supported by Gambit. Any number of Scheme threads can call C functions, but only one Scheme thread at a time can do a Scheme to C call that calls back to Scheme. This is because the C stack is shared by all the Scheme to C calls… if the Scheme threads T1 and T2 call C (in the order T1 then T2) and if T1 returns to Scheme while T2’s call is still in progress, then T2’s C stack frame will be removed when T1’s C stack frame is removed (because T2’s frame was added after T1’s frame).
The Gambit manual describes a related case to avoid:
Gambit maintains the Scheme continuation separately from the C stack, thus allowing the Scheme continuation to be unwound independently from the C stack. The C stack frame created for the C function @samp{f} is only removed from the C stack when control returns from @samp{f} or when control returns to a C function ``above'' @samp{f}. Special care is required for programs which escape to Scheme (using first-class continuations) from a Scheme to C (to Scheme) call because the C stack frame will remain on the stack. The C stack may overflow if this happens in a loop with no intervening return to a C function. To avoid this problem make sure the C stack gets cleaned up by executing a normal return from a Scheme to C call.
Gambit Scheme thread context switching is implemented with continuations, so only one Scheme thread at a time can do Scheme to C to Scheme calls. One option is to disable context switching, but this is not easy to do reliably (both synchronous and preemptive context switching must be avoided).
Marc
On Aug 7, 2020, at 11:59 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello Marc,
rtl;dr: I dare to bet that this is either a bug or a to-be-documented shortcoming of Gambit Scheme. No proof, however, just a wager: Thread switches within a Scheme-to-C-to-Scheme callback (no matter what caused them, blocking operations or heartbeat) from one module which cause another Scheme-to-C-to-Scheme call (in another thread and another module/Scheme file) to return can/will confuse gambit runtime.
The Long Story:
The issue appears to be the than-expected regression of my question from May: https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html when I wrote:
But I'm afraid it might be a red herring and just mitigate the effect for the hell to come back under more load.
The lambdanative project disabled heartbeat interrupts (on Android, i.e. on slower hardware) a while back "because it caused issues". Which issues exactly is "lost in constitutional memory" I learned when I asked. Well lambdanative uses many external libraries.
What I did after I asked a week ago:
a) I extended the code attached to the posting cited above. Now, whenever calling into Scheme the heartbeat interrupts are disabled too. New version attached.
Towards the end of the file there is a c-safe-lambda macro definition I use instead of c-lambda whenever the C call might call back to Scheme. Additionally I use ##safe-lambda-post! from within C-to-Scheme callbacks to postpone thunks for execution after the Scheme-to-C call returned. All possibly blocking operations, `thread-start!` etc. are deferred that way.
This was rolled out onto 6 machines and works since without the issue coming up again.
b) I tried the old code on the slowest of those machine. After a short while the tight loop re-appeared. Then I tried your suggestions:
Am Sun, 2 Aug 2020 13:29:32 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
Hello Jorg. The heartbeat interrupts are not necessarily problematic. They will occur if your program is running, for example in a busy loop (in your program or the runtime library). Can you interrupt your program to get a REPL and see what it is doing?
No. Not when I locked up. When I leave a repl running right from the start it does no longer respond either.
A convenient way to do this is to start your program with the -:daR runtime option. When you hit ctrl-C a REPL will be started in the context of what the program was executing, so a “,b” command will give you a backtrace.
OK. This one I did not try. But i tried that version:
Another way is to start your program with the -:d$ runtime option that starts a REPL server listening on port 44555. Then you can start a REPL by connecting to that port (for example with the “nc” program) and use the “top” procedure to see what the threads are doing. For example, in one shell do
% gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
and in another shell do
% $ nc localhost 44555 Gambit v4.9.3-1201-g3b320533
No. nc will connect only as long as the issue did not come up. Afterwards if will hang in the connection attempt.
Note that it might be really tough to find a simple test case to reproduce the issue using heartbeats. I my case there are two libraries lwIP and onetierzero[1]. For each of them there is a C file/module with gambit ffi bindings. Both are driven from two sources: each needs a timer for housekeeping tasks and when gambit receives network traffic the corresponding gambit thread will call them too. So each network packet will cause roughly two Scheme-to-C-to-Scheme calls. Still it takes hours to enter the issue via heartbeat.
However the posting from may might make things easier: when two Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex) in each two modules and then the mutexs unlocked in the same sequence as they where locked, the issue *should* arise. (Speculating here: I did not yet try this one yet.)
Best Jörg
[1] https://github.com/0-8-15/onetierzero
------------------ for reference only -----------
(top 20 (thread-thread-group #1))
*** THREAD LIST: #<thread #1 primordial> SLEEPING 3.8s
The “top” procedure will monitor the threads for 20 seconds (the default is 10 seconds in the current thread’s thread group) and refresh the status every second. In the above the primordial thread was sleeping (and 3.8 seconds were left to sleep). If you remove the (thread-sleep! 5) to get an infinite loop, the status would be
#<thread #1 primordial> RUNNING P0
which indicates that the primordial thread is running (on processor #0). If a thread is blocked reading a port you will see something like
#<thread #1 primordial> WAITING #<input-port #2 (stdin)>
If on the other hand your program is totally unresponsive, it could be a bug in Gambit’s runtime system. This is much harder to debug, especially if it only happens after a long time. In that case you could compile Gambit with --enable-debug and not --enable-single-host and use gdb or lldb to attach to the process and interrupt it. If you also configure Gambit with --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program when it seems to be stuck and you will get a history of the 1024 last Scheme procedure calls, plus a C backtrace, on the gambit.log file. This should improve your understanding of the issue…
Marc
On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello,
on Linux I observe gambit entering a tight loop.
Attached the result of "strace -r -p <pid> -o ot0.trace".
To me this looks like pattern of roughly 15/80/15/118... repeating the heartbeat operation. In this state the program takes almost 100% CPU and does not respond to anything. Normally it runs on around 40-60 open file descriptors (most from UDP and TCP).
The problem arises essentially for sure. Depending on the hardware sooner or later. (On 32bit armel after hours to few days, aarch64 takes sometimes almost week, AMD64 even longer.)
Anybody having an idea how to track this down?
Thanks so much
Jörg <ot0.trace>_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
<0007-gambit-foreign.scm>
Reconsidering I wonder: how to combine independent gambit libraries?
In oder to do Scheme-to-C calls there has to be a global mutex (or some other way of synchronization) to protect against calls done in parallel.
Assuming the libraries all being independently developed, distributed and standalone usable, all of them need to import a reference the same mutex.
The only solution I see at this point: import from gambit core itself.
Do I miss the alternative or am hereby I asking for such a mutex?
Jörg
Am Sun, 9 Aug 2020 11:02:05 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
Good idea to add that to the documentation. Will do.
Marc
On Aug 7, 2020, at 2:49 PM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Thank you Marc,
this essentially confirms my hypothesis uttered in May, that gambit threading might be equivalent to continuation capture and thus fall under section 19.7 of the manual.
May I suggest to add a line there that "Scheme thread context switching is implemented with continuations, so only one Scheme thread at a time can do Scheme to C to Scheme calls" in order to not trap too many newbies like me with the same rope.
Best
Jörg
Am Fri, 7 Aug 2020 13:31:19 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
The situation you describe is not supported by Gambit. Any number of Scheme threads can call C functions, but only one Scheme thread at a time can do a Scheme to C call that calls back to Scheme. This is because the C stack is shared by all the Scheme to C calls… if the Scheme threads T1 and T2 call C (in the order T1 then T2) and if T1 returns to Scheme while T2’s call is still in progress, then T2’s C stack frame will be removed when T1’s C stack frame is removed (because T2’s frame was added after T1’s frame).
The Gambit manual describes a related case to avoid:
Gambit maintains the Scheme continuation separately from the C stack, thus allowing the Scheme continuation to be unwound independently from the C stack. The C stack frame created for the C function @samp{f} is only removed from the C stack when control returns from @samp{f} or when control returns to a C function ``above'' @samp{f}. Special care is required for programs which escape to Scheme (using first-class continuations) from a Scheme to C (to Scheme) call because the C stack frame will remain on the stack. The C stack may overflow if this happens in a loop with no intervening return to a C function. To avoid this problem make sure the C stack gets cleaned up by executing a normal return from a Scheme to C call.
Gambit Scheme thread context switching is implemented with continuations, so only one Scheme thread at a time can do Scheme to C to Scheme calls. One option is to disable context switching, but this is not easy to do reliably (both synchronous and preemptive context switching must be avoided).
Marc
On Aug 7, 2020, at 11:59 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello Marc,
rtl;dr: I dare to bet that this is either a bug or a to-be-documented shortcoming of Gambit Scheme. No proof, however, just a wager: Thread switches within a Scheme-to-C-to-Scheme callback (no matter what caused them, blocking operations or heartbeat) from one module which cause another Scheme-to-C-to-Scheme call (in another thread and another module/Scheme file) to return can/will confuse gambit runtime.
The Long Story:
The issue appears to be the than-expected regression of my question from May: https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html when I wrote:
But I'm afraid it might be a red herring and just mitigate the effect for the hell to come back under more load.
The lambdanative project disabled heartbeat interrupts (on Android, i.e. on slower hardware) a while back "because it caused issues". Which issues exactly is "lost in constitutional memory" I learned when I asked. Well lambdanative uses many external libraries.
What I did after I asked a week ago:
a) I extended the code attached to the posting cited above. Now, whenever calling into Scheme the heartbeat interrupts are disabled too. New version attached.
Towards the end of the file there is a c-safe-lambda macro definition I use instead of c-lambda whenever the C call might call back to Scheme. Additionally I use ##safe-lambda-post! from within C-to-Scheme callbacks to postpone thunks for execution after the Scheme-to-C call returned. All possibly blocking operations, `thread-start!` etc. are deferred that way.
This was rolled out onto 6 machines and works since without the issue coming up again.
b) I tried the old code on the slowest of those machine. After a short while the tight loop re-appeared. Then I tried your suggestions:
Am Sun, 2 Aug 2020 13:29:32 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
Hello Jorg. The heartbeat interrupts are not necessarily problematic. They will occur if your program is running, for example in a busy loop (in your program or the runtime library). Can you interrupt your program to get a REPL and see what it is doing?
No. Not when I locked up. When I leave a repl running right from the start it does no longer respond either.
A convenient way to do this is to start your program with the -:daR runtime option. When you hit ctrl-C a REPL will be started in the context of what the program was executing, so a “,b” command will give you a backtrace.
OK. This one I did not try. But i tried that version:
Another way is to start your program with the -:d$ runtime option that starts a REPL server listening on port 44555. Then you can start a REPL by connecting to that port (for example with the “nc” program) and use the “top” procedure to see what the threads are doing. For example, in one shell do
% gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
and in another shell do
% $ nc localhost 44555 Gambit v4.9.3-1201-g3b320533
No. nc will connect only as long as the issue did not come up. Afterwards if will hang in the connection attempt.
Note that it might be really tough to find a simple test case to reproduce the issue using heartbeats. I my case there are two libraries lwIP and onetierzero[1]. For each of them there is a C file/module with gambit ffi bindings. Both are driven from two sources: each needs a timer for housekeeping tasks and when gambit receives network traffic the corresponding gambit thread will call them too. So each network packet will cause roughly two Scheme-to-C-to-Scheme calls. Still it takes hours to enter the issue via heartbeat.
However the posting from may might make things easier: when two Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex) in each two modules and then the mutexs unlocked in the same sequence as they where locked, the issue *should* arise. (Speculating here: I did not yet try this one yet.)
Best Jörg
[1] https://github.com/0-8-15/onetierzero
------------------ for reference only -----------
(top 20 (thread-thread-group #1))
*** THREAD LIST: #<thread #1 primordial> SLEEPING 3.8s
The “top” procedure will monitor the threads for 20 seconds (the default is 10 seconds in the current thread’s thread group) and refresh the status every second. In the above the primordial thread was sleeping (and 3.8 seconds were left to sleep). If you remove the (thread-sleep! 5) to get an infinite loop, the status would be
#<thread #1 primordial> RUNNING P0
which indicates that the primordial thread is running (on processor #0). If a thread is blocked reading a port you will see something like
#<thread #1 primordial> WAITING #<input-port #2 (stdin)>
If on the other hand your program is totally unresponsive, it could be a bug in Gambit’s runtime system. This is much harder to debug, especially if it only happens after a long time. In that case you could compile Gambit with --enable-debug and not --enable-single-host and use gdb or lldb to attach to the process and interrupt it. If you also configure Gambit with --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program when it seems to be stuck and you will get a history of the 1024 last Scheme procedure calls, plus a C backtrace, on the gambit.log file. This should improve your understanding of the issue…
Marc
On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello,
on Linux I observe gambit entering a tight loop.
Attached the result of "strace -r -p <pid> -o ot0.trace".
To me this looks like pattern of roughly 15/80/15/118... repeating the heartbeat operation. In this state the program takes almost 100% CPU and does not respond to anything. Normally it runs on around 40-60 open file descriptors (most from UDP and TCP).
The problem arises essentially for sure. Depending on the hardware sooner or later. (On 32bit armel after hours to few days, aarch64 takes sometimes almost week, AMD64 even longer.)
Anybody having an idea how to track this down?
Thanks so much
Jörg <ot0.trace>_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
<0007-gambit-foreign.scm>
-
Alexander Shendi -
Jörg F. Wittenberger -
Marc Feeley -
William