Hello Marc,
rtl;dr: I dare to bet that this is either a bug or a to-be-documented shortcoming of Gambit Scheme. No proof, however, just a wager: Thread switches within a Scheme-to-C-to-Scheme callback (no matter what caused them, blocking operations or heartbeat) from one module which cause another Scheme-to-C-to-Scheme call (in another thread and another module/Scheme file) to return can/will confuse gambit runtime.
The Long Story:
The issue appears to be the than-expected regression of my question from May: https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html when I wrote:
But I'm afraid it might be a red herring and just mitigate the effect for the hell to come back under more load.
The lambdanative project disabled heartbeat interrupts (on Android, i.e. on slower hardware) a while back "because it caused issues". Which issues exactly is "lost in constitutional memory" I learned when I asked. Well lambdanative uses many external libraries.
What I did after I asked a week ago:
a) I extended the code attached to the posting cited above. Now, whenever calling into Scheme the heartbeat interrupts are disabled too. New version attached.
Towards the end of the file there is a c-safe-lambda macro definition I use instead of c-lambda whenever the C call might call back to Scheme. Additionally I use ##safe-lambda-post! from within C-to-Scheme callbacks to postpone thunks for execution after the Scheme-to-C call returned. All possibly blocking operations, `thread-start!` etc. are deferred that way.
This was rolled out onto 6 machines and works since without the issue coming up again.
b) I tried the old code on the slowest of those machine. After a short while the tight loop re-appeared. Then I tried your suggestions:
Am Sun, 2 Aug 2020 13:29:32 -0400 schrieb Marc Feeley feeley@iro.umontreal.ca:
Hello Jorg. The heartbeat interrupts are not necessarily problematic. They will occur if your program is running, for example in a busy loop (in your program or the runtime library). Can you interrupt your program to get a REPL and see what it is doing?
No. Not when I locked up. When I leave a repl running right from the start it does no longer respond either.
A convenient way to do this is to start your program with the -:daR runtime option. When you hit ctrl-C a REPL will be started in the context of what the program was executing, so a “,b” command will give you a backtrace.
OK. This one I did not try. But i tried that version:
Another way is to start your program with the -:d$ runtime option that starts a REPL server listening on port 44555. Then you can start a REPL by connecting to that port (for example with the “nc” program) and use the “top” procedure to see what the threads are doing. For example, in one shell do
% gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
and in another shell do
% $ nc localhost 44555 Gambit v4.9.3-1201-g3b320533
No. nc will connect only as long as the issue did not come up. Afterwards if will hang in the connection attempt.
Note that it might be really tough to find a simple test case to reproduce the issue using heartbeats. I my case there are two libraries lwIP and onetierzero[1]. For each of them there is a C file/module with gambit ffi bindings. Both are driven from two sources: each needs a timer for housekeeping tasks and when gambit receives network traffic the corresponding gambit thread will call them too. So each network packet will cause roughly two Scheme-to-C-to-Scheme calls. Still it takes hours to enter the issue via heartbeat.
However the posting from may might make things easier: when two Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex) in each two modules and then the mutexs unlocked in the same sequence as they where locked, the issue *should* arise. (Speculating here: I did not yet try this one yet.)
Best Jörg
[1] https://github.com/0-8-15/onetierzero
------------------ for reference only -----------
(top 20 (thread-thread-group #1))
*** THREAD LIST: #<thread #1 primordial> SLEEPING 3.8s
The “top” procedure will monitor the threads for 20 seconds (the default is 10 seconds in the current thread’s thread group) and refresh the status every second. In the above the primordial thread was sleeping (and 3.8 seconds were left to sleep). If you remove the (thread-sleep! 5) to get an infinite loop, the status would be
#<thread #1 primordial> RUNNING P0
which indicates that the primordial thread is running (on processor #0). If a thread is blocked reading a port you will see something like
#<thread #1 primordial> WAITING #<input-port #2 (stdin)>
If on the other hand your program is totally unresponsive, it could be a bug in Gambit’s runtime system. This is much harder to debug, especially if it only happens after a long time. In that case you could compile Gambit with --enable-debug and not --enable-single-host and use gdb or lldb to attach to the process and interrupt it. If you also configure Gambit with --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program when it seems to be stuck and you will get a history of the 1024 last Scheme procedure calls, plus a C backtrace, on the gambit.log file. This should improve your understanding of the issue…
Marc
On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger Joerg.Wittenberger@softeyes.net wrote:
Hello,
on Linux I observe gambit entering a tight loop.
Attached the result of "strace -r -p <pid> -o ot0.trace".
To me this looks like pattern of roughly 15/80/15/118... repeating the heartbeat operation. In this state the program takes almost 100% CPU and does not respond to anything. Normally it runs on around 40-60 open file descriptors (most from UDP and TCP).
The problem arises essentially for sure. Depending on the hardware sooner or later. (On 32bit armel after hours to few days, aarch64 takes sometimes almost week, AMD64 even longer.)
Anybody having an idea how to track this down?
Thanks so much
Jörg <ot0.trace>_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list