[gambit-list] Need help to track down tight loop
Marc Feeley
feeley at iro.umontreal.ca
Fri Aug 7 13:31:19 EDT 2020
The situation you describe is not supported by Gambit. Any number of Scheme threads can call C functions, but only one Scheme thread at a time can do a Scheme to C call that calls back to Scheme. This is because the C stack is shared by all the Scheme to C calls… if the Scheme threads T1 and T2 call C (in the order T1 then T2) and if T1 returns to Scheme while T2’s call is still in progress, then T2’s C stack frame will be removed when T1’s C stack frame is removed (because T2’s frame was added after T1’s frame).
The Gambit manual describes a related case to avoid:
Gambit maintains the Scheme continuation separately from the C stack,
thus allowing the Scheme continuation to be unwound independently from
the C stack. The C stack frame created for the C function @samp{f} is
only removed from the C stack when control returns from @samp{f} or when
control returns to a C function ``above'' @samp{f}. Special care is
required for programs which escape to Scheme (using first-class
continuations) from a Scheme to C (to Scheme) call because the C stack
frame will remain on the stack. The C stack may overflow if this
happens in a loop with no intervening return to a C function. To avoid
this problem make sure the C stack gets cleaned up by executing a normal
return from a Scheme to C call.
Gambit Scheme thread context switching is implemented with continuations, so only one Scheme thread at a time can do Scheme to C to Scheme calls. One option is to disable context switching, but this is not easy to do reliably (both synchronous and preemptive context switching must be avoided).
Marc
> On Aug 7, 2020, at 11:59 AM, Jörg F. Wittenberger <Joerg.Wittenberger at softeyes.net> wrote:
>
> Hello Marc,
>
> rtl;dr: I dare to bet that this is either a bug or a to-be-documented
> shortcoming of Gambit Scheme. No proof, however, just a wager: Thread
> switches within a Scheme-to-C-to-Scheme callback (no matter what caused
> them, blocking operations or heartbeat) from one module which cause
> another Scheme-to-C-to-Scheme call (in another thread and another
> module/Scheme file) to return can/will confuse gambit runtime.
>
> The Long Story:
>
> The issue appears to be the than-expected regression of my question
> from May:
> https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html
> when I wrote:
>> But I'm afraid it might be a red herring and just mitigate the effect
>> for the hell to come back under more load.
>
> The lambdanative project disabled heartbeat interrupts (on Android,
> i.e. on slower hardware) a while back "because it caused issues".
> Which issues exactly is "lost in constitutional memory" I learned when I
> asked. Well lambdanative uses many external libraries.
>
> What I did after I asked a week ago:
>
> a) I extended the code attached to the posting cited above. Now,
> whenever calling into Scheme the heartbeat interrupts are disabled
> too. New version attached.
>
> Towards the end of the file there is a c-safe-lambda macro
> definition I use instead of c-lambda whenever the C call might call
> back to Scheme. Additionally I use ##safe-lambda-post! from
> within C-to-Scheme callbacks to postpone thunks for execution after
> the Scheme-to-C call returned. All possibly blocking operations,
> `thread-start!` etc. are deferred that way.
>
> This was rolled out onto 6 machines and works since without the issue
> coming up again.
>
> b) I tried the old code on the slowest of those machine. After a short
> while the tight loop re-appeared. Then I tried your suggestions:
>
> Am Sun, 2 Aug 2020 13:29:32 -0400
> schrieb Marc Feeley <feeley at iro.umontreal.ca>:
>
>> Hello Jorg. The heartbeat interrupts are not necessarily
>> problematic. They will occur if your program is running, for example
>> in a busy loop (in your program or the runtime library). Can you
>> interrupt your program to get a REPL and see what it is doing?
>
> No. Not when I locked up. When I leave a repl running right from the
> start it does no longer respond either.
>
>> A convenient way to do this is to start your program with the -:daR
>> runtime option. When you hit ctrl-C a REPL will be started in the
>> context of what the program was executing, so a “,b” command will
>> give you a backtrace.
>
> OK. This one I did not try. But i tried that version:
>
>> Another way is to start your program with the -:d$ runtime option
>> that starts a REPL server listening on port 44555. Then you can
>> start a REPL by connecting to that port (for example with the “nc”
>> program) and use the “top” procedure to see what the threads are
>> doing. For example, in one shell do
>>
>> % gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
>>
>> and in another shell do
>>
>> % $ nc localhost 44555
>> Gambit v4.9.3-1201-g3b320533
>
> No. nc will connect only as long as the issue did not come up.
> Afterwards if will hang in the connection attempt.
>
> Note that it might be really tough to find a simple test case to
> reproduce the issue using heartbeats. I my case there are two libraries
> lwIP and onetierzero[1]. For each of them there is a C file/module with
> gambit ffi bindings. Both are driven from two sources: each needs a
> timer for housekeeping tasks and when gambit receives network traffic
> the corresponding gambit thread will call them too. So each network
> packet will cause roughly two Scheme-to-C-to-Scheme calls. Still it
> takes hours to enter the issue via heartbeat.
>
> However the posting from may might make things easier: when two
> Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex) in
> each two modules and then the mutexs unlocked in the same sequence as
> they where locked, the issue *should* arise. (Speculating here: I did
> not yet try this one yet.)
>
> Best Jörg
>
> [1] https://github.com/0-8-15/onetierzero
>
> ------------------ for reference only -----------
>
>>> (top 20 (thread-thread-group #1))
>> *** THREAD LIST:
>> #<thread #1 primordial> SLEEPING 3.8s
>>
>> The “top” procedure will monitor the threads for 20 seconds (the
>> default is 10 seconds in the current thread’s thread group) and
>> refresh the status every second. In the above the primordial thread
>> was sleeping (and 3.8 seconds were left to sleep). If you remove the
>> (thread-sleep! 5) to get an infinite loop, the status would be
>>
>> #<thread #1 primordial> RUNNING P0
>>
>> which indicates that the primordial thread is running (on processor
>> #0). If a thread is blocked reading a port you will see something
>> like
>>
>> #<thread #1 primordial> WAITING #<input-port #2 (stdin)>
>>
>> If on the other hand your program is totally unresponsive, it could
>> be a bug in Gambit’s runtime system. This is much harder to debug,
>> especially if it only happens after a long time. In that case you
>> could compile Gambit with --enable-debug and not --enable-single-host
>> and use gdb or lldb to attach to the process and interrupt it. If
>> you also configure Gambit with --enable-debug-c-backtrace and
>> --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program
>> when it seems to be stuck and you will get a history of the 1024 last
>> Scheme procedure calls, plus a C backtrace, on the gambit.log file.
>> This should improve your understanding of the issue…
>>
>> Marc
>>
>>
>>
>>> On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger
>>> <Joerg.Wittenberger at softeyes.net> wrote:
>>>
>>> Hello,
>>>
>>> on Linux I observe gambit entering a tight loop.
>>>
>>> Attached the result of "strace -r -p <pid> -o ot0.trace".
>>>
>>> To me this looks like pattern of roughly 15/80/15/118... repeating
>>> the heartbeat operation. In this state the program takes almost
>>> 100% CPU and does not respond to anything. Normally it runs on
>>> around 40-60 open file descriptors (most from UDP and TCP).
>>>
>>> The problem arises essentially for sure. Depending on the hardware
>>> sooner or later. (On 32bit armel after hours to few days, aarch64
>>> takes sometimes almost week, AMD64 even longer.)
>>>
>>> Anybody having an idea how to track this down?
>>>
>>> Thanks so much
>>>
>>> Jörg
>>> <ot0.trace>_______________________________________________
>>> Gambit-list mailing list
>>> Gambit-list at iro.umontreal.ca
>>> https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
>>
>>
>
> <0007-gambit-foreign.scm>
More information about the Gambit-list
mailing list