[gambit-list] Need help to track down tight loop
Marc Feeley
feeley at iro.umontreal.ca
Sun Aug 9 11:02:05 EDT 2020
Good idea to add that to the documentation. Will do.
Marc
> On Aug 7, 2020, at 2:49 PM, Jörg F. Wittenberger <Joerg.Wittenberger at softeyes.net> wrote:
>
> Thank you Marc,
>
> this essentially confirms my hypothesis uttered in May, that gambit
> threading might be equivalent to continuation capture and thus fall
> under section 19.7 of the manual.
>
> May I suggest to add a line there that "Scheme thread context switching
> is implemented with continuations, so only one Scheme thread at a time
> can do Scheme to C to Scheme calls" in order to not trap too many
> newbies like me with the same rope.
>
> Best
>
> Jörg
>
> Am Fri, 7 Aug 2020 13:31:19 -0400
> schrieb Marc Feeley <feeley at iro.umontreal.ca>:
>
>> The situation you describe is not supported by Gambit. Any number of
>> Scheme threads can call C functions, but only one Scheme thread at a
>> time can do a Scheme to C call that calls back to Scheme. This is
>> because the C stack is shared by all the Scheme to C calls… if the
>> Scheme threads T1 and T2 call C (in the order T1 then T2) and if T1
>> returns to Scheme while T2’s call is still in progress, then T2’s C
>> stack frame will be removed when T1’s C stack frame is removed
>> (because T2’s frame was added after T1’s frame).
>>
>> The Gambit manual describes a related case to avoid:
>>
>> Gambit maintains the Scheme continuation separately from the C
>> stack, thus allowing the Scheme continuation to be unwound
>> independently from the C stack. The C stack frame created for the C
>> function @samp{f} is only removed from the C stack when control
>> returns from @samp{f} or when control returns to a C function
>> ``above'' @samp{f}. Special care is required for programs which
>> escape to Scheme (using first-class continuations) from a Scheme to C
>> (to Scheme) call because the C stack frame will remain on the stack.
>> The C stack may overflow if this happens in a loop with no
>> intervening return to a C function. To avoid this problem make sure
>> the C stack gets cleaned up by executing a normal return from a
>> Scheme to C call.
>>
>> Gambit Scheme thread context switching is implemented with
>> continuations, so only one Scheme thread at a time can do Scheme to C
>> to Scheme calls. One option is to disable context switching, but
>> this is not easy to do reliably (both synchronous and preemptive
>> context switching must be avoided).
>>
>> Marc
>>
>>
>>
>>> On Aug 7, 2020, at 11:59 AM, Jörg F. Wittenberger
>>> <Joerg.Wittenberger at softeyes.net> wrote:
>>>
>>> Hello Marc,
>>>
>>> rtl;dr: I dare to bet that this is either a bug or a
>>> to-be-documented shortcoming of Gambit Scheme. No proof, however,
>>> just a wager: Thread switches within a Scheme-to-C-to-Scheme
>>> callback (no matter what caused them, blocking operations or
>>> heartbeat) from one module which cause another
>>> Scheme-to-C-to-Scheme call (in another thread and another
>>> module/Scheme file) to return can/will confuse gambit runtime.
>>>
>>> The Long Story:
>>>
>>> The issue appears to be the than-expected regression of my question
>>> from May:
>>> https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html
>>> when I wrote:
>>>> But I'm afraid it might be a red herring and just mitigate the
>>>> effect for the hell to come back under more load.
>>>
>>> The lambdanative project disabled heartbeat interrupts (on Android,
>>> i.e. on slower hardware) a while back "because it caused issues".
>>> Which issues exactly is "lost in constitutional memory" I learned
>>> when I asked. Well lambdanative uses many external libraries.
>>>
>>> What I did after I asked a week ago:
>>>
>>> a) I extended the code attached to the posting cited above. Now,
>>> whenever calling into Scheme the heartbeat interrupts are disabled
>>> too. New version attached.
>>>
>>> Towards the end of the file there is a c-safe-lambda macro
>>> definition I use instead of c-lambda whenever the C call might
>>> call back to Scheme. Additionally I use ##safe-lambda-post! from
>>> within C-to-Scheme callbacks to postpone thunks for execution
>>> after the Scheme-to-C call returned. All possibly blocking
>>> operations, `thread-start!` etc. are deferred that way.
>>>
>>> This was rolled out onto 6 machines and works since without the
>>> issue coming up again.
>>>
>>> b) I tried the old code on the slowest of those machine. After a
>>> short while the tight loop re-appeared. Then I tried your
>>> suggestions:
>>>
>>> Am Sun, 2 Aug 2020 13:29:32 -0400
>>> schrieb Marc Feeley <feeley at iro.umontreal.ca>:
>>>
>>>> Hello Jorg. The heartbeat interrupts are not necessarily
>>>> problematic. They will occur if your program is running, for
>>>> example in a busy loop (in your program or the runtime library).
>>>> Can you interrupt your program to get a REPL and see what it is
>>>> doing?
>>>
>>> No. Not when I locked up. When I leave a repl running right from
>>> the start it does no longer respond either.
>>>
>>>> A convenient way to do this is to start your program with the -:daR
>>>> runtime option. When you hit ctrl-C a REPL will be started in the
>>>> context of what the program was executing, so a “,b” command will
>>>> give you a backtrace.
>>>
>>> OK. This one I did not try. But i tried that version:
>>>
>>>> Another way is to start your program with the -:d$ runtime option
>>>> that starts a REPL server listening on port 44555. Then you can
>>>> start a REPL by connecting to that port (for example with the “nc”
>>>> program) and use the “top” procedure to see what the threads are
>>>> doing. For example, in one shell do
>>>>
>>>> % gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
>>>>
>>>> and in another shell do
>>>>
>>>> % $ nc localhost 44555
>>>> Gambit v4.9.3-1201-g3b320533
>>>
>>> No. nc will connect only as long as the issue did not come up.
>>> Afterwards if will hang in the connection attempt.
>>>
>>> Note that it might be really tough to find a simple test case to
>>> reproduce the issue using heartbeats. I my case there are two
>>> libraries lwIP and onetierzero[1]. For each of them there is a C
>>> file/module with gambit ffi bindings. Both are driven from two
>>> sources: each needs a timer for housekeeping tasks and when gambit
>>> receives network traffic the corresponding gambit thread will call
>>> them too. So each network packet will cause roughly two
>>> Scheme-to-C-to-Scheme calls. Still it takes hours to enter the
>>> issue via heartbeat.
>>>
>>> However the posting from may might make things easier: when two
>>> Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex)
>>> in each two modules and then the mutexs unlocked in the same
>>> sequence as they where locked, the issue *should* arise.
>>> (Speculating here: I did not yet try this one yet.)
>>>
>>> Best Jörg
>>>
>>> [1] https://github.com/0-8-15/onetierzero
>>>
>>> ------------------ for reference only -----------
>>>
>>>>> (top 20 (thread-thread-group #1))
>>>> *** THREAD LIST:
>>>> #<thread #1 primordial> SLEEPING 3.8s
>>>>
>>>> The “top” procedure will monitor the threads for 20 seconds (the
>>>> default is 10 seconds in the current thread’s thread group) and
>>>> refresh the status every second. In the above the primordial
>>>> thread was sleeping (and 3.8 seconds were left to sleep). If you
>>>> remove the (thread-sleep! 5) to get an infinite loop, the status
>>>> would be
>>>>
>>>> #<thread #1 primordial> RUNNING P0
>>>>
>>>> which indicates that the primordial thread is running (on processor
>>>> #0). If a thread is blocked reading a port you will see something
>>>> like
>>>>
>>>> #<thread #1 primordial> WAITING #<input-port #2 (stdin)>
>>>>
>>>> If on the other hand your program is totally unresponsive, it could
>>>> be a bug in Gambit’s runtime system. This is much harder to debug,
>>>> especially if it only happens after a long time. In that case you
>>>> could compile Gambit with --enable-debug and not
>>>> --enable-single-host and use gdb or lldb to attach to the process
>>>> and interrupt it. If you also configure Gambit with
>>>> --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history
>>>> you can “kill -3 <pid>” the program when it seems to be stuck and
>>>> you will get a history of the 1024 last Scheme procedure calls,
>>>> plus a C backtrace, on the gambit.log file. This should improve
>>>> your understanding of the issue…
>>>>
>>>> Marc
>>>>
>>>>
>>>>
>>>>> On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger
>>>>> <Joerg.Wittenberger at softeyes.net> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> on Linux I observe gambit entering a tight loop.
>>>>>
>>>>> Attached the result of "strace -r -p <pid> -o ot0.trace".
>>>>>
>>>>> To me this looks like pattern of roughly 15/80/15/118... repeating
>>>>> the heartbeat operation. In this state the program takes almost
>>>>> 100% CPU and does not respond to anything. Normally it runs on
>>>>> around 40-60 open file descriptors (most from UDP and TCP).
>>>>>
>>>>> The problem arises essentially for sure. Depending on the
>>>>> hardware sooner or later. (On 32bit armel after hours to few
>>>>> days, aarch64 takes sometimes almost week, AMD64 even longer.)
>>>>>
>>>>> Anybody having an idea how to track this down?
>>>>>
>>>>> Thanks so much
>>>>>
>>>>> Jörg
>>>>> <ot0.trace>_______________________________________________
>>>>> Gambit-list mailing list
>>>>> Gambit-list at iro.umontreal.ca
>>>>> https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list
>>>>
>>>>
>>>
>>> <0007-gambit-foreign.scm>
>>
>>
>
>
More information about the Gambit-list
mailing list