[gambit-list] Need help to track down tight loop

Marc Feeley feeley at iro.umontreal.ca
Sun Aug 9 11:02:05 EDT 2020


Good idea to add that to the documentation.  Will do.

Marc



> On Aug 7, 2020, at 2:49 PM, Jörg F. Wittenberger <Joerg.Wittenberger at softeyes.net> wrote:
> 
> Thank you Marc,
> 
> this essentially confirms my hypothesis uttered in May, that gambit
> threading might be equivalent to continuation capture and thus fall
> under section 19.7 of the manual.
> 
> May I suggest to add a line there that "Scheme thread context switching
> is implemented with continuations, so only one Scheme thread at a time
> can do Scheme to C to Scheme calls" in order to not trap too many
> newbies like me with the same rope.
> 
> Best
> 
> Jörg
> 
> Am Fri, 7 Aug 2020 13:31:19 -0400
> schrieb Marc Feeley <feeley at iro.umontreal.ca>:
> 
>> The situation you describe is not supported by Gambit.  Any number of
>> Scheme threads can call C functions, but only one Scheme thread at a
>> time can do a Scheme to C call that calls back to Scheme.  This is
>> because the C stack is shared by all the Scheme to C calls… if the
>> Scheme threads T1 and T2 call C (in the order T1 then T2) and if T1
>> returns to Scheme while T2’s call is still in progress, then T2’s C
>> stack frame will be removed when T1’s C stack frame is removed
>> (because T2’s frame was added after T1’s frame).
>> 
>> The Gambit manual describes a related case to avoid:
>> 
>>   Gambit maintains the Scheme continuation separately from the C
>> stack, thus allowing the Scheme continuation to be unwound
>> independently from the C stack.  The C stack frame created for the C
>> function @samp{f} is only removed from the C stack when control
>> returns from @samp{f} or when control returns to a C function
>> ``above'' @samp{f}.  Special care is required for programs which
>> escape to Scheme (using first-class continuations) from a Scheme to C
>> (to Scheme) call because the C stack frame will remain on the stack.
>> The C stack may overflow if this happens in a loop with no
>> intervening return to a C function.  To avoid this problem make sure
>> the C stack gets cleaned up by executing a normal return from a
>> Scheme to C call.
>> 
>> Gambit Scheme thread context switching is implemented with
>> continuations, so only one Scheme thread at a time can do Scheme to C
>> to Scheme calls.  One option is to disable context switching, but
>> this is not easy to do reliably (both synchronous and preemptive
>> context switching must be avoided).
>> 
>> Marc
>> 
>> 
>> 
>>> On Aug 7, 2020, at 11:59 AM, Jörg F. Wittenberger
>>> <Joerg.Wittenberger at softeyes.net> wrote:
>>> 
>>> Hello Marc,
>>> 
>>> rtl;dr: I dare to bet that this is either a bug or a
>>> to-be-documented shortcoming of Gambit Scheme.  No proof, however,
>>> just a wager: Thread switches within a Scheme-to-C-to-Scheme
>>> callback (no matter what caused them, blocking operations or
>>> heartbeat) from one module which cause another
>>> Scheme-to-C-to-Scheme call (in another thread and another
>>> module/Scheme file) to return can/will confuse gambit runtime.
>>> 
>>> The Long Story:
>>> 
>>> The issue appears to be the than-expected regression of my question
>>> from May:
>>> https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html
>>> when I wrote:  
>>>> But I'm afraid it might be a red herring and just mitigate the
>>>> effect for the hell to come back under more load.  
>>> 
>>> The lambdanative project disabled heartbeat interrupts (on Android,
>>> i.e. on slower hardware) a while back "because it caused issues".
>>> Which issues exactly is "lost in constitutional memory" I learned
>>> when I asked.  Well lambdanative uses many external libraries.
>>> 
>>> What I did after I asked a week ago:
>>> 
>>> a) I extended the code attached to the posting cited above.  Now,
>>> whenever calling into Scheme the heartbeat interrupts are disabled
>>> too.  New version attached.
>>> 
>>>   Towards the end of the file there is a c-safe-lambda macro
>>>   definition I use instead of c-lambda whenever the C call might
>>> call back to Scheme.   Additionally I use ##safe-lambda-post! from
>>>   within C-to-Scheme callbacks to postpone thunks for execution
>>> after the Scheme-to-C call returned.  All possibly blocking
>>> operations, `thread-start!` etc. are deferred that way.
>>> 
>>> This was rolled out onto 6 machines and works since without the
>>> issue coming up again.
>>> 
>>> b) I tried the old code on the slowest of those machine.  After a
>>> short while the tight loop re-appeared.  Then I tried your
>>> suggestions:
>>> 
>>> Am Sun, 2 Aug 2020 13:29:32 -0400
>>> schrieb Marc Feeley <feeley at iro.umontreal.ca>:
>>> 
>>>> Hello Jorg.  The heartbeat interrupts are not necessarily
>>>> problematic.  They will occur if your program is running, for
>>>> example in a busy loop (in your program or the runtime library).
>>>> Can you interrupt your program to get a REPL and see what it is
>>>> doing?  
>>> 
>>> No.  Not when I locked up.  When I leave a repl running right from
>>> the start it does no longer respond either.
>>> 
>>>> A convenient way to do this is to start your program with the -:daR
>>>> runtime option.  When you hit ctrl-C a REPL will be started in the
>>>> context of what the program was executing, so a “,b” command will
>>>> give you a backtrace.  
>>> 
>>> OK. This one I did not try.  But i tried that version:
>>> 
>>>> Another way is to start your program with the -:d$ runtime option
>>>> that starts a REPL server listening on port 44555.  Then you can
>>>> start a REPL by connecting to that port (for example with the “nc”
>>>> program) and use the “top” procedure to see what the threads are
>>>> doing.  For example, in one shell do
>>>> 
>>>> % gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
>>>> 
>>>> and in another shell do
>>>> 
>>>> % $ nc localhost 44555
>>>> Gambit v4.9.3-1201-g3b320533  
>>> 
>>> No.  nc will connect only as long as the issue did not come up.
>>> Afterwards if will hang in the connection attempt.
>>> 
>>> Note that it might be really tough to find a simple test case to
>>> reproduce the issue using heartbeats.  I my case there are two
>>> libraries lwIP and onetierzero[1].  For each of them there is a C
>>> file/module with gambit ffi bindings.  Both are driven from two
>>> sources: each needs a timer for housekeeping tasks and when gambit
>>> receives network traffic the corresponding gambit thread will call
>>> them too.  So each network packet will cause roughly two
>>> Scheme-to-C-to-Scheme calls.  Still it takes hours to enter the
>>> issue via heartbeat.
>>> 
>>> However the posting from may might make things easier: when two
>>> Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex)
>>> in each two modules and then the mutexs unlocked in the same
>>> sequence as they where locked, the issue *should* arise.
>>> (Speculating here: I did not yet try this one yet.)
>>> 
>>> Best Jörg
>>> 
>>> [1] https://github.com/0-8-15/onetierzero
>>> 
>>> ------------------ for reference only -----------
>>> 
>>>>> (top 20 (thread-thread-group #1))    
>>>> *** THREAD LIST:
>>>> #<thread #1 primordial>   SLEEPING 3.8s
>>>> 
>>>> The “top” procedure will monitor the threads for 20 seconds (the
>>>> default is 10 seconds in the current thread’s thread group) and
>>>> refresh the status every second.  In the above the primordial
>>>> thread was sleeping (and 3.8 seconds were left to sleep).  If you
>>>> remove the (thread-sleep! 5) to get an infinite loop, the status
>>>> would be
>>>> 
>>>> #<thread #1 primordial>   RUNNING P0
>>>> 
>>>> which indicates that the primordial thread is running (on processor
>>>> #0).  If a thread is blocked reading a port you will see something
>>>> like
>>>> 
>>>> #<thread #1 primordial>   WAITING #<input-port #2 (stdin)>
>>>> 
>>>> If on the other hand your program is totally unresponsive, it could
>>>> be a bug in Gambit’s runtime system.  This is much harder to debug,
>>>> especially if it only happens after a long time.  In that case you
>>>> could compile Gambit with --enable-debug and not
>>>> --enable-single-host and use gdb or lldb to attach to the process
>>>> and interrupt it.  If you also configure Gambit with
>>>> --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history
>>>> you can “kill -3 <pid>” the program when it seems to be stuck and
>>>> you will get a history of the 1024 last Scheme procedure calls,
>>>> plus a C backtrace, on the gambit.log file. This should improve
>>>> your understanding of the issue…
>>>> 
>>>> Marc
>>>> 
>>>> 
>>>> 
>>>>> On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger
>>>>> <Joerg.Wittenberger at softeyes.net> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> on Linux I observe gambit entering a tight loop.
>>>>> 
>>>>> Attached the result of "strace -r -p <pid> -o ot0.trace".
>>>>> 
>>>>> To me this looks like pattern of roughly 15/80/15/118... repeating
>>>>> the heartbeat operation.  In this state the program takes almost
>>>>> 100% CPU and does not respond to anything.  Normally it runs on
>>>>> around 40-60 open file descriptors (most from UDP and TCP).
>>>>> 
>>>>> The problem arises essentially for sure.  Depending on the
>>>>> hardware sooner or later.  (On 32bit armel after hours to few
>>>>> days, aarch64 takes sometimes almost week, AMD64 even longer.)
>>>>> 
>>>>> Anybody having an idea how to track this down?
>>>>> 
>>>>> Thanks so much
>>>>> 
>>>>> Jörg
>>>>> <ot0.trace>_______________________________________________
>>>>> Gambit-list mailing list
>>>>> Gambit-list at iro.umontreal.ca
>>>>> https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list    
>>>> 
>>>> 
>>> 
>>> <0007-gambit-foreign.scm>  
>> 
>> 
> 
> 





More information about the Gambit-list mailing list