[gambit-list] Need help to track down tight loop

Marc Feeley feeley at iro.umontreal.ca
Fri Aug 7 13:31:19 EDT 2020


The situation you describe is not supported by Gambit.  Any number of Scheme threads can call C functions, but only one Scheme thread at a time can do a Scheme to C call that calls back to Scheme.  This is because the C stack is shared by all the Scheme to C calls… if the Scheme threads T1 and T2 call C (in the order T1 then T2) and if T1 returns to Scheme while T2’s call is still in progress, then T2’s C stack frame will be removed when T1’s C stack frame is removed (because T2’s frame was added after T1’s frame).

The Gambit manual describes a related case to avoid:

   Gambit maintains the Scheme continuation separately from the C stack,
   thus allowing the Scheme continuation to be unwound independently from
   the C stack.  The C stack frame created for the C function @samp{f} is
   only removed from the C stack when control returns from @samp{f} or when
   control returns to a C function ``above'' @samp{f}.  Special care is
   required for programs which escape to Scheme (using first-class
   continuations) from a Scheme to C (to Scheme) call because the C stack
   frame will remain on the stack.  The C stack may overflow if this
   happens in a loop with no intervening return to a C function.  To avoid
   this problem make sure the C stack gets cleaned up by executing a normal
   return from a Scheme to C call.

Gambit Scheme thread context switching is implemented with continuations, so only one Scheme thread at a time can do Scheme to C to Scheme calls.  One option is to disable context switching, but this is not easy to do reliably (both synchronous and preemptive context switching must be avoided).

Marc



> On Aug 7, 2020, at 11:59 AM, Jörg F. Wittenberger <Joerg.Wittenberger at softeyes.net> wrote:
> 
> Hello Marc,
> 
> rtl;dr: I dare to bet that this is either a bug or a to-be-documented
> shortcoming of Gambit Scheme.  No proof, however, just a wager: Thread
> switches within a Scheme-to-C-to-Scheme callback (no matter what caused
> them, blocking operations or heartbeat) from one module which cause
> another Scheme-to-C-to-Scheme call (in another thread and another
> module/Scheme file) to return can/will confuse gambit runtime.
> 
> The Long Story:
> 
> The issue appears to be the than-expected regression of my question
> from May:
> https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html
> when I wrote:
>> But I'm afraid it might be a red herring and just mitigate the effect
>> for the hell to come back under more load.
> 
> The lambdanative project disabled heartbeat interrupts (on Android,
> i.e. on slower hardware) a while back "because it caused issues".
> Which issues exactly is "lost in constitutional memory" I learned when I
> asked.  Well lambdanative uses many external libraries.
> 
> What I did after I asked a week ago:
> 
> a) I extended the code attached to the posting cited above.  Now,
> whenever calling into Scheme the heartbeat interrupts are disabled
> too.  New version attached.
> 
>    Towards the end of the file there is a c-safe-lambda macro
>    definition I use instead of c-lambda whenever the C call might call
>    back to Scheme.   Additionally I use ##safe-lambda-post! from
>    within C-to-Scheme callbacks to postpone thunks for execution after
>    the Scheme-to-C call returned.  All possibly blocking operations,
>    `thread-start!` etc. are deferred that way.
> 
> This was rolled out onto 6 machines and works since without the issue
> coming up again.
> 
> b) I tried the old code on the slowest of those machine.  After a short
> while the tight loop re-appeared.  Then I tried your suggestions:
> 
> Am Sun, 2 Aug 2020 13:29:32 -0400
> schrieb Marc Feeley <feeley at iro.umontreal.ca>:
> 
>> Hello Jorg.  The heartbeat interrupts are not necessarily
>> problematic.  They will occur if your program is running, for example
>> in a busy loop (in your program or the runtime library).  Can you
>> interrupt your program to get a REPL and see what it is doing?
> 
> No.  Not when I locked up.  When I leave a repl running right from the
> start it does no longer respond either.
> 
>> A convenient way to do this is to start your program with the -:daR
>> runtime option.  When you hit ctrl-C a REPL will be started in the
>> context of what the program was executing, so a “,b” command will
>> give you a backtrace.
> 
> OK. This one I did not try.  But i tried that version:
> 
>> Another way is to start your program with the -:d$ runtime option
>> that starts a REPL server listening on port 44555.  Then you can
>> start a REPL by connecting to that port (for example with the “nc”
>> program) and use the “top” procedure to see what the threads are
>> doing.  For example, in one shell do
>> 
>> % gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
>> 
>> and in another shell do
>> 
>> % $ nc localhost 44555
>> Gambit v4.9.3-1201-g3b320533
> 
> No.  nc will connect only as long as the issue did not come up.
> Afterwards if will hang in the connection attempt.
> 
> Note that it might be really tough to find a simple test case to
> reproduce the issue using heartbeats.  I my case there are two libraries
> lwIP and onetierzero[1].  For each of them there is a C file/module with
> gambit ffi bindings.  Both are driven from two sources: each needs a
> timer for housekeeping tasks and when gambit receives network traffic
> the corresponding gambit thread will call them too.  So each network
> packet will cause roughly two Scheme-to-C-to-Scheme calls.  Still it
> takes hours to enter the issue via heartbeat.
> 
> However the posting from may might make things easier: when two
> Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex) in
> each two modules and then the mutexs unlocked in the same sequence as
> they where locked, the issue *should* arise.  (Speculating here: I did
> not yet try this one yet.)
> 
> Best Jörg
> 
> [1] https://github.com/0-8-15/onetierzero
> 
> ------------------ for reference only -----------
> 
>>> (top 20 (thread-thread-group #1))  
>> *** THREAD LIST:
>> #<thread #1 primordial>   SLEEPING 3.8s
>> 
>> The “top” procedure will monitor the threads for 20 seconds (the
>> default is 10 seconds in the current thread’s thread group) and
>> refresh the status every second.  In the above the primordial thread
>> was sleeping (and 3.8 seconds were left to sleep).  If you remove the
>> (thread-sleep! 5) to get an infinite loop, the status would be
>> 
>> #<thread #1 primordial>   RUNNING P0
>> 
>> which indicates that the primordial thread is running (on processor
>> #0).  If a thread is blocked reading a port you will see something
>> like
>> 
>> #<thread #1 primordial>   WAITING #<input-port #2 (stdin)>
>> 
>> If on the other hand your program is totally unresponsive, it could
>> be a bug in Gambit’s runtime system.  This is much harder to debug,
>> especially if it only happens after a long time.  In that case you
>> could compile Gambit with --enable-debug and not --enable-single-host
>> and use gdb or lldb to attach to the process and interrupt it.  If
>> you also configure Gambit with --enable-debug-c-backtrace and
>> --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program
>> when it seems to be stuck and you will get a history of the 1024 last
>> Scheme procedure calls, plus a C backtrace, on the gambit.log file.
>> This should improve your understanding of the issue…
>> 
>> Marc
>> 
>> 
>> 
>>> On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger
>>> <Joerg.Wittenberger at softeyes.net> wrote:
>>> 
>>> Hello,
>>> 
>>> on Linux I observe gambit entering a tight loop.
>>> 
>>> Attached the result of "strace -r -p <pid> -o ot0.trace".
>>> 
>>> To me this looks like pattern of roughly 15/80/15/118... repeating
>>> the heartbeat operation.  In this state the program takes almost
>>> 100% CPU and does not respond to anything.  Normally it runs on
>>> around 40-60 open file descriptors (most from UDP and TCP).
>>> 
>>> The problem arises essentially for sure.  Depending on the hardware
>>> sooner or later.  (On 32bit armel after hours to few days, aarch64
>>> takes sometimes almost week, AMD64 even longer.)
>>> 
>>> Anybody having an idea how to track this down?
>>> 
>>> Thanks so much
>>> 
>>> Jörg
>>> <ot0.trace>_______________________________________________
>>> Gambit-list mailing list
>>> Gambit-list at iro.umontreal.ca
>>> https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list  
>> 
>> 
> 
> <0007-gambit-foreign.scm>





More information about the Gambit-list mailing list