[gambit-list] Need help to track down tight loop

Jörg F. Wittenberger Joerg.Wittenberger at softeyes.net
Fri Aug 7 11:59:39 EDT 2020


Hello Marc,

rtl;dr: I dare to bet that this is either a bug or a to-be-documented
shortcoming of Gambit Scheme.  No proof, however, just a wager: Thread
switches within a Scheme-to-C-to-Scheme callback (no matter what caused
them, blocking operations or heartbeat) from one module which cause
another Scheme-to-C-to-Scheme call (in another thread and another
module/Scheme file) to return can/will confuse gambit runtime.

The Long Story:

The issue appears to be the than-expected regression of my question
from May:
https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html
when I wrote:
> But I'm afraid it might be a red herring and just mitigate the effect
> for the hell to come back under more load.

The lambdanative project disabled heartbeat interrupts (on Android,
i.e. on slower hardware) a while back "because it caused issues".
Which issues exactly is "lost in constitutional memory" I learned when I
asked.  Well lambdanative uses many external libraries.

What I did after I asked a week ago:

a) I extended the code attached to the posting cited above.  Now,
whenever calling into Scheme the heartbeat interrupts are disabled
too.  New version attached.

    Towards the end of the file there is a c-safe-lambda macro
    definition I use instead of c-lambda whenever the C call might call
    back to Scheme.   Additionally I use ##safe-lambda-post! from
    within C-to-Scheme callbacks to postpone thunks for execution after
    the Scheme-to-C call returned.  All possibly blocking operations,
    `thread-start!` etc. are deferred that way.

This was rolled out onto 6 machines and works since without the issue
coming up again.

b) I tried the old code on the slowest of those machine.  After a short
while the tight loop re-appeared.  Then I tried your suggestions:

Am Sun, 2 Aug 2020 13:29:32 -0400
schrieb Marc Feeley <feeley at iro.umontreal.ca>:

> Hello Jorg.  The heartbeat interrupts are not necessarily
> problematic.  They will occur if your program is running, for example
> in a busy loop (in your program or the runtime library).  Can you
> interrupt your program to get a REPL and see what it is doing?

No.  Not when I locked up.  When I leave a repl running right from the
start it does no longer respond either.

> A convenient way to do this is to start your program with the -:daR
> runtime option.  When you hit ctrl-C a REPL will be started in the
> context of what the program was executing, so a “,b” command will
> give you a backtrace.

OK. This one I did not try.  But i tried that version:

> Another way is to start your program with the -:d$ runtime option
> that starts a REPL server listening on port 44555.  Then you can
> start a REPL by connecting to that port (for example with the “nc”
> program) and use the “top” procedure to see what the threads are
> doing.  For example, in one shell do
> 
> % gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
> 
> and in another shell do
> 
> % $ nc localhost 44555
> Gambit v4.9.3-1201-g3b320533

No.  nc will connect only as long as the issue did not come up.
Afterwards if will hang in the connection attempt.

Note that it might be really tough to find a simple test case to
reproduce the issue using heartbeats.  I my case there are two libraries
lwIP and onetierzero[1].  For each of them there is a C file/module with
gambit ffi bindings.  Both are driven from two sources: each needs a
timer for housekeeping tasks and when gambit receives network traffic
the corresponding gambit thread will call them too.  So each network
packet will cause roughly two Scheme-to-C-to-Scheme calls.  Still it
takes hours to enter the issue via heartbeat.

However the posting from may might make things easier: when two
Scheme-to-C-to-Scheme calls are explicitly blocked (say on a mutex) in
each two modules and then the mutexs unlocked in the same sequence as
they where locked, the issue *should* arise.  (Speculating here: I did
not yet try this one yet.)

Best Jörg

[1] https://github.com/0-8-15/onetierzero

------------------ for reference only -----------

> > (top 20 (thread-thread-group #1))  
> *** THREAD LIST:
> #<thread #1 primordial>   SLEEPING 3.8s
> 
> The “top” procedure will monitor the threads for 20 seconds (the
> default is 10 seconds in the current thread’s thread group) and
> refresh the status every second.  In the above the primordial thread
> was sleeping (and 3.8 seconds were left to sleep).  If you remove the
> (thread-sleep! 5) to get an infinite loop, the status would be
> 
> #<thread #1 primordial>   RUNNING P0
> 
> which indicates that the primordial thread is running (on processor
> #0).  If a thread is blocked reading a port you will see something
> like
> 
> #<thread #1 primordial>   WAITING #<input-port #2 (stdin)>
> 
> If on the other hand your program is totally unresponsive, it could
> be a bug in Gambit’s runtime system.  This is much harder to debug,
> especially if it only happens after a long time.  In that case you
> could compile Gambit with --enable-debug and not --enable-single-host
> and use gdb or lldb to attach to the process and interrupt it.  If
> you also configure Gambit with --enable-debug-c-backtrace and
> --enable-debug-ctrl-flow-history you can “kill -3 <pid>” the program
> when it seems to be stuck and you will get a history of the 1024 last
> Scheme procedure calls, plus a C backtrace, on the gambit.log file.
> This should improve your understanding of the issue…
> 
> Marc
> 
> 
> 
> > On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger
> > <Joerg.Wittenberger at softeyes.net> wrote:
> > 
> > Hello,
> > 
> > on Linux I observe gambit entering a tight loop.
> > 
> > Attached the result of "strace -r -p <pid> -o ot0.trace".
> > 
> > To me this looks like pattern of roughly 15/80/15/118... repeating
> > the heartbeat operation.  In this state the program takes almost
> > 100% CPU and does not respond to anything.  Normally it runs on
> > around 40-60 open file descriptors (most from UDP and TCP).
> > 
> > The problem arises essentially for sure.  Depending on the hardware
> > sooner or later.  (On 32bit armel after hours to few days, aarch64
> > takes sometimes almost week, AMD64 even longer.)
> > 
> > Anybody having an idea how to track this down?
> > 
> > Thanks so much
> > 
> > Jörg
> > <ot0.trace>_______________________________________________
> > Gambit-list mailing list
> > Gambit-list at iro.umontreal.ca
> > https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list  
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0007-gambit-foreign.scm
Type: text/x-scheme
Size: 5285 bytes
Desc: not available
URL: <http://mailman.iro.umontreal.ca/pipermail/gambit-list/attachments/20200807/eba68dc5/attachment.bin>


More information about the Gambit-list mailing list