[gambit-list] Need help to track down tight loop

Jörg F. Wittenberger Joerg.Wittenberger at softeyes.net
Sun Aug 9 12:33:38 EDT 2020


Reconsidering I wonder: how to combine independent gambit libraries?

In oder to do Scheme-to-C calls there has to be a global mutex (or some
other way of synchronization) to protect against calls done in parallel.

Assuming the libraries all being independently developed, distributed
and standalone usable, all of them need to import a reference the same
mutex.

The only solution I see at this point: import from gambit core itself.

Do I miss the alternative or am hereby I asking for such a mutex?

Jörg

Am Sun, 9 Aug 2020 11:02:05 -0400
schrieb Marc Feeley <feeley at iro.umontreal.ca>:

> Good idea to add that to the documentation.  Will do.
> 
> Marc
> 
> 
> 
> > On Aug 7, 2020, at 2:49 PM, Jörg F. Wittenberger
> > <Joerg.Wittenberger at softeyes.net> wrote:
> > 
> > Thank you Marc,
> > 
> > this essentially confirms my hypothesis uttered in May, that gambit
> > threading might be equivalent to continuation capture and thus fall
> > under section 19.7 of the manual.
> > 
> > May I suggest to add a line there that "Scheme thread context
> > switching is implemented with continuations, so only one Scheme
> > thread at a time can do Scheme to C to Scheme calls" in order to
> > not trap too many newbies like me with the same rope.
> > 
> > Best
> > 
> > Jörg
> > 
> > Am Fri, 7 Aug 2020 13:31:19 -0400
> > schrieb Marc Feeley <feeley at iro.umontreal.ca>:
> >   
> >> The situation you describe is not supported by Gambit.  Any number
> >> of Scheme threads can call C functions, but only one Scheme thread
> >> at a time can do a Scheme to C call that calls back to Scheme.
> >> This is because the C stack is shared by all the Scheme to C
> >> calls… if the Scheme threads T1 and T2 call C (in the order T1
> >> then T2) and if T1 returns to Scheme while T2’s call is still in
> >> progress, then T2’s C stack frame will be removed when T1’s C
> >> stack frame is removed (because T2’s frame was added after T1’s
> >> frame).
> >> 
> >> The Gambit manual describes a related case to avoid:
> >> 
> >>   Gambit maintains the Scheme continuation separately from the C
> >> stack, thus allowing the Scheme continuation to be unwound
> >> independently from the C stack.  The C stack frame created for the
> >> C function @samp{f} is only removed from the C stack when control
> >> returns from @samp{f} or when control returns to a C function
> >> ``above'' @samp{f}.  Special care is required for programs which
> >> escape to Scheme (using first-class continuations) from a Scheme
> >> to C (to Scheme) call because the C stack frame will remain on the
> >> stack. The C stack may overflow if this happens in a loop with no
> >> intervening return to a C function.  To avoid this problem make
> >> sure the C stack gets cleaned up by executing a normal return from
> >> a Scheme to C call.
> >> 
> >> Gambit Scheme thread context switching is implemented with
> >> continuations, so only one Scheme thread at a time can do Scheme
> >> to C to Scheme calls.  One option is to disable context switching,
> >> but this is not easy to do reliably (both synchronous and
> >> preemptive context switching must be avoided).
> >> 
> >> Marc
> >> 
> >> 
> >>   
> >>> On Aug 7, 2020, at 11:59 AM, Jörg F. Wittenberger
> >>> <Joerg.Wittenberger at softeyes.net> wrote:
> >>> 
> >>> Hello Marc,
> >>> 
> >>> rtl;dr: I dare to bet that this is either a bug or a
> >>> to-be-documented shortcoming of Gambit Scheme.  No proof, however,
> >>> just a wager: Thread switches within a Scheme-to-C-to-Scheme
> >>> callback (no matter what caused them, blocking operations or
> >>> heartbeat) from one module which cause another
> >>> Scheme-to-C-to-Scheme call (in another thread and another
> >>> module/Scheme file) to return can/will confuse gambit runtime.
> >>> 
> >>> The Long Story:
> >>> 
> >>> The issue appears to be the than-expected regression of my
> >>> question from May:
> >>> https://mailman.iro.umontreal.ca/pipermail/gambit-list/2020-May/009457.html
> >>> when I wrote:    
> >>>> But I'm afraid it might be a red herring and just mitigate the
> >>>> effect for the hell to come back under more load.    
> >>> 
> >>> The lambdanative project disabled heartbeat interrupts (on
> >>> Android, i.e. on slower hardware) a while back "because it caused
> >>> issues". Which issues exactly is "lost in constitutional memory"
> >>> I learned when I asked.  Well lambdanative uses many external
> >>> libraries.
> >>> 
> >>> What I did after I asked a week ago:
> >>> 
> >>> a) I extended the code attached to the posting cited above.  Now,
> >>> whenever calling into Scheme the heartbeat interrupts are disabled
> >>> too.  New version attached.
> >>> 
> >>>   Towards the end of the file there is a c-safe-lambda macro
> >>>   definition I use instead of c-lambda whenever the C call might
> >>> call back to Scheme.   Additionally I use ##safe-lambda-post! from
> >>>   within C-to-Scheme callbacks to postpone thunks for execution
> >>> after the Scheme-to-C call returned.  All possibly blocking
> >>> operations, `thread-start!` etc. are deferred that way.
> >>> 
> >>> This was rolled out onto 6 machines and works since without the
> >>> issue coming up again.
> >>> 
> >>> b) I tried the old code on the slowest of those machine.  After a
> >>> short while the tight loop re-appeared.  Then I tried your
> >>> suggestions:
> >>> 
> >>> Am Sun, 2 Aug 2020 13:29:32 -0400
> >>> schrieb Marc Feeley <feeley at iro.umontreal.ca>:
> >>>   
> >>>> Hello Jorg.  The heartbeat interrupts are not necessarily
> >>>> problematic.  They will occur if your program is running, for
> >>>> example in a busy loop (in your program or the runtime library).
> >>>> Can you interrupt your program to get a REPL and see what it is
> >>>> doing?    
> >>> 
> >>> No.  Not when I locked up.  When I leave a repl running right from
> >>> the start it does no longer respond either.
> >>>   
> >>>> A convenient way to do this is to start your program with the
> >>>> -:daR runtime option.  When you hit ctrl-C a REPL will be
> >>>> started in the context of what the program was executing, so a
> >>>> “,b” command will give you a backtrace.    
> >>> 
> >>> OK. This one I did not try.  But i tried that version:
> >>>   
> >>>> Another way is to start your program with the -:d$ runtime option
> >>>> that starts a REPL server listening on port 44555.  Then you can
> >>>> start a REPL by connecting to that port (for example with the
> >>>> “nc” program) and use the “top” procedure to see what the
> >>>> threads are doing.  For example, in one shell do
> >>>> 
> >>>> % gsi -:d$ -e '(let loop () (thread-sleep! 5) (loop))'
> >>>> 
> >>>> and in another shell do
> >>>> 
> >>>> % $ nc localhost 44555
> >>>> Gambit v4.9.3-1201-g3b320533    
> >>> 
> >>> No.  nc will connect only as long as the issue did not come up.
> >>> Afterwards if will hang in the connection attempt.
> >>> 
> >>> Note that it might be really tough to find a simple test case to
> >>> reproduce the issue using heartbeats.  I my case there are two
> >>> libraries lwIP and onetierzero[1].  For each of them there is a C
> >>> file/module with gambit ffi bindings.  Both are driven from two
> >>> sources: each needs a timer for housekeeping tasks and when gambit
> >>> receives network traffic the corresponding gambit thread will call
> >>> them too.  So each network packet will cause roughly two
> >>> Scheme-to-C-to-Scheme calls.  Still it takes hours to enter the
> >>> issue via heartbeat.
> >>> 
> >>> However the posting from may might make things easier: when two
> >>> Scheme-to-C-to-Scheme calls are explicitly blocked (say on a
> >>> mutex) in each two modules and then the mutexs unlocked in the
> >>> same sequence as they where locked, the issue *should* arise.
> >>> (Speculating here: I did not yet try this one yet.)
> >>> 
> >>> Best Jörg
> >>> 
> >>> [1] https://github.com/0-8-15/onetierzero
> >>> 
> >>> ------------------ for reference only -----------
> >>>   
> >>>>> (top 20 (thread-thread-group #1))      
> >>>> *** THREAD LIST:
> >>>> #<thread #1 primordial>   SLEEPING 3.8s
> >>>> 
> >>>> The “top” procedure will monitor the threads for 20 seconds (the
> >>>> default is 10 seconds in the current thread’s thread group) and
> >>>> refresh the status every second.  In the above the primordial
> >>>> thread was sleeping (and 3.8 seconds were left to sleep).  If you
> >>>> remove the (thread-sleep! 5) to get an infinite loop, the status
> >>>> would be
> >>>> 
> >>>> #<thread #1 primordial>   RUNNING P0
> >>>> 
> >>>> which indicates that the primordial thread is running (on
> >>>> processor #0).  If a thread is blocked reading a port you will
> >>>> see something like
> >>>> 
> >>>> #<thread #1 primordial>   WAITING #<input-port #2 (stdin)>
> >>>> 
> >>>> If on the other hand your program is totally unresponsive, it
> >>>> could be a bug in Gambit’s runtime system.  This is much harder
> >>>> to debug, especially if it only happens after a long time.  In
> >>>> that case you could compile Gambit with --enable-debug and not
> >>>> --enable-single-host and use gdb or lldb to attach to the process
> >>>> and interrupt it.  If you also configure Gambit with
> >>>> --enable-debug-c-backtrace and --enable-debug-ctrl-flow-history
> >>>> you can “kill -3 <pid>” the program when it seems to be stuck and
> >>>> you will get a history of the 1024 last Scheme procedure calls,
> >>>> plus a C backtrace, on the gambit.log file. This should improve
> >>>> your understanding of the issue…
> >>>> 
> >>>> Marc
> >>>> 
> >>>> 
> >>>>   
> >>>>> On Aug 2, 2020, at 9:53 AM, Jörg F. Wittenberger
> >>>>> <Joerg.Wittenberger at softeyes.net> wrote:
> >>>>> 
> >>>>> Hello,
> >>>>> 
> >>>>> on Linux I observe gambit entering a tight loop.
> >>>>> 
> >>>>> Attached the result of "strace -r -p <pid> -o ot0.trace".
> >>>>> 
> >>>>> To me this looks like pattern of roughly 15/80/15/118...
> >>>>> repeating the heartbeat operation.  In this state the program
> >>>>> takes almost 100% CPU and does not respond to anything.
> >>>>> Normally it runs on around 40-60 open file descriptors (most
> >>>>> from UDP and TCP).
> >>>>> 
> >>>>> The problem arises essentially for sure.  Depending on the
> >>>>> hardware sooner or later.  (On 32bit armel after hours to few
> >>>>> days, aarch64 takes sometimes almost week, AMD64 even longer.)
> >>>>> 
> >>>>> Anybody having an idea how to track this down?
> >>>>> 
> >>>>> Thanks so much
> >>>>> 
> >>>>> Jörg
> >>>>> <ot0.trace>_______________________________________________
> >>>>> Gambit-list mailing list
> >>>>> Gambit-list at iro.umontreal.ca
> >>>>> https://mailman.iro.umontreal.ca/cgi-bin/mailman/listinfo/gambit-list      
> >>>> 
> >>>>   
> >>> 
> >>> <0007-gambit-foreign.scm>    
> >> 
> >>   
> > 
> >   
> 
> 





More information about the Gambit-list mailing list