Progress with the memory leak hunting, the root reference is a 'thread-call-result mutex referencing a continuation starting w ##thread-call . What this means?
Dear Marc, Soo I've hunted this memory leak down a bit now and I'm starting to feel I got somewhere: So basically I noted that my heap was filling up with a vector object that I use for IO. Now analyzed what references it using ##resolve-referencing-objects . This memory leak seems to happen for something-like-in-the-range every 10:th incoming TCP connection. It's handled by Sack, and then as we see left on the heap are some structures regarding its I/O. This is a bit funny as I have the impression that Sack can run without producing any memory leaks too. Here's the tracing of the root for such a vector: A stack frame is referencing it, which is of a procedure that I provide, whose invocation ended long ago. That stack frame in turn is referenced by the procedure that created the vector. And that stack frame in turn is referenced by ##read-u8 (!) And that stack frame in turn is referenced by ##byte-rbuf-fill (!) And that stack frame in turn is referenced by ##wait-for-io! (!) And that stack frame in turn is referenced by ##thread-interrupt! (!) And that stack frame in turn is referenced by ##thread-call (!) And that stack frame in turn is referenced by a continuation object. The continuation object's denv is #(#((#<procedure #N current-directory> . "/home/USER/") () ()) #(0) 0 0 (#<procedure #N current-exception-handler> . #<procedure #N primordial-exception-handler>) (#<procedure #N current-input-port> . #<input-output-port #N dummy>) (#<procedure #N current-output-port> . #<input-output-port #N dummy>) (#f . #f)) That continuation object in turn is referenced by * A mutex with the mutex-name 'thread-call-result , mutex-state #<thread #1 primordial> , mutex-specific set to the continuation itself. * This vector #(3 2 #<continuation #N> #!unbound) - I think that's a product of the evaluator * (#<continuation #N>) * (#<continuation #N> . #t) The mutex is in turn referenced by another three mutexes, each being a 'thread-call-result mutex with another continuation as its specific. If there's anything more that's relevant for me to check here, feel free to let me know. Do you see any plausible reason for the leak? Best regards, Mikael
Afficher les réponses par date
If, perchance, any of the routines you are discovering are not documented, perhaps you could document each of them a bit and send a pull request to Marc. Brad On Jun 20, 2013, at 10:38 AM, Mikael wrote:
Dear Marc,
Soo I've hunted this memory leak down a bit now and I'm starting to feel I got somewhere:
So basically I noted that my heap was filling up with a vector object that I use for IO.
Now analyzed what references it using ##resolve-referencing-objects .
This memory leak seems to happen for something-like-in-the-range every 10:th incoming TCP connection.
It's handled by Sack, and then as we see left on the heap are some structures regarding its I/O.
This is a bit funny as I have the impression that Sack can run without producing any memory leaks too.
Here's the tracing of the root for such a vector:
A stack frame is referencing it, which is of a procedure that I provide, whose invocation ended long ago.
That stack frame in turn is referenced by the procedure that created the vector.
And that stack frame in turn is referenced by ##read-u8 (!)
And that stack frame in turn is referenced by ##byte-rbuf-fill (!)
And that stack frame in turn is referenced by ##wait-for-io! (!)
And that stack frame in turn is referenced by ##thread-interrupt! (!)
And that stack frame in turn is referenced by ##thread-call (!)
And that stack frame in turn is referenced by a continuation object.
The continuation object's denv is
#(#((#<procedure #N current-directory> . "/home/USER/") () ()) #(0) 0 0 (#<procedure #N current-exception-handler> . #<procedure #N primordial-exception-handler>) (#<procedure #N current-input-port> . #<input-output-port #N dummy>) (#<procedure #N current-output-port> . #<input-output-port #N dummy>) (#f . #f))
That continuation object in turn is referenced by
* A mutex with the mutex-name 'thread-call-result ,
mutex-state #<thread #1 primordial> ,
mutex-specific set to the continuation itself.
* This vector #(3 2 #<continuation #N> #!unbound) - I think that's a product of the evaluator
* (#<continuation #N>)
* (#<continuation #N> . #t)
The mutex is in turn referenced by another three mutexes, each being a 'thread-call-result mutex with another continuation as its specific.
If there's anything more that's relevant for me to check here, feel free to let me know.
Do you see any plausible reason for the leak?
Best regards, Mikael
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
That's right, not documented. You mean documented as in in the Gambit manual, or documented as in a comment provided at their location in Gambit's sources? 2013/6/20 Bradley Lucier <lucier@math.purdue.edu>
If, perchance, any of the routines you are discovering are not documented, perhaps you could document each of them a bit and send a pull request to Marc.
Brad
On Jun 20, 2013, at 10:38 AM, Mikael wrote:
Dear Marc,
Soo I've hunted this memory leak down a bit now and I'm starting to feel I got somewhere:
So basically I noted that my heap was filling up with a vector object that I use for IO.
Now analyzed what references it using ##resolve-referencing-objects .
This memory leak seems to happen for something-like-in-the-range every 10:th incoming TCP connection.
It's handled by Sack, and then as we see left on the heap are some structures regarding its I/O.
This is a bit funny as I have the impression that Sack can run without producing any memory leaks too.
Here's the tracing of the root for such a vector:
A stack frame is referencing it, which is of a procedure that I provide, whose invocation ended long ago.
That stack frame in turn is referenced by the procedure that created the vector.
And that stack frame in turn is referenced by ##read-u8 (!)
And that stack frame in turn is referenced by ##byte-rbuf-fill (!)
And that stack frame in turn is referenced by ##wait-for-io! (!)
And that stack frame in turn is referenced by ##thread-interrupt! (!)
And that stack frame in turn is referenced by ##thread-call (!)
And that stack frame in turn is referenced by a continuation object.
The continuation object's denv is
#(#((#<procedure #N current-directory> . "/home/USER/") () ()) #(0) 0 0 (#<procedure #N current-exception-handler> . #<procedure #N primordial-exception-handler>) (#<procedure #N current-input-port> . #<input-output-port #N dummy>) (#<procedure #N current-output-port> . #<input-output-port #N dummy>) (#f . #f))
That continuation object in turn is referenced by
* A mutex with the mutex-name 'thread-call-result ,
mutex-state #<thread #1 primordial> ,
mutex-specific set to the continuation itself.
* This vector #(3 2 #<continuation #N> #!unbound) - I think that's a product of the evaluator
* (#<continuation #N>)
* (#<continuation #N> . #t)
The mutex is in turn referenced by another three mutexes, each being a 'thread-call-result mutex with another continuation as its specific.
If there's anything more that's relevant for me to check here, feel free to let me know.
Do you see any plausible reason for the leak?
Best regards, Mikael
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Well, I meant a few lines in the sources, to document the internals. Brad On Jun 20, 2013, at 11:04 AM, Mikael wrote:
That's right, not documented.
You mean documented as in in the Gambit manual, or documented as in a comment provided at their location in Gambit's sources?
2013/6/20 Bradley Lucier <lucier@math.purdue.edu> If, perchance, any of the routines you are discovering are not documented, perhaps you could document each of them a bit and send a pull request to Marc.
Brad
On Jun 20, 2013, at 10:38 AM, Mikael wrote:
Dear Marc,
Soo I've hunted this memory leak down a bit now and I'm starting to feel I got somewhere:
So basically I noted that my heap was filling up with a vector object that I use for IO.
Now analyzed what references it using ##resolve-referencing-objects .
This memory leak seems to happen for something-like-in-the-range every 10:th incoming TCP connection.
It's handled by Sack, and then as we see left on the heap are some structures regarding its I/O.
This is a bit funny as I have the impression that Sack can run without producing any memory leaks too.
Here's the tracing of the root for such a vector:
A stack frame is referencing it, which is of a procedure that I provide, whose invocation ended long ago.
That stack frame in turn is referenced by the procedure that created the vector.
And that stack frame in turn is referenced by ##read-u8 (!)
And that stack frame in turn is referenced by ##byte-rbuf-fill (!)
And that stack frame in turn is referenced by ##wait-for-io! (!)
And that stack frame in turn is referenced by ##thread-interrupt! (!)
And that stack frame in turn is referenced by ##thread-call (!)
And that stack frame in turn is referenced by a continuation object.
The continuation object's denv is
#(#((#<procedure #N current-directory> . "/home/USER/") () ()) #(0) 0 0 (#<procedure #N current-exception-handler> . #<procedure #N primordial-exception-handler>) (#<procedure #N current-input-port> . #<input-output-port #N dummy>) (#<procedure #N current-output-port> . #<input-output-port #N dummy>) (#f . #f))
That continuation object in turn is referenced by
* A mutex with the mutex-name 'thread-call-result ,
mutex-state #<thread #1 primordial> ,
mutex-specific set to the continuation itself.
* This vector #(3 2 #<continuation #N> #!unbound) - I think that's a product of the evaluator
* (#<continuation #N>)
* (#<continuation #N> . #t)
The mutex is in turn referenced by another three mutexes, each being a 'thread-call-result mutex with another continuation as its specific.
If there's anything more that's relevant for me to check here, feel free to let me know.
Do you see any plausible reason for the leak?
Best regards, Mikael
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
There is also the 6.4 Undocumented extensions section in the Gambit manual that is there for this purpose, correct me if I'm wrong Marc. On Thu, Jun 20, 2013 at 11:10 AM, Bradley Lucier <lucier@math.purdue.edu>wrote:
Well, I meant a few lines in the sources, to document the internals.
Brad
On Jun 20, 2013, at 11:04 AM, Mikael wrote:
That's right, not documented.
You mean documented as in in the Gambit manual, or documented as in a comment provided at their location in Gambit's sources?
2013/6/20 Bradley Lucier <lucier@math.purdue.edu>
If, perchance, any of the routines you are discovering are not documented, perhaps you could document each of them a bit and send a pull request to Marc.
Brad
On Jun 20, 2013, at 10:38 AM, Mikael wrote:
Dear Marc,
Soo I've hunted this memory leak down a bit now and I'm starting to feel I got somewhere:
So basically I noted that my heap was filling up with a vector object that I use for IO.
Now analyzed what references it using ##resolve-referencing-objects .
This memory leak seems to happen for something-like-in-the-range every 10:th incoming TCP connection.
It's handled by Sack, and then as we see left on the heap are some structures regarding its I/O.
This is a bit funny as I have the impression that Sack can run without producing any memory leaks too.
Here's the tracing of the root for such a vector:
A stack frame is referencing it, which is of a procedure that I provide, whose invocation ended long ago.
That stack frame in turn is referenced by the procedure that created the vector.
And that stack frame in turn is referenced by ##read-u8 (!)
And that stack frame in turn is referenced by ##byte-rbuf-fill (!)
And that stack frame in turn is referenced by ##wait-for-io! (!)
And that stack frame in turn is referenced by ##thread-interrupt! (!)
And that stack frame in turn is referenced by ##thread-call (!)
And that stack frame in turn is referenced by a continuation object.
The continuation object's denv is
#(#((#<procedure #N current-directory> . "/home/USER/") () ()) #(0) 0 0 (#<procedure #N current-exception-handler> . #<procedure #N primordial-exception-handler>) (#<procedure #N current-input-port> . #<input-output-port #N dummy>) (#<procedure #N current-output-port> . #<input-output-port #N dummy>) (#f . #f))
That continuation object in turn is referenced by
* A mutex with the mutex-name 'thread-call-result ,
mutex-state #<thread #1 primordial> ,
mutex-specific set to the continuation itself.
* This vector #(3 2 #<continuation #N> #!unbound) - I think that's a product of the evaluator
* (#<continuation #N>)
* (#<continuation #N> . #t)
The mutex is in turn referenced by another three mutexes, each being a 'thread-call-result mutex with another continuation as its specific.
If there's anything more that's relevant for me to check here, feel free to let me know.
Do you see any plausible reason for the leak?
Best regards, Mikael
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Jun 20, 2013, at 10:09 AM, Guillaume Cartier <gcartier@jazzscheme.org> wrote:
There is also the 6.4 Undocumented extensions section in the Gambit manual that is there for this purpose, correct me if I'm wrong Marc.
In principle that section if for documenting the existence of procedures that don't have documentation. I see that the first few are documented and should be moved elsewhere! The documentation should be cleaned up. Are there people out there with some free time and a generous nature? Marc
If there was a way to scrape documentation from source comments, I would love to send pull requests to make that happen. Any work I did would be in small batches that would arrive at random intervals. -Jason On Thu, Jun 20, 2013 at 1:29 PM, Marc Feeley <feeley@iro.umontreal.ca>wrote:
On Jun 20, 2013, at 10:09 AM, Guillaume Cartier <gcartier@jazzscheme.org> wrote:
There is also the 6.4 Undocumented extensions section in the Gambit manual that is there for this purpose, correct me if I'm wrong Marc.
In principle that section if for documenting the existence of procedures that don't have documentation. I see that the first few are documented and should be moved elsewhere! The documentation should be cleaned up. Are there people out there with some free time and a generous nature?
Marc
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Oh, part of the reason why I'm interested in it is because I'd love to see documentation available in the REPL. I know about the help browser = lynx thing, but I think it would be cleaner if documentation was associated with the functions in the REPL. -Jason On Thu, Jun 20, 2013 at 1:53 PM, Jason Felice <jason.m.felice@gmail.com>wrote:
If there was a way to scrape documentation from source comments, I would love to send pull requests to make that happen. Any work I did would be in small batches that would arrive at random intervals.
-Jason
On Thu, Jun 20, 2013 at 1:29 PM, Marc Feeley <feeley@iro.umontreal.ca>wrote:
On Jun 20, 2013, at 10:09 AM, Guillaume Cartier <gcartier@jazzscheme.org> wrote:
There is also the 6.4 Undocumented extensions section in the Gambit manual that is there for this purpose, correct me if I'm wrong Marc.
In principle that section if for documenting the existence of procedures that don't have documentation. I see that the first few are documented and should be moved elsewhere! The documentation should be cleaned up. Are there people out there with some free time and a generous nature?
Marc
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Jun 20, 2013, at 10:54 AM, Jason Felice <jason.m.felice@gmail.com> wrote:
Oh, part of the reason why I'm interested in it is because I'd love to see documentation available in the REPL. I know about the help browser = lynx thing, but I think it would be cleaner if documentation was associated with the functions in the REPL.
-Jason
Unfortunately, that would bloat the system if this feature is on by default. So I'm not against it, but it should be one of the configure options, and be off by default. Marc
On Jun 20, 2013, at 10:53 AM, Jason Felice <jason.m.felice@gmail.com> wrote:
If there was a way to scrape documentation from source comments, I would love to send pull requests to make that happen. Any work I did would be in small batches that would arrive at random intervals.
-Jason
Hey that's a great idea. All we have to do is agree on a source documentation format… ;-) Marc
That would be generally useful! 2013/6/20 Marc Feeley <feeley@iro.umontreal.ca>
On Jun 20, 2013, at 10:53 AM, Jason Felice <jason.m.felice@gmail.com> wrote:
If there was a way to scrape documentation from source comments, I would love to send pull requests to make that happen. Any work I did would be in small batches that would arrive at random intervals.
-Jason
Hey that's a great idea. All we have to do is agree on a source documentation format… ;-)
Marc
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Just for if it is of any value in understanding the problem, find below the variables bound to all of those stack frames and their liveness. Will be very happy to understand the nature of this issue further. To start with, it is rare. Dumping was done by (define (frame-o frame) (map-interval (lambda (i) (list (##frame-ref frame i) (##frame-slot-live? frame i))) 0 (- (##frame-fs frame) 1))) For reference, I got that "*** FATAL ERROR -- GC inconsistency detected" error I mentioned in an earlier email, when I resolved the references to #<frame one deeper than f> below. Find console output for that attached here. 2013/6/20 Mikael <mikael.rcv@gmail.com>
A stack frame is referencing it, which is of a procedure that I provide, whose invocation ended long ago.
((5140 #f) (#<procedure #N> #t) (300 #t) (THE-IO-VECTOR #t) (#t #t) (1080 #t) (#<frame one deeper than f> #t) ("http" #t) (50 #t))
That stack frame in turn is referenced by the procedure that created the vector.
(frame-o f2) ((3092 #f) (#<input-output-port #N (tcp-client)> #t) (#t #t) (#!eof #t) (#<mutex #N> #t) (#<frame f > #t) (#t #t) (() #t) (#<unknown> #f))
And that stack frame in turn is referenced by ##read-u8 (!)
(frame-o f3) ((3092 #f) (#<frame f2? > #t) (#<input-output-port #N (tcp-client)> #t) (#\U32bfaeaf #f) (#\U32bfaeaf #f))
And that stack frame in turn is referenced by ##byte-rbuf-fill (!)
(frame-o f4) ((1044 #f) (-515899357 #t) (#<frame f3? > #t) (#<input-output-port #N (tcp-client)> #t) (1 #t))
And that stack frame in turn is referenced by ##wait-for-io! (!)
(frame-o f5) ((1044 #f))
And that stack frame in turn is referenced by ##thread-interrupt! (!)
(frame-o f6) ((3092 #f))
And that stack frame in turn is referenced by ##thread-call (!)
(frame-o f7) ((1026 #f) (#<frame f6?? > #t) (#<procedure #N > #t) (-5 #f) (4 #f))
I've gotten to a point where my best understanding is that this memory leak problem consists of: When an IO timeout exception happens, the thread is removed from any thread groups so its execution is halted, but a memory reference is left to it on the heap. This is what seems to be going on here. The stack frame analysis I made in the previous commit, essentially shows Sack's ordinary calling of ##read-u8 to read a HTTP header, and IO timeout here is the ordinary exit path. I'll try to reduce the error and to quantify its occurrence more. Mikael 2013/6/20 Mikael <mikael.rcv@gmail.com>
Dear Marc,
Soo I've hunted this memory leak down a bit now and I'm starting to feel I got somewhere:
So basically I noted that my heap was filling up with a vector object that I use for IO.
Now analyzed what references it using ##resolve-referencing-objects .
This memory leak seems to happen for something-like-in-the-range every 10:th incoming TCP connection.
It's handled by Sack, and then as we see left on the heap are some structures regarding its I/O.
This is a bit funny as I have the impression that Sack can run without producing any memory leaks too.
Here's the tracing of the root for such a vector:
A stack frame is referencing it, which is of a procedure that I provide, whose invocation ended long ago.
That stack frame in turn is referenced by the procedure that created the vector.
And that stack frame in turn is referenced by ##read-u8 (!)
And that stack frame in turn is referenced by ##byte-rbuf-fill (!)
And that stack frame in turn is referenced by ##wait-for-io! (!)
And that stack frame in turn is referenced by ##thread-interrupt! (!)
And that stack frame in turn is referenced by ##thread-call (!)
And that stack frame in turn is referenced by a continuation object.
The continuation object's denv is
#(#((#<procedure #N current-directory> . "/home/USER/") () ()) #(0) 0 0 (#<procedure #N current-exception-handler> . #<procedure #N primordial-exception-handler>) (#<procedure #N current-input-port> . #<input-output-port #N dummy>) (#<procedure #N current-output-port> . #<input-output-port #N dummy>) (#f . #f))
That continuation object in turn is referenced by
* A mutex with the mutex-name 'thread-call-result ,
mutex-state #<thread #1 primordial> ,
mutex-specific set to the continuation itself.
* This vector #(3 2 #<continuation #N> #!unbound) - I think that's a product of the evaluator
* (#<continuation #N>)
* (#<continuation #N> . #t)
The mutex is in turn referenced by another three mutexes, each being a 'thread-call-result mutex with another continuation as its specific.
If there's anything more that's relevant for me to check here, feel free to let me know.
Do you see any plausible reason for the leak?
Best regards, Mikael
participants (5)
-
Bradley Lucier -
Guillaume Cartier -
Jason Felice -
Marc Feeley -
Mikael