Dear Marc,
I'm having a memory leak issue that takes hours and hours to resolve.
Is there any way to traverse the heap as to find the object that references a particular object?
E.g. (object-references object) => referencing-objects-list
It's completely OK if it requires scanning all the heap :}
If it's not in Gambit already, I guess one such routine could be created easily-enough based on the GC mark routines.
Thanks, Mikael
Afficher les réponses par date
On 2013-06-18, at 8:51 AM, Mikael mikael.rcv@gmail.com wrote:
Dear Marc,
I'm having a memory leak issue that takes hours and hours to resolve.
Is there any way to traverse the heap as to find the object that references a particular object?
E.g. (object-references object) => referencing-objects-list
It's completely OK if it requires scanning all the heap :}
If it's not in Gambit already, I guess one such routine could be created easily-enough based on the GC mark routines.
It is not in Gambit, but it would be easy to modify the GC to do this. By the way, your proposed API is not quite right because the references can be in non Scheme objects (for example global variables, a stack frame, a C allocated object). Also, it could be that the object is a ___STILL object that has a non-zero reference count. So a leak is possible if there's a missing reference count "decrement" in the code.
Marc
2013/6/18 Marc Feeley feeley@iro.umontreal.ca
It is not in Gambit, but it would be easy to modify the GC to do this.
Right!
Three q:s on how to do this -
I'm reading mem.c right now and it looks like the mark_array() procedure would be the place to put this check in.
The GC copy phase is about changing every object reference to a new one. So, if during the mark phase I'd print out the ___SCMOBJ of the referencing object, that would change during the GC.
* Where in mem.c is the copy phase implemented? I can't find it or any reference to it in ___garbage_collect , that would not be the section "Move the stack", would it??
* Could I safely run ___garbage_collect up to where the copy begins and then just return() it right there as to keep the validity of the references of the referencing objects I found?
By the way, your proposed API is not quite right because the references can be in non Scheme objects (for example global variables, a stack frame, a C allocated object). Also, it could be that the object is a ___STILL object that has a non-zero reference count.
That would be cool - the objects I'm looking to track references of are deep in the Scheme logics only and unrelated with any FFI or other special object management. So I'd guess the place to check for the object would be in the loop in mark_array:
while (n > 0) { ___WORD obj = *start;
[Here - row 2100]
if (___MEM_ALLOCATED(obj)) { ___WORD *body; ___WORD head; int head_typ; int subtype;
* Last, in mark_array, can I easily detect the object reference to the object whose content is being marked here, or do I need to add this as another argument to mark_array?
So a leak is possible if there's a missing reference count "decrement" in the code.
Ah right - the nature of this memory leak is not such. The only ___STILL variables I have are u8vectors , and the issue I have is that some procedures and vectors fill up the heap -
##serial-number-to-object-table is growing up to heap overflow!
Marc
Best regards, Mikael
First you have to
./configure --enable-debug
this will enable the code in the GC that does consistency checking and also maintains some auxiliairy information that is useful.
The function mark_array is the only function doing the marking. If the debug level is >= 1 it will call the function validate_old_obj(obj) to check that the object reference is indeed pointing to the old semispace. This is a convenient place to do something with the reference, such as checking if it is the one you are looking for, i.e.
if (obj == looking_for) printf("found!\n");
*All* live references will be passed to this function (if your runtime debug options are -:d1 or above).
In validate_old_obj you can use the variables reference_location and container_body to determine where the reference was found. Check the definition of the function explain_problem to see how to use this information. The reference_location can be
#define IN_OBJECT 0 #define IN_REGISTER 1 #define IN_GLOBAL_VAR 2 #define IN_WILL_LIST 3 #define IN_CONTINUATION 4 #define IN_RC 5
The variable container_body is only valid when reference_location == IN_OBJECT.
Marc
On 2013-06-18, at 9:30 AM, Mikael mikael.rcv@gmail.com wrote:
2013/6/18 Marc Feeley feeley@iro.umontreal.ca
It is not in Gambit, but it would be easy to modify the GC to do this.
Right!
Three q:s on how to do this -
I'm reading mem.c right now and it looks like the mark_array() procedure would be the place to put this check in.
The GC copy phase is about changing every object reference to a new one. So, if during the mark phase I'd print out the ___SCMOBJ of the referencing object, that would change during the GC.
Where in mem.c is the copy phase implemented? I can't find it or any reference to it in ___garbage_collect , that would not be the section "Move the stack", would it??
Could I safely run ___garbage_collect up to where the copy begins and then just return() it right there as to keep the validity of the references of the referencing objects I found?
By the way, your proposed API is not quite right because the references can be in non Scheme objects (for example global variables, a stack frame, a C allocated object). Also, it could be that the object is a ___STILL object that has a non-zero reference count.
That would be cool - the objects I'm looking to track references of are deep in the Scheme logics only and unrelated with any FFI or other special object management. So I'd guess the place to check for the object would be in the loop in mark_array:
while (n > 0) { ___WORD obj = *start;
[Here - row 2100] if (___MEM_ALLOCATED(obj)) { ___WORD *body; ___WORD head; int head_typ; int subtype;
- Last, in mark_array, can I easily detect the object reference to the object whose content is being marked here, or do I need to add this as another argument to mark_array?
So a leak is possible if there's a missing reference count "decrement" in the code.
Ah right - the nature of this memory leak is not such. The only ___STILL variables I have are u8vectors , and the issue I have is that some procedures and vectors fill up the heap -
##serial-number-to-object-table is growing up to heap overflow!
Marc
Best regards, Mikael
Great!
So, say that I have an object z that I want to check all refs to:
(define x (vector 1 2 3))
well, and let's create two refs so the environment is completely set up:
(define y (lambda () z)) (define z (vector x))
Scheme object references can be exported and imported using
(define int->scheme-object (c-lambda (unsigned-int64) scheme-object "___result = ___arg1;")) (define scheme-object->int (c-lambda (scheme-object) unsigned-int64 "___result = ___arg1;"))
so I do
(define r (scheme-object->int x)) r
which shows that this vector has reference number 18446744072497743289 .
To the beginning of mem.c (row 20 and on) I add
___WORD DEBUG_scanned_for_object_ref = -1;
and in GDB I do
set var DEBUG_scanned_for_object_ref=18446744072497743289
and then to validate_old_obj (or mark_array) I add the equivalence check:
if (obj == DEBUG_scanned_for_object_ref) { fprintf(stderr,"The scanned for object reference %lu was referenced to by %lu .\n",DEBUG_scanned_for_object_ref,DEBUG_at_object); }
Indeed it trigs as it should!
Now, question is just how to extract DEBUG_at_object .
So, presuming reference_location == IN_OBJECT , we have the content of the object containing the reference in container_body , and we can print some debug info about this object using explain_problem .
How do I get the ___SCMOBJ that container_body will have after the garbage collection is performed?
(Or alternatively, how do I cancel the garbage collection, and get the ___SCMOBJ for container_body in its present location)
2013/6/18 Marc Feeley feeley@iro.umontreal.ca
First you have to
./configure --enable-debug
this will enable the code in the GC that does consistency checking and also maintains some auxiliairy information that is useful.
The function mark_array is the only function doing the marking. If the debug level is >= 1 it will call the function validate_old_obj(obj) to check that the object reference is indeed pointing to the old semispace. This is a convenient place to do something with the reference, such as checking if it is the one you are looking for, i.e.
if (obj == looking_for) printf("found!\n");
*All* live references will be passed to this function (if your runtime debug options are -:d1 or above).
In validate_old_obj you can use the variables reference_location and container_body to determine where the reference was found. Check the definition of the function explain_problem to see how to use this information. The reference_location can be
#define IN_OBJECT 0 #define IN_REGISTER 1 #define IN_GLOBAL_VAR 2 #define IN_WILL_LIST 3 #define IN_CONTINUATION 4 #define IN_RC 5
The variable container_body is only valid when reference_location == IN_OBJECT.
Marc
On 2013-06-18, at 9:30 AM, Mikael mikael.rcv@gmail.com wrote:
2013/6/18 Marc Feeley feeley@iro.umontreal.ca
It is not in Gambit, but it would be easy to modify the GC to do this.
Right!
Three q:s on how to do this -
I'm reading mem.c right now and it looks like the mark_array() procedure
would be the place to put this check in.
The GC copy phase is about changing every object reference to a new one. So, if during the mark phase I'd print out the ___SCMOBJ of the
referencing object, that would change during the GC.
- Where in mem.c is the copy phase implemented? I can't find it or any
reference to it in ___garbage_collect , that would not be the section "Move the stack", would it??
- Could I safely run ___garbage_collect up to where the copy begins and
then just return() it right there as to keep the validity of the references of the referencing objects I found?
By the way, your proposed API is not quite right because the references
can be in non Scheme objects (for example global variables, a stack frame, a C allocated object). Also, it could be that the object is a ___STILL object that has a non-zero reference count.
That would be cool - the objects I'm looking to track references of are
deep in the Scheme logics only and unrelated with any FFI or other special object management. So I'd guess the place to check for the object would be in the loop in mark_array:
while (n > 0) { ___WORD obj = *start;
[Here - row 2100] if (___MEM_ALLOCATED(obj)) { ___WORD *body; ___WORD head; int head_typ; int subtype;
- Last, in mark_array, can I easily detect the object reference to the
object whose content is being marked here, or do I need to add this as another argument to mark_array?
So a leak is possible if there's a missing reference count "decrement"
in the code.
Ah right - the nature of this memory leak is not such. The only ___STILL
variables I have are u8vectors , and the issue I have is that some procedures and vectors fill up the heap -
##serial-number-to-object-table is growing up to heap overflow!
Marc
Best regards, Mikael
On 2013-06-18, at 10:28 AM, Mikael mikael.rcv@gmail.com wrote:
Great!
So, say that I have an object z that I want to check all refs to:
(define x (vector 1 2 3))
well, and let's create two refs so the environment is completely set up:
(define y (lambda () z)) (define z (vector x))
Scheme object references can be exported and imported using
(define int->scheme-object (c-lambda (unsigned-int64) scheme-object "___result = ___arg1;")) (define scheme-object->int (c-lambda (scheme-object) unsigned-int64 "___result = ___arg1;"))
These functions already exist:
(##object->encoding (cons 11 22))
4562002011
(##encoding->object 4562002011)
(11 . 22)
Of course, this is unsafe...
so I do
(define r (scheme-object->int x)) r
which shows that this vector has reference number 18446744072497743289 .
To the beginning of mem.c (row 20 and on) I add
___WORD DEBUG_scanned_for_object_ref = -1;
and in GDB I do
set var DEBUG_scanned_for_object_ref=18446744072497743289
and then to validate_old_obj (or mark_array) I add the equivalence check:
if (obj == DEBUG_scanned_for_object_ref) { fprintf(stderr,"The scanned for object reference %lu was referenced to by %lu .\n",DEBUG_scanned_for_object_ref,DEBUG_at_object); }
Indeed it trigs as it should!
Now, question is just how to extract DEBUG_at_object .
So, presuming reference_location == IN_OBJECT , we have the content of the object containing the reference in container_body , and we can print some debug info about this object using explain_problem .
How do I get the ___SCMOBJ that container_body will have after the garbage collection is performed?
The container_body is a pointer to the body of the copied object. So it is the new location.
(Or alternatively, how do I cancel the garbage collection, and get the ___SCMOBJ for container_body in its present location)
I don't think aborting a GC is safe. *Don't do it*
Marc
2013/6/18 Marc Feeley feeley@iro.umontreal.ca
These functions already exist:
(##object->encoding (cons 11 22))
4562002011
(##encoding->object 4562002011)
(11 . 22)
Of course, this is unsafe...
Great!
The container_body is a pointer to the body of the copied object. So it is
the new location.
Super!
(Or alternatively, how do I cancel the garbage collection, and get the
___SCMOBJ for container_body in its present location)
I don't think aborting a GC is safe. *Don't do it*
Ok. Considering the above there's no reason anyhow.
Soo:
The equivalence printing block, I defined now as
fprintf(stderr,"The scanned for object reference %lu was referenced to by %lu .\n",DEBUG_scanned_for_object_ref,(reference_location == IN_OBJECT) ? ((int*) container_body) : 0);
So if there's a valid object reference, it should be printed out. (The int typecast here may be a bit dirty but on this 32bit setup it should be non-destructive)
Run code:
(define a (vector 1 2 3))
(define b (lambda () a)) (define c (vector a)) (define d (list a))
(define r (##object->encoding a)) r
prints 3081924705.
In GDB, I do
set var DEBUG_scanned_for_object_ref=3081924705 cont
and then in Gambit
(##gc)
And Gambit prints:
The scanned for object reference 3081924705 was referenced to by 0 . The scanned for object reference 3081924705 was referenced to by 3083239844 . The scanned for object reference 3081924705 was referenced to by 3083239968 .
The 0 should refer to the reference to the object from the global variable a, so that's fine.
Also, I guess we see here that the lambda refers to the global variable slot in a general way and not to this variable specifically, so therefore the lambda doesn't create an object reference.
Then to inspect the objects refering to a:
(##encoding->object 3083239844)
-302931863
(##encoding->object 3083239968)
-302931832
So something is invalid about doing what I did in the printf above, which should correspond to
___SCMOBJ newref = (reference_location == IN_OBJECT) ? ((int*) container_body) : ___FAL;
Em. The ___SCMOBJ *is* the new location, isn't it, so this should have worked -
What did I miss, how is this done correctly?
2013/6/18 Mikael mikael.rcv@gmail.com
So something is invalid about doing what I did in the printf above, which should correspond to
___SCMOBJ newref = (reference_location == IN_OBJECT) ? ((int*) container_body) : ___FAL;
Em. The ___SCMOBJ *is* the new location, isn't it, so this should have worked -
What did I miss, how is this done correctly?
Aah right - container_body needs to be given the proper header tag in order to be a ___SCMOBJ. So to get it:
When reference_location == IN_OBJECT , to container_body, I need to add an object type mask based on ___HD_WORDS(head) and ___HD_SUBTYPE(head) where head = container_body[-1] .
This would be done by applying on container_body, depending on its type:
* If fixnum: << ___TB + ___tFIXNUM
* If special object (#f etc.): << ___TB + ___tSPECIAL
* If pair: << ___TB + ___tPAIR
* If other type: How, there's some macros like __BODY , would any of those be of use?
Also would there be some macro for the fixnum/special/pair cases?
Ah, yey, your code in explain_problem already provided an example of how to do this!
Will write here soon how to do this lookup of referencing objects soon.
Also, your code would be useful to others, so please do a pull request when you are done.
Marc
On Jun 18, 2013, at 12:00 PM, Mikael mikael.rcv@gmail.com wrote:
Ah, yey, your code in explain_problem already provided an example of how to do this!
Will write here soon how to do this lookup of referencing objects soon.