Dear Marc,
Just realized there's a memory leak in many Gambit libraries, such as the sqlite FFI - their u8vector allocation in C boils down to:
___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,len,___MOVABLE0);
Studying ##make-u8vector of _kernel.scm provides some inspiration: It
1) Shows that it's fine to use ___STILL
2) Uses ___still_obj_refcount_dec ,
3) Has its C code in a ##c-code and not a ##c-lambda
4) Has its surrounding Scheme code in (##declare (not interrupts-enabled))
5) Surrounds the ___alloc_scmobj call by some magic ( ___FRAME_STORE_RA(___R0)___W_ALL ___R_ALL ___SET_R0(___FRAME_FETCH_RA) )
(##make-u8vector provides a code path for small objects apart from this, though the use regarded here should be for u8vectors of nontrivial size so going with the codepath for large objects should always be fine.)
By applying 1) and 2) I get something that works but can't know if it's really correct and safe:
How do you allocate an u8vector in C code in a way that does not leak memory, and that safely transfers the new object reference from the C world to the Scheme world?
Thanks! Mikael
*How to reproduce* Running a test that repeats
((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___MOVABLE0);"))
over and over e.g.
(##gc-report-set! #t) (for-each (lambda (x) ((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___MOVABLE0);"))) (string->list (make-string 100000)) (##gc)
shows that this memory leaks all the allocated memory (!).
*Fix, duno if safe & complete*
(##gc-report-set! #t) (for-each (lambda (x) ((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___STILL); if (!___FIXNUMP(___result)) ___still_obj_refcount_dec (___result); "))) (string->list (make-string 100000)) (##gc)
*Copy of ##make-u8vector of _kernel.scm for reference*
(define-prim (##make-u8vector k #!optional (fill (macro-absent-obj))) (##declare (not interrupts-enabled)) (let ((v (##c-code #<<end-of-code
long i; long n = ___INT(___ARG1); long words = ___WORDS(n) + 1; ___SCMOBJ result; if (n > ___CAST(___WORD, ___LMASK>>___LF)) result = ___FIX(___HEAP_OVERFLOW_ERR); /* requested object is too big! */ else if (words > ___MSECTION_BIGGEST) { ___FRAME_STORE_RA(___R0) ___W_ALL result = ___alloc_scmobj (___sU8VECTOR, n, ___STILL); ___R_ALL ___SET_R0(___FRAME_FETCH_RA) if (!___FIXNUMP(result)) ___still_obj_refcount_dec (result); } else { ___BOOL overflow = 0; ___hp += words; if (___hp > ___ps->heap_limit) { ___FRAME_STORE_RA(___R0) ___W_ALL overflow = ___heap_limit () && ___garbage_collect (0); ___R_ALL ___SET_R0(___FRAME_FETCH_RA) } else ___hp -= words; if (overflow) result = ___FIX(___HEAP_OVERFLOW_ERR); else { result = ___TAG(___hp, ___tSUBTYPED); ___HEADER(result) = ___MAKE_HD_BYTES(n, ___sU8VECTOR); ___hp += words; } } if (!___FIXNUMP(result) && ___ARG2 != ___ABSENT) { for (i=0; i<n; i++) ___U8VECTORSET(result,___FIX(i),___ARG2) } ___RESULT = result;
end-of-code
k fill))) (if (##fixnum? v) (begin (##raise-heap-overflow-exception) (##make-u8vector k fill)) v)))
Afficher les réponses par date
For a side note, this one leaks too - ___release_scmobj does not fix it.
(##gc-report-set! #t) (for-each (lambda (x) ((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___MOVABLE0); if (!___FIXNUMP(___result)) ___EXT(___release_scmobj) (r); "))) (string->list (make-string 100000)) (##gc)
While I don't know if the
___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,len,___STILL); if (!___FIXNUMP(___result)) ___still_obj_refcount_dec(___result);
variant suggested in the previous email is really safe & complete, experience til now seems to say it's alright.
Brgds
2013/6/16 Mikael mikael.rcv@gmail.com ..
*Fix, duno if safe & complete*
(##gc-report-set! #t) (for-each (lambda (x) ((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___STILL); if (!___FIXNUMP(___result)) ___still_obj_refcount_dec (___result); "))) (string->list (make-string 100000)) (##gc)
Mikael, Marc, the mention of sqlite specifically caught my eye.
I use the following code for accessing blobs in the database as any homogeneous vector type in my own SQLite3 FFI of last summer. The code was exercised as part of a larger program and seemed to work. __STILL was the key.
Maybe it should decrease a refcount when __alloc_scmobj fails?
(c-declare #<<c-declare-end static ___SCMOBJ resqlite3_column_blob(sqlite3_stmt* stmt, int col, int type, int size) { const void *buf = sqlite3_column_blob(stmt, col); int n = sqlite3_column_bytes(stmt, col); if (n % size) return ___FIX(___UNKNOWN_ERR); ___SCMOBJ result = ___EXT(___alloc_scmobj)(type,n,___STILL); if (___FIXNUMP(result)) return result; memcpy(___BODY(result), buf, n); return result; } c-declare-end )
(define sqlite3-column-u8vector-or-code (c-lambda (sqlite3-prepared-statement int) scheme-object "___result = resqlite3_column_blob(___arg1,___arg2,___sU8VECTOR,1);"))
The latter definition is then repeated for each vector type and element size.
Mikael mikael.rcv@gmail.com writes:
Dear Marc,
Just realized there's a memory leak in many Gambit libraries, such as the sqlite FFI - their u8vector allocation in C boils down to:
___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,len,___MOVABLE0);
Studying ##make-u8vector of _kernel.scm provides some inspiration: It
Shows that it's fine to use ___STILL
Uses ___still_obj_refcount_dec ,
Has its C code in a ##c-code and not a ##c-lambda
Has its surrounding Scheme code in (##declare (not interrupts-enabled))
Surrounds the ___alloc_scmobj call by some magic (
___FRAME_STORE_RA(___R0)___W_ALL ___R_ALL ___SET_R0(___FRAME_FETCH_RA) )
(##make-u8vector provides a code path for small objects apart from this, though the use regarded here should be for u8vectors of nontrivial size so going with the codepath for large objects should always be fine.)
By applying 1) and 2) I get something that works but can't know if it's really correct and safe:
How do you allocate an u8vector in C code in a way that does not leak memory, and that safely transfers the new object reference from the C world to the Scheme world?
Thanks! Mikael
*How to reproduce* Running a test that repeats
((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___MOVABLE0);"))
over and over e.g.
(##gc-report-set! #t) (for-each (lambda (x) ((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___MOVABLE0);"))) (string->list (make-string 100000)) (##gc)
shows that this memory leaks all the allocated memory (!).
*Fix, duno if safe & complete*
(##gc-report-set! #t) (for-each (lambda (x) ((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___STILL); if (!___FIXNUMP(___result)) ___still_obj_refcount_dec (___result); "))) (string->list (make-string 100000)) (##gc)
*Copy of ##make-u8vector of _kernel.scm for reference*
(define-prim (##make-u8vector k #!optional (fill (macro-absent-obj))) (##declare (not interrupts-enabled)) (let ((v (##c-code #<<end-of-code
long i; long n = ___INT(___ARG1); long words = ___WORDS(n) + 1; ___SCMOBJ result; if (n > ___CAST(___WORD, ___LMASK>>___LF)) result = ___FIX(___HEAP_OVERFLOW_ERR); /* requested object is too big! */ else if (words > ___MSECTION_BIGGEST) { ___FRAME_STORE_RA(___R0) ___W_ALL result = ___alloc_scmobj (___sU8VECTOR, n, ___STILL); ___R_ALL ___SET_R0(___FRAME_FETCH_RA) if (!___FIXNUMP(result)) ___still_obj_refcount_dec (result); } else { ___BOOL overflow = 0; ___hp += words; if (___hp > ___ps->heap_limit) { ___FRAME_STORE_RA(___R0) ___W_ALL overflow = ___heap_limit () && ___garbage_collect (0); ___R_ALL ___SET_R0(___FRAME_FETCH_RA) } else ___hp -= words; if (overflow) result = ___FIX(___HEAP_OVERFLOW_ERR); else { result = ___TAG(___hp, ___tSUBTYPED); ___HEADER(result) = ___MAKE_HD_BYTES(n, ___sU8VECTOR); ___hp += words; } } if (!___FIXNUMP(result) && ___ARG2 != ___ABSENT) { for (i=0; i<n; i++) ___U8VECTORSET(result,___FIX(i),___ARG2) } ___RESULT = result;
end-of-code
k fill))) (if (##fixnum? v) (begin (##raise-heap-overflow-exception) (##make-u8vector k fill)) v)))
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
The decrease of refcount in ##make-u8vector 's code could possibly have to do with that this procedure uses ##c-code rather than ##c-lambda - if the latter would have been used, maybe the refcount decrease wouldn't have been needed.
I'm still experiencing the memory leak problem though am unclear about where it comes from.
If it would be sqlite now, it would be because that alloc_scmobj somehow was made to leak due to some characteristic of the bigger execution context, that my reduced test case doesn't contain. Possibly I could add that stack management magic code that ##make-u8vector uses just to see if that would make a difference.
The leak manifests as a constant increasing of Gambit's live heap size, as shown by (##display-gc-report) / (##gc-report-set! #t) . *debugging*.
Noone ever experienced a leak in the IO subsystem, right?
Regards, Mikael
2013/6/16 Jussi Piitulainen jpiitula@ling.helsinki.fi
Mikael, Marc, the mention of sqlite specifically caught my eye.
I use the following code for accessing blobs in the database as any homogeneous vector type in my own SQLite3 FFI of last summer. The code was exercised as part of a larger program and seemed to work. __STILL was the key.
Maybe it should decrease a refcount when __alloc_scmobj fails?
(c-declare #<<c-declare-end static ___SCMOBJ resqlite3_column_blob(sqlite3_stmt* stmt, int col, int type, int size) { const void *buf = sqlite3_column_blob(stmt, col); int n = sqlite3_column_bytes(stmt, col); if (n % size) return ___FIX(___UNKNOWN_ERR); ___SCMOBJ result = ___EXT(___alloc_scmobj)(type,n,___STILL); if (___FIXNUMP(result)) return result; memcpy(___BODY(result), buf, n); return result; } c-declare-end )
(define sqlite3-column-u8vector-or-code (c-lambda (sqlite3-prepared-statement int) scheme-object "___result = resqlite3_column_blob(___arg1,___arg2,___sU8VECTOR,1);"))
The latter definition is then repeated for each vector type and element size.
Mikael mikael.rcv@gmail.com writes:
Dear Marc,
Just realized there's a memory leak in many Gambit libraries, such as the sqlite FFI - their u8vector allocation in C boils down to:
___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,len,___MOVABLE0);
Studying ##make-u8vector of _kernel.scm provides some inspiration: It
Shows that it's fine to use ___STILL
Uses ___still_obj_refcount_dec ,
Has its C code in a ##c-code and not a ##c-lambda
Has its surrounding Scheme code in (##declare (not
interrupts-enabled))
- Surrounds the ___alloc_scmobj call by some magic (
___FRAME_STORE_RA(___R0)___W_ALL ___R_ALL ___SET_R0(___FRAME_FETCH_RA) )
(##make-u8vector provides a code path for small objects apart from this, though the use regarded here should be for u8vectors of nontrivial size
so
going with the codepath for large objects should always be fine.)
By applying 1) and 2) I get something that works but can't know if it's really correct and safe:
How do you allocate an u8vector in C code in a way that does not leak memory, and that safely transfers the new object reference from the C
world
to the Scheme world?
Thanks! Mikael
*How to reproduce* Running a test that repeats
((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___MOVABLE0);"))
over and over e.g.
(##gc-report-set! #t) (for-each (lambda (x) ((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___MOVABLE0);"))) (string->list (make-string 100000)) (##gc)
shows that this memory leaks all the allocated memory (!).
*Fix, duno if safe & complete*
(##gc-report-set! #t) (for-each (lambda (x) ((c-lambda () scheme-object "___result = ___EXT(___alloc_scmobj)(___sU8VECTOR,10000,___STILL); if (!___FIXNUMP(___result)) ___still_obj_refcount_dec (___result); "))) (string->list (make-string 100000)) (##gc)
*Copy of ##make-u8vector of _kernel.scm for reference*
(define-prim (##make-u8vector k #!optional (fill (macro-absent-obj))) (##declare (not interrupts-enabled)) (let ((v (##c-code #<<end-of-code
long i; long n = ___INT(___ARG1); long words = ___WORDS(n) + 1; ___SCMOBJ result; if (n > ___CAST(___WORD, ___LMASK>>___LF)) result = ___FIX(___HEAP_OVERFLOW_ERR); /* requested object is too big!
*/
else if (words > ___MSECTION_BIGGEST) { ___FRAME_STORE_RA(___R0) ___W_ALL result = ___alloc_scmobj (___sU8VECTOR, n, ___STILL); ___R_ALL ___SET_R0(___FRAME_FETCH_RA) if (!___FIXNUMP(result)) ___still_obj_refcount_dec (result); } else { ___BOOL overflow = 0; ___hp += words; if (___hp > ___ps->heap_limit) { ___FRAME_STORE_RA(___R0) ___W_ALL overflow = ___heap_limit () && ___garbage_collect (0); ___R_ALL ___SET_R0(___FRAME_FETCH_RA) } else ___hp -= words; if (overflow) result = ___FIX(___HEAP_OVERFLOW_ERR); else { result = ___TAG(___hp, ___tSUBTYPED); ___HEADER(result) = ___MAKE_HD_BYTES(n, ___sU8VECTOR); ___hp += words; } } if (!___FIXNUMP(result) && ___ARG2 != ___ABSENT) { for (i=0; i<n; i++) ___U8VECTORSET(result,___FIX(i),___ARG2) } ___RESULT = result;
end-of-code
k fill))) (if (##fixnum? v) (begin (##raise-heap-overflow-exception) (##make-u8vector k fill)) v)))
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list