Hello list,
I am currently writing some FFI code that interfaces with libtiff to read tags from TIFF image files. The values of these tags have different C types and using the right type is needed to avoid memory corruption.
For reading tags with uint16 values the code is straightforward:
``` static ___SCMOBJ uint16_tag(TIFF *tiff, ttag_t tag) { uint16 val; int ret = TIFFGetField(tiff, tag, &val);
return ret == 1 ? ___FIX(val) : ___FAL; } ```
But, since fixnums have less than 32 bits I cannot use the same code. It seems the solution would be something similar to:
``` static ___SCMOBJ uint32_tag(TIFF *tiff, ttag_t tag) { uint32 val;
if(TIFFGetField(tiff, tag, &val) == 1) { ___SCMOBJ ret; ___U32_to_SCMOBJ(___ps, val, &ret, ___RETURN_POS); <---- ERROR return ret; } else { return ___FAL; } }
```
except that I don't know how to get hold of the processor state. Can anybody shed a light here?
Thanks, -alex http://unendli.ch/
Afficher les réponses par date
One way would be to return a special C "int-or-boolean" structure and then access it via special accessors. E.g. (let* ((r (your-c-call))) (if (struct=false? r) #f (struct-int r))) ?
Marc, any better ideas?
2016-01-06 19:41 GMT+08:00 Alex Silva asandroq@gmail.com:
Hello list,
I am currently writing some FFI code that interfaces with libtiff to read tags from TIFF image files. The values of these tags have different C types and using the right type is needed to avoid memory corruption.
For reading tags with uint16 values the code is straightforward:
static ___SCMOBJ uint16_tag(TIFF *tiff, ttag_t tag) { uint16 val; int ret = TIFFGetField(tiff, tag, &val); return ret == 1 ? ___FIX(val) : ___FAL; }
But, since fixnums have less than 32 bits I cannot use the same code. It seems the solution would be something similar to:
static ___SCMOBJ uint32_tag(TIFF *tiff, ttag_t tag) { uint32 val; if(TIFFGetField(tiff, tag, &val) == 1) { ___SCMOBJ ret; ___U32_to_SCMOBJ(___ps, val, &ret, ___RETURN_POS); <---- ERROR return ret; } else { return ___FAL; } }
except that I don't know how to get hold of the processor state. Can anybody shed a light here?
Thanks, -alex http://unendli.ch/ _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Hallo,
On 06/01/16 12:47, Adam wrote:
One way would be to return a special C "int-or-boolean" structure and then access it via special accessors. E.g. (let* ((r (your-c-call))) (if (struct=false? r) #f (struct-int r))) ?
Well, I am sure it is possible to pack an uint32 inside a scheme object, using the API, and return that. I just don't know the right incantation. :)
Cheers, -alex http://unendli.ch/
The FFI supports all the C integer types so it is simplest to let Gambit do the conversion with (c-lambda (TIFF* ttag_t) unsigned-int32 "uint32_tag").
Is the issue that you need to return an integer or #f? Could one of the integer values be used instead of #f?
Marc
On Jan 6, 2016, at 6:47 AM, Adam adam.mlmb@gmail.com wrote:
One way would be to return a special C "int-or-boolean" structure and then access it via special accessors. E.g. (let* ((r (your-c-call))) (if (struct=false? r) #f (struct-int r))) ?
Marc, any better ideas?
2016-01-06 19:41 GMT+08:00 Alex Silva asandroq@gmail.com: Hello list,
I am currently writing some FFI code that interfaces with libtiff to read tags from TIFF image files. The values of these tags have different C types and using the right type is needed to avoid memory corruption.
For reading tags with uint16 values the code is straightforward:
static ___SCMOBJ uint16_tag(TIFF *tiff, ttag_t tag) { uint16 val; int ret = TIFFGetField(tiff, tag, &val); return ret == 1 ? ___FIX(val) : ___FAL; }
But, since fixnums have less than 32 bits I cannot use the same code. It seems the solution would be something similar to:
static ___SCMOBJ uint32_tag(TIFF *tiff, ttag_t tag) { uint32 val; if(TIFFGetField(tiff, tag, &val) == 1) { ___SCMOBJ ret; ___U32_to_SCMOBJ(___ps, val, &ret, ___RETURN_POS); <---- ERROR return ret; } else { return ___FAL; } }
except that I don't know how to get hold of the processor state. Can anybody shed a light here?
Thanks, -alex http://unendli.ch/ _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Hi Marc,
On 06/01/16 14:22, Marc Feeley wrote:
Is the issue that you need to return an integer or #f? Could one of the integer values be used instead of #f?
Indeed this is the issue. Some tags are optional and I would like to receive #f in the Scheme side.
Cheers,
-alex http://unendli.ch/
Right, that the boolean value not has a mapping within the int32 "namespace".
How would you suggest to pass it then, Marc?
I don't see any clear way this can be done with Scheme structures, so simply passing a C structure as result value that has accessors, per the example in my previous email, seems like the best thing. That one has the issue that it requires a malloc()/free() combo which is a bit unfortunate.
Returning an uint64 would work of course even if it would be a hack - Marc, what is the C FFI integer type names specified to specific numbers of bits? (Usually you just say "integer" etc.)
2016-01-06 21:22 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca:
The FFI supports all the C integer types so it is simplest to let Gambit do the conversion with (c-lambda (TIFF* ttag_t) unsigned-int32 "uint32_tag").
Is the issue that you need to return an integer or #f? Could one of the integer values be used instead of #f?
Marc
On Jan 6, 2016, at 6:47 AM, Adam adam.mlmb@gmail.com wrote:
One way would be to return a special C "int-or-boolean" structure and
then access it via special accessors. E.g. (let* ((r (your-c-call))) (if (struct=false? r) #f (struct-int r))) ?
Marc, any better ideas?
2016-01-06 19:41 GMT+08:00 Alex Silva asandroq@gmail.com: Hello list,
I am currently writing some FFI code that interfaces with libtiff to read tags from TIFF image files. The values of these tags have different C types and using the right type is needed to avoid memory corruption.
For reading tags with uint16 values the code is straightforward:
static ___SCMOBJ uint16_tag(TIFF *tiff, ttag_t tag) { uint16 val; int ret = TIFFGetField(tiff, tag, &val); return ret == 1 ? ___FIX(val) : ___FAL; }
But, since fixnums have less than 32 bits I cannot use the same code. It seems the solution would be something similar to:
static ___SCMOBJ uint32_tag(TIFF *tiff, ttag_t tag) { uint32 val; if(TIFFGetField(tiff, tag, &val) == 1) { ___SCMOBJ ret; ___U32_to_SCMOBJ(___ps, val, &ret, ___RETURN_POS); <---- ERROR return ret; } else { return ___FAL; } }
except that I don't know how to get hold of the processor state. Can anybody shed a light here?
Thanks, -alex http://unendli.ch/ _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Hallo,
On 06/01/16 15:06, Adam wrote:
I don't see any clear way this can be done with Scheme structures, so simply passing a C structure as result value that has accessors, per the example in my previous email, seems like the best thing. That one has the issue that it requires a malloc()/free() combo which is a bit unfortunate.
Oh well, I guess that's life. I guess I will return a signed 64-bit int and test against -1. Not ranting against Gambit, but this makes one appreciate more the brilliance of the Lua C API.
Cheers,
-alex http://unendli.ch/
I’m curious… how would it be done in Lua?
Marc
On Jan 6, 2016, at 9:32 AM, Alex Silva asandroq@gmail.com wrote:
Oh well, I guess that's life. I guess I will return a signed 64-bit int and test against -1. Not ranting against Gambit, but this makes one appreciate more the brilliance of the Lua C API.
Hallo,
On 06/01/16 15:37, Marc Feeley wrote:
I’m curious… how would it be done in Lua?
All C functions exported to the Lua VM must take a `lua_State*` argument. The Lua VM has no global variables and is completely contained in the passed state.
Cheers,
Could you show the actual code required for Lua? Is it able to catch heap overflows in Lua code? When there is a conversion error, does the error message indicate which parameter is causing the problem? How must the glue code be written to avoid space leaks and dangling references? Does it support bignums?
The Gambit FFI was designed to interface to C libraries directly (without having to write glue code). Is this possible with Lua’s FFI?
Marc
On Jan 6, 2016, at 9:41 AM, Alex Silva asandroq@gmail.com wrote:
Hallo,
On 06/01/16 15:37, Marc Feeley wrote:
I’m curious… how would it be done in Lua?
All C functions exported to the Lua VM must take a `lua_State*` argument. The Lua VM has no global variables and is completely contained in the passed state.
Cheers,
Cheers,
-alex http://unendli.ch/
Hi Marc,
On 6 January 2016 at 16:10, Marc Feeley feeley@iro.umontreal.ca wrote:
Could you show the actual code required for Lua? Is it able to catch heap overflows in Lua code? When there is a conversion error, does the error message indicate which parameter is causing the problem? How must the glue code be written to avoid space leaks and dangling references? Does it support bignums?
The Gambit FFI was designed to interface to C libraries directly (without having to write glue code). Is this possible with Lua’s FFI?
The Lua C API always require glue functions. In fact, all C functions callable by the VM have the same fixed prototype. The same code would be:
``` static int uint32_tag(lua_State *L) { TIFF *tiff = (TIFF*)lua_touserdata(L, 1); ttag_t tag = (ttag_t)lua_tointeger(L, 2);
uint32 val; if(TIFFGetField(tiff, tag, &val) == 1) { lua_pushinteger(L, val); } else { lua_pushnil(L); }
return 1; } ``` The Lua API is based around a stack where arguments and return values are passed back and forth. The `lua_to*` and `lua_push*` functions take care of casting, boxing and memory management. Although the FFI code may be a little verbose for simpler functions, I never found a situation where I had to work around something. There are also check functions that help checking the type of arguments are throwing exceptions if the types are wrong.
Cheers,
On Wed, Jan 06, 2016 at 11:59:00PM +0100, Alex Queiroz wrote:
Hi Marc,
On 6 January 2016 at 16:10, Marc Feeley feeley@iro.umontreal.ca wrote:
Could you show the actual code required for Lua? Is it able to catch heap overflows in Lua code? When there is a conversion error, does the error message indicate which parameter is causing the problem? How must the glue code be written to avoid space leaks and dangling references? Does it support bignums?
The Gambit FFI was designed to interface to C libraries directly (without having to write glue code). Is this possible with Lua’s FFI?
The Lua C API always require glue functions. In fact, all C functions callable by the VM have the same fixed prototype. The same code would be:
static int uint32_tag(lua_State *L) { TIFF *tiff = (TIFF*)lua_touserdata(L, 1); ttag_t tag = (ttag_t)lua_tointeger(L, 2); uint32 val; if(TIFFGetField(tiff, tag, &val) == 1) { lua_pushinteger(L, val); } else { lua_pushnil(L); } return 1; }
The Lua API is based around a stack where arguments and return values are passed back and forth. The `lua_to*` and `lua_push*` functions take care of casting, boxing and memory management. Although the FFI code may be a little verbose for simpler functions, I never found a situation where I had to work around something. There are also check functions that help checking the type of arguments are throwing exceptions if the types are wrong.
I once wrote a prototype for an int0-memory low-level code generator that was intended to be used in an interpreter. I ended up using tha same kind of stack API. It all seemed kind of FORTH-like.
-- hendrik
I can see a few problems with the glue code you wrote to interface your C function to Lua:
- it isn’t checking that the number of parameters received by the function is correct - it isn’t checking that the first parameter is a pointer of type TIFF* - it isn’t checking that the first parameter is not NULL - it doesn’t appear to do the right thing when val is a uint32 with the upper bit set (I suspect it will return a negative integer) - if there is a type error (say the second parameter is not an integer), will there be a precise error message or exception indicating that the second parameter was not of the right type? can the exception be caught in Lua? - I suspect that lua_pushinteger may allocate memory (for a bignum or flonum?), so what happens if this allocation causes the heap to overflow? can this be caught in Lua?
In this particular example, Gambit’s glue code is not much longer than the one for Lua, but it doesn’t suffer from these issues. For a typical function with simple types for parameters and result, Gambit’s FFI does not require any glue code.
So as you can probably tell, I’m more enclined to qualify Lua’s FFI as “simple” and not “brilliant”… :-)
Marc
On Jan 6, 2016, at 5:59 PM, Alex Queiroz asandroq@gmail.com wrote:
Hi Marc,
On 6 January 2016 at 16:10, Marc Feeley feeley@iro.umontreal.ca wrote:
Could you show the actual code required for Lua? Is it able to catch heap overflows in Lua code? When there is a conversion error, does the error message indicate which parameter is causing the problem? How must the glue code be written to avoid space leaks and dangling references? Does it support bignums?
The Gambit FFI was designed to interface to C libraries directly (without having to write glue code). Is this possible with Lua’s FFI?
The Lua C API always require glue functions. In fact, all C functions callable by the VM have the same fixed prototype. The same code would be:
static int uint32_tag(lua_State *L) { TIFF *tiff = (TIFF*)lua_touserdata(L, 1); ttag_t tag = (ttag_t)lua_tointeger(L, 2); uint32 val; if(TIFFGetField(tiff, tag, &val) == 1) { lua_pushinteger(L, val); } else { lua_pushnil(L); } return 1; }
The Lua API is based around a stack where arguments and return values are passed back and forth. The `lua_to*` and `lua_push*` functions take care of casting, boxing and memory management. Although the FFI code may be a little verbose for simpler functions, I never found a situation where I had to work around something. There are also check functions that help checking the type of arguments are throwing exceptions if the types are wrong.
Cheers,
-alex http://unendli.ch/
On Wed, Jan 06, 2016 at 11:00:26PM -0500, Marc Feeley wrote:
I can see a few problems with the glue code you wrote to interface your C function to Lua:
- it isn’t checking that the number of parameters received by the function is correct
- it isn’t checking that the first parameter is a pointer of type TIFF*
- it isn’t checking that the first parameter is not NULL
- it doesn’t appear to do the right thing when val is a uint32 with the upper bit set (I suspect it will return a negative integer)
- if there is a type error (say the second parameter is not an integer), will there be a precise error message or exception indicating that the second parameter was not of the right type? can the exception be caught in Lua?
- I suspect that lua_pushinteger may allocate memory (for a bignum or flonum?), so what happens if this allocation causes the heap to overflow? can this be caught in Lua?
In this particular example, Gambit’s glue code is not much longer than the one for Lua, but it doesn’t suffer from these issues. For a typical function with simple types for parameters and result, Gambit’s FFI does not require any glue code.
Let me guess that your FFI accomplishes its goals by generating and compiling C code. You can do a lot that way.
I believe that Lua was originally an interpreter written in portable C, and there was no mechanism in C to call a statically unknown function with a statically unknown number of parameters of statically unknown type. C is not the only language with this problem, and the Lua C interface is about as good as you can get under this constraint.
I don't know if there is such a mechanism now; I've heard that there are such gcc extensions, but that's not as portable as one might like.
So as you can probably tell, I’m more enclined to qualify Lua’s FFI as “simple” and not “brilliant”… :-)
A simplicity born of necessity.
Gambit had entirely different constraints. Didn't it originate as a compiler from Scheme to C?
-- hendrik
You are arguing that you can’t do much better than Lua’s FFI if you don’t have a compiler (code generator?) to generate the glue code. That seems to be an artificial constraint, because it is possible to implement such a code generator! My point is that, for the user of the system, it is better when the process of checking types, number of parameters, heap overflow conditions, exception raising, etc are automated. When the process is not automated, these things are not typically done by the user (or are done incorrectly or not completely) because it is tedious and error prone to check all conditions.
Marc
On Jan 6, 2016, at 11:25 PM, Hendrik Boom hendrik@topoi.pooq.com wrote:
On Wed, Jan 06, 2016 at 11:00:26PM -0500, Marc Feeley wrote:
I can see a few problems with the glue code you wrote to interface your C function to Lua:
- it isn’t checking that the number of parameters received by the function is correct
- it isn’t checking that the first parameter is a pointer of type TIFF*
- it isn’t checking that the first parameter is not NULL
- it doesn’t appear to do the right thing when val is a uint32 with the upper bit set (I suspect it will return a negative integer)
- if there is a type error (say the second parameter is not an integer), will there be a precise error message or exception indicating that the second parameter was not of the right type? can the exception be caught in Lua?
- I suspect that lua_pushinteger may allocate memory (for a bignum or flonum?), so what happens if this allocation causes the heap to overflow? can this be caught in Lua?
In this particular example, Gambit’s glue code is not much longer than the one for Lua, but it doesn’t suffer from these issues. For a typical function with simple types for parameters and result, Gambit’s FFI does not require any glue code.
Let me guess that your FFI accomplishes its goals by generating and compiling C code. You can do a lot that way.
I believe that Lua was originally an interpreter written in portable C, and there was no mechanism in C to call a statically unknown function with a statically unknown number of parameters of statically unknown type. C is not the only language with this problem, and the Lua C interface is about as good as you can get under this constraint.
I don't know if there is such a mechanism now; I've heard that there are such gcc extensions, but that's not as portable as one might like.
So as you can probably tell, I’m more enclined to qualify Lua’s FFI as “simple” and not “brilliant”… :-)
A simplicity born of necessity.
Gambit had entirely different constraints. Didn't it originate as a compiler from Scheme to C?
-- hendrik _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Wed, Jan 06, 2016 at 11:44:51PM -0500, Marc Feeley wrote:
You are arguing that you can’t do much better than Lua’s FFI if you don’t have a compiler (code generator?) to generate the glue code. That seems to be an artificial constraint, because it is possible to implement such a code generator!
It's hard to implement a portable code generator. The generated code tends to e achinee-dependent.
But the real prolem here is that C lacks an essential feature. Lisp got it right. In Lisp you can call a statically unknown function with an arbitrary list of arguments.
My point is that, for the user of the system, it is better when the
process of checking types, number of parameters, heap overflow conditions, exception raising, etc are automated. When the process is not automated, these things are not typically done by the user (or are done incorrectly or not completely) because it is tedious and error prone to check all conditions.
Completely agree.
-- hendrik
Marc
On Jan 6, 2016, at 11:25 PM, Hendrik Boom hendrik@topoi.pooq.com wrote:
On Wed, Jan 06, 2016 at 11:00:26PM -0500, Marc Feeley wrote:
I can see a few problems with the glue code you wrote to interface your C function to Lua:
- it isn’t checking that the number of parameters received by the function is correct
- it isn’t checking that the first parameter is a pointer of type TIFF*
- it isn’t checking that the first parameter is not NULL
- it doesn’t appear to do the right thing when val is a uint32 with the upper bit set (I suspect it will return a negative integer)
- if there is a type error (say the second parameter is not an integer), will there be a precise error message or exception indicating that the second parameter was not of the right type? can the exception be caught in Lua?
- I suspect that lua_pushinteger may allocate memory (for a bignum or flonum?), so what happens if this allocation causes the heap to overflow? can this be caught in Lua?
In this particular example, Gambit’s glue code is not much longer than the one for Lua, but it doesn’t suffer from these issues. For a typical function with simple types for parameters and result, Gambit’s FFI does not require any glue code.
Let me guess that your FFI accomplishes its goals by generating and compiling C code. You can do a lot that way.
I believe that Lua was originally an interpreter written in portable C, and there was no mechanism in C to call a statically unknown function with a statically unknown number of parameters of statically unknown type. C is not the only language with this problem, and the Lua C interface is about as good as you can get under this constraint.
I don't know if there is such a mechanism now; I've heard that there are such gcc extensions, but that's not as portable as one might like.
So as you can probably tell, I’m more enclined to qualify Lua’s FFI as “simple” and not “brilliant”… :-)
A simplicity born of necessity.
Gambit had entirely different constraints. Didn't it originate as a compiler from Scheme to C?
-- hendrik _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Of course one way would be to return a double (which has 53 bits of precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you wouldn’t, then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } } else { ___result = ___FAL; } "))
Marc
On Jan 6, 2016, at 9:06 AM, Adam adam.mlmb@gmail.com wrote:
Right, that the boolean value not has a mapping within the int32 "namespace".
How would you suggest to pass it then, Marc?
I don't see any clear way this can be done with Scheme structures, so simply passing a C structure as result value that has accessors, per the example in my previous email, seems like the best thing. That one has the issue that it requires a malloc()/free() combo which is a bit unfortunate.
Returning an uint64 would work of course even if it would be a hack - Marc, what is the C FFI integer type names specified to specific numbers of bits? (Usually you just say "integer" etc.)
2016-01-06 21:22 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca: The FFI supports all the C integer types so it is simplest to let Gambit do the conversion with (c-lambda (TIFF* ttag_t) unsigned-int32 "uint32_tag").
Is the issue that you need to return an integer or #f? Could one of the integer values be used instead of #f?
Marc
On Jan 6, 2016, at 6:47 AM, Adam adam.mlmb@gmail.com wrote:
One way would be to return a special C "int-or-boolean" structure and then access it via special accessors. E.g. (let* ((r (your-c-call))) (if (struct=false? r) #f (struct-int r))) ?
Marc, any better ideas?
2016-01-06 19:41 GMT+08:00 Alex Silva asandroq@gmail.com: Hello list,
I am currently writing some FFI code that interfaces with libtiff to read tags from TIFF image files. The values of these tags have different C types and using the right type is needed to avoid memory corruption.
For reading tags with uint16 values the code is straightforward:
static ___SCMOBJ uint16_tag(TIFF *tiff, ttag_t tag) { uint16 val; int ret = TIFFGetField(tiff, tag, &val); return ret == 1 ? ___FIX(val) : ___FAL; }
But, since fixnums have less than 32 bits I cannot use the same code. It seems the solution would be something similar to:
static ___SCMOBJ uint32_tag(TIFF *tiff, ttag_t tag) { uint32 val; if(TIFFGetField(tiff, tag, &val) == 1) { ___SCMOBJ ret; ___U32_to_SCMOBJ(___ps, val, &ret, ___RETURN_POS); <---- ERROR return ret; } else { return ___FAL; } }
except that I don't know how to get hold of the processor state. Can anybody shed a light here?
Thanks, -alex http://unendli.ch/ _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
2016-01-06 22:37 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca:
Of course one way would be to return a double (which has 53 bits of precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you wouldn’t, then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } } else { ___result = ___FAL; } "))
Neat! What is the risk involved with this one - I mean, the Scheme object allocation on the Gambit heap could fail under certain circumstances. But would that only be on malloc() failure?
On a 64 bit machine all uint32 values fit in a fixnum so there is no allocation possible. On a 32 bit machine, converting a large uint32 will cause a heap allocation of a bignum. However the code is written so that it detects heap overflows and raises a Scheme exception in that case (the Scheme exception can be handled in Scheme because the GC will have prereserved some room in the heap to do further processing in Scheme).
A Scheme heap overflow exception could be raised by a malloc failure or if the Scheme heap size limit is reached (see -:h runtime option).
Try
% gsi -:h1000 -e "(let loop ((x 0)) (loop (list x)))" *** ERROR IN (string)@1.1 -- Heap overflow
% gsi -:h1000 -e "(with-exception-catcher (lambda (e) (pp (list 'caught e))) (lambda () (let loop ((x 0)) (loop (list x)))))" (caught #<heap-overflow-exception #2>)
Marc
On Jan 6, 2016, at 9:48 AM, Adam adam.mlmb@gmail.com wrote:
2016-01-06 22:37 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca: Of course one way would be to return a double (which has 53 bits of precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you wouldn’t, then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } } else { ___result = ___FAL; } "))
Neat! What is the risk involved with this one - I mean, the Scheme object allocation on the Gambit heap could fail under certain circumstances. But would that only be on malloc() failure?
Marc,
So what we get to then is that that Scheme object allocation could fail. In that case, would the mature way of dealing with it for the code return a special error, and then have a Scheme-side exception handler?
Also just to understand - is the only thin that would allocating objects in the C world to fail, would be that malloc() failed, or are there other more local things that could make the object allocation fail?
(define uint32_tag (lambda (a) (let* ((r ((c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } else { ___result = [ SOME ERROR CODE ]; ? } } else { ___result = ___FAL; } ") a))) (if (eq? a some-error-code) (##raise-heap-overflow-exception) a))
2016-01-06 23:02 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca:
On a 64 bit machine all uint32 values fit in a fixnum so there is no allocation possible. On a 32 bit machine, converting a large uint32 will cause a heap allocation of a bignum. However the code is written so that it detects heap overflows and raises a Scheme exception in that case (the Scheme exception can be handled in Scheme because the GC will have prereserved some room in the heap to do further processing in Scheme).
A Scheme heap overflow exception could be raised by a malloc failure or if the Scheme heap size limit is reached (see -:h runtime option).
Try
% gsi -:h1000 -e "(let loop ((x 0)) (loop (list x)))" *** ERROR IN (string)@1.1 -- Heap overflow
% gsi -:h1000 -e "(with-exception-catcher (lambda (e) (pp (list 'caught e))) (lambda () (let loop ((x 0)) (loop (list x)))))" (caught #<heap-overflow-exception #2>)
Marc
On Jan 6, 2016, at 9:48 AM, Adam adam.mlmb@gmail.com wrote:
2016-01-06 22:37 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca: Of course one way would be to return a double (which has 53 bits of
precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you wouldn’t,
then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result,
___RETURN_POS)) == ___FIX(___NO_ERR)) {
___release_scmobj (___result); } } else { ___result = ___FAL; }
"))
Neat! What is the risk involved with this one - I mean, the Scheme
object allocation on the Gambit heap could fail under certain circumstances. But would that only be on malloc() failure?
The code I sent previously already handles heap overflows properly (with raising an exception on the Scheme side). When ___err is not ___FIX(___NO_ERR), the operation failed and the glue code that was generated by the FFI will handle this properly by raising an exception.
As I said, a heap overflow could also happen when the heap limit that the user specified with -:hNNN is reached. But that is not a fatal error because the exception can be caught in Scheme.
Marc
On Jan 6, 2016, at 4:20 PM, Adam adam.mlmb@gmail.com wrote:
Marc,
So what we get to then is that that Scheme object allocation could fail. In that case, would the mature way of dealing with it for the code return a special error, and then have a Scheme-side exception handler?
Also just to understand - is the only thin that would allocating objects in the C world to fail, would be that malloc() failed, or are there other more local things that could make the object allocation fail?
(define uint32_tag (lambda (a) (let* ((r ((c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } else { ___result = [ SOME ERROR CODE ]; ? } } else { ___result = ___FAL; } ") a))) (if (eq? a some-error-code) (##raise-heap-overflow-exception) a))
2016-01-06 23:02 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca: On a 64 bit machine all uint32 values fit in a fixnum so there is no allocation possible. On a 32 bit machine, converting a large uint32 will cause a heap allocation of a bignum. However the code is written so that it detects heap overflows and raises a Scheme exception in that case (the Scheme exception can be handled in Scheme because the GC will have prereserved some room in the heap to do further processing in Scheme).
A Scheme heap overflow exception could be raised by a malloc failure or if the Scheme heap size limit is reached (see -:h runtime option).
Try
% gsi -:h1000 -e "(let loop ((x 0)) (loop (list x)))" *** ERROR IN (string)@1.1 -- Heap overflow
% gsi -:h1000 -e "(with-exception-catcher (lambda (e) (pp (list 'caught e))) (lambda () (let loop ((x 0)) (loop (list x)))))" (caught #<heap-overflow-exception #2>)
Marc
On Jan 6, 2016, at 9:48 AM, Adam adam.mlmb@gmail.com wrote:
2016-01-06 22:37 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca: Of course one way would be to return a double (which has 53 bits of precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you wouldn’t, then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } } else { ___result = ___FAL; } "))
Neat! What is the risk involved with this one - I mean, the Scheme object allocation on the Gambit heap could fail under certain circumstances. But would that only be on malloc() failure?
Aha interesting, so every c-lambda that returns a scheme-object, will check if ___result is set to an error. Does that check trig on all errors, what's its definition.., is it implemented in any particular macro or alike?
Just for me as a user to be able to know exactly.
2016-01-07 5:28 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca:
The code I sent previously already handles heap overflows properly (with raising an exception on the Scheme side). When ___err is not ___FIX(___NO_ERR), the operation failed and the glue code that was generated by the FFI will handle this properly by raising an exception.
As I said, a heap overflow could also happen when the heap limit that the user specified with -:hNNN is reached. But that is not a fatal error because the exception can be caught in Scheme.
Marc
On Jan 6, 2016, at 4:20 PM, Adam adam.mlmb@gmail.com wrote:
Marc,
So what we get to then is that that Scheme object allocation could fail.
In that case, would the mature way of dealing with it for the code return a special error, and then have a Scheme-side exception handler?
Also just to understand - is the only thin that would allocating objects
in the C world to fail, would be that malloc() failed, or are there other more local things that could make the object allocation fail?
(define uint32_tag (lambda (a) (let* ((r ((c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result,
___RETURN_POS)) == ___FIX(___NO_ERR)) {
___release_scmobj (___result); } else { ___result = [ SOME ERROR CODE ]; ? } } else { ___result = ___FAL; }
") a))) (if (eq? a some-error-code) (##raise-heap-overflow-exception)
a))
2016-01-06 23:02 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca: On a 64 bit machine all uint32 values fit in a fixnum so there is no
allocation possible. On a 32 bit machine, converting a large uint32 will cause a heap allocation of a bignum. However the code is written so that it detects heap overflows and raises a Scheme exception in that case (the Scheme exception can be handled in Scheme because the GC will have prereserved some room in the heap to do further processing in Scheme).
A Scheme heap overflow exception could be raised by a malloc failure or
if the Scheme heap size limit is reached (see -:h runtime option).
Try
% gsi -:h1000 -e "(let loop ((x 0)) (loop (list x)))" *** ERROR IN (string)@1.1 -- Heap overflow
% gsi -:h1000 -e "(with-exception-catcher (lambda (e) (pp (list 'caught
e))) (lambda () (let loop ((x 0)) (loop (list x)))))"
(caught #<heap-overflow-exception #2>)
Marc
On Jan 6, 2016, at 9:48 AM, Adam adam.mlmb@gmail.com wrote:
2016-01-06 22:37 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca: Of course one way would be to return a double (which has 53 bits of
precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you
wouldn’t, then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result,
___RETURN_POS)) == ___FIX(___NO_ERR)) {
___release_scmobj (___result); } } else { ___result = ___FAL; }
"))
Neat! What is the risk involved with this one - I mean, the Scheme
object allocation on the Gambit heap could fail under certain circumstances. But would that only be on malloc() failure?
Actually, all c-lambdas that can generate an exception will check the variable ___err to see if there was an error.
The implementation is the macro ___CFUN_CONV_ERROR in gambit.h .
Marc
On Jan 6, 2016, at 4:32 PM, Adam adam.mlmb@gmail.com wrote:
Aha interesting, so every c-lambda that returns a scheme-object, will check if ___result is set to an error. Does that check trig on all errors, what's its definition.., is it implemented in any particular macro or alike?
Just for me as a user to be able to know exactly.
Aha.
How can I know based on the return value of some random Gambit C function like ___aloc_scmobj or ___U32_to_SCMOBJ that it guaranteedly will trig an exception when returning to the Scheme world, without me needing to take any futher action for that to happen - will that happen in *all* non-___FIX(___NO_ERR) result cases?
2016-01-07 5:37 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca:
Actually, all c-lambdas that can generate an exception will check the variable ___err to see if there was an error.
The implementation is the macro ___CFUN_CONV_ERROR in gambit.h .
Marc
On Jan 6, 2016, at 4:32 PM, Adam adam.mlmb@gmail.com wrote:
Aha interesting, so every c-lambda that returns a scheme-object, will
check if ___result is set to an error. Does that check trig on all errors, what's its definition.., is it implemented in any particular macro or alike?
Just for me as a user to be able to know exactly.
If you go through a c-lambda, the glue code generated by the FFI will check ___err and raise an exception accordingly if ___err != ___FIX(___NO_ERR). It is up to the C code to set ___err correctly. In the code I gave it is the part:
if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); }
Note that this will set ___err in addition to testing if the conversion succeeded (in which case the resulting Scheme object must be released (reference count decremented) because the C world will no longer point to it when it is returned).
If you call ___alloc_scmobj and other internal Gambit runtime functions then you have to check the result explicitly and errors are indicated in two ways. In the case of ___alloc_scmobj, ___make_pair, ___make_vector, and similar functions returning a useful Scheme object (but not a fixnum), then if ___FIXNUMP(result) there was an error and result contains the error code. Functions with side-effects, such as ___setup, ___call, and all the type conversion functions always return a fixnum and anything other than ___FIX(___NO_ERR) indicates an error. If you look at the Gambit sources, you’ll see that sometimes these error checks are done in C and sometimes in Scheme (look at the code for ##process-statistics in lib/_kernel.scm which does both).
Marc
On Jan 6, 2016, at 4:42 PM, Adam adam.mlmb@gmail.com wrote:
Aha.
How can I know based on the return value of some random Gambit C function like ___aloc_scmobj or ___U32_to_SCMOBJ that it guaranteedly will trig an exception when returning to the Scheme world, without me needing to take any futher action for that to happen - will that happen in all non-___FIX(___NO_ERR) result cases?
2016-01-07 5:37 GMT+08:00 Marc Feeley feeley@iro.umontreal.ca: Actually, all c-lambdas that can generate an exception will check the variable ___err to see if there was an error.
The implementation is the macro ___CFUN_CONV_ERROR in gambit.h .
Marc
On Jan 6, 2016, at 4:32 PM, Adam adam.mlmb@gmail.com wrote:
Aha interesting, so every c-lambda that returns a scheme-object, will check if ___result is set to an error. Does that check trig on all errors, what's its definition.., is it implemented in any particular macro or alike?
Just for me as a user to be able to know exactly.
Hallo,
On 06/01/16 15:37, Marc Feeley wrote:
Of course one way would be to return a double (which has 53 bits of precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you wouldn’t, then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } } else { ___result = ___FAL; } "))
Thanks, this looks promising. But I do not understand the call to `___release_scmobj`, wouldn't that release the object that we are returning? Or is it just decrementing some reference count?
Cheers,
The FFI always allocates “still” objects when it converts from a C type to a Scheme type. So the (possibly) bignum that is allocated by ___U32_to_SCMOBJ is a still object. Still objects are managed by a mark and sweep GC algorithm. They have the property that they will never move to a different address during their lifetime. This is a nice property when manipulating these objects from C because if the C code triggers a garbage collection (by allocating other objects), then the reference to the bignum in ___result will still point to the right place.
Still objects contain a reference count which indicates how many references to the object exist from ***C***. A still object will be reclaimed by the GC when there is no reference to the object from the Scheme stack and heap ***and*** the reference count is 0. When a still object is allocated its reference count is initialized to 1. When the C code drops a reference to the object it calls ___release_scmobj to decrement the reference count. This doesn’t reclaim the object (when the reference count is 0). It only notifies the GC that it can eventually reclaim the object if there are no references to it from the Sheme stack and heap.
This approach allows allocating several Scheme objects in a given C function (for example to create a Scheme list of things) without too much code to protect the manipulated objects from being reclaimed by the GC in the middle of the function.
Marc
On Jan 7, 2016, at 5:00 AM, Alex Silva asandroq@gmail.com wrote:
Hallo,
On 06/01/16 15:37, Marc Feeley wrote:
Of course one way would be to return a double (which has 53 bits of precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you wouldn’t, then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } } else { ___result = ___FAL; } "))
Thanks, this looks promising. But I do not understand the call to `___release_scmobj`, wouldn't that release the object that we are returning? Or is it just decrementing some reference count?
Cheers,
-alex http://unendli.ch/
Hallo,
On 06/01/16 15:37, Marc Feeley wrote:
Of course one way would be to return a double (which has 53 bits of precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you wouldn’t, then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } } else { ___result = ___FAL; } "))
I ended up mimicking Lua and passing the state:
tiff.scm.c: ``` static ___SCMOBJ uint32_tag(___processor_state ps, TIFF *tiff, ttag_t tag, ___SCMOBJ *err) { uint32 val;
if(TIFFGetField(tiff, tag, &val) == 1) { ___SCMOBJ ret; if((*err = ___U32_to_SCMOBJ(ps, val, &ret, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj(ret); } return ret; } else { return ___FAL; } }
// example usage static ___SCMOBJ tag_badfaxlines(___processor_state ps, TIFF *tiff, ___SCMOBJ *err) { return uint32_tag(ps, tiff, TIFFTAG_BADFAXLINES, err); } ```
tiff.scm: ``` ;; example usage (define tiff:tag-badfaxlines (c-lambda (TIFF) scheme-object "___result = tag_badfaxlines(___PSTATE, ___arg1, &___err);")) ```
I usually split the code into two files because I really dislike to write C code inside strings.
Cheers,
I see you are passing the processor state as a parameter. The macro ___PSTATE can get the processor state at any point in the code. This avoids having to thread the processor state from one function to the next. Note that inside an “inline” c-lambda, you can access the processor state using ___ps, which is faster than using ___PSTATE (because that macro may access a global or thread local storage).
Marc
On Jan 7, 2016, at 11:24 AM, Alex Silva asandroq@gmail.com wrote:
Hallo,
On 06/01/16 15:37, Marc Feeley wrote:
Of course one way would be to return a double (which has 53 bits of precision) and use NaN to encode “missing tag”. I’m not kidding!
If you don’t like this approach, and I can understand why you wouldn’t, then you have to do the conversion to ___SCMOBJ by directly calling the conversion macros. Something like this:
(define uint32_tag (c-lambda (TIFF* ttag_t) scheme-object " ___U32 val; if (TIFFGetField(___arg1, ___arg2, &val) == 1) { if ((___err = ___U32_to_SCMOBJ (___PSTATE, val, &___result, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj (___result); } } else { ___result = ___FAL; } "))
I ended up mimicking Lua and passing the state:
tiff.scm.c:
static ___SCMOBJ uint32_tag(___processor_state ps, TIFF *tiff, ttag_t tag, ___SCMOBJ *err) { uint32 val; if(TIFFGetField(tiff, tag, &val) == 1) { ___SCMOBJ ret; if((*err = ___U32_to_SCMOBJ(ps, val, &ret, ___RETURN_POS)) == ___FIX(___NO_ERR)) { ___release_scmobj(ret); } return ret; } else { return ___FAL; } } // example usage static ___SCMOBJ tag_badfaxlines(___processor_state ps, TIFF *tiff, ___SCMOBJ *err) { return uint32_tag(ps, tiff, TIFFTAG_BADFAXLINES, err); }
tiff.scm:
;; example usage (define tiff:tag-badfaxlines (c-lambda (TIFF) scheme-object "___result = tag_badfaxlines(___PSTATE, ___arg1, &___err);"))
I usually split the code into two files because I really dislike to write C code inside strings.
Cheers,
-alex http://unendli.ch/