Based on some of the feedback I have updated the Gambit benchmark suite (http://www.iro.umontreal.ca/~gambit/bench.tar.gz) and the result page (http://www.iro.umontreal.ca/~gambit/bench.html). This time the benchmarks were run in three different situations. This was done by using compiler options and/or declarations.
- Safe with generic arithmetic - generic arithmetic operations - overflow detection on fixnum arithmetic (either produce a bignum or signal an exception) - immutable bindings for the definitions in the benchmark - immutable predefined bindings (for +, car, ...) - safe execution (i.e. an exception must be signalled on errors)
- Safe with fixnum/flonum specializations - arithmetic operations are specialized to fixnum or flonum arguments as appropriate (for example turning + into fl+) and the fixnum operations may wrap on overflow - immutable bindings for the definitions in the benchmark - immutable predefined bindings (for +, car, ...) - safe execution (i.e. an exception must be signalled on errors)
- Unsafe with fixnum/flonum specializations - like ``Safe with fixnum/flonum specializations'' but errors are not checked
I only ran MzScheme in the ``Safe with generic arithmetic'' situation because I didn't have time to figure out what was needed to get it to work in the other situations (actually I don't think MzScheme has an unsafe mode, but I could be wrong).
Anyway, in half of the benchmarks Gambit-C is faster than Bigloo when using fixnum/flonum specializations (in the safe and unsafe situations). For some benchmarks which use call/cc Gambit-C is 50 to 60 times faster than Bigloo. On floating point intensive code, Gambit-C can be 12 times slower than Bigloo but sometimes 3 times faster. The worst case for Gambit-C is the "string" benchmark which is 15 times slower (a large part of the slowdown is due to the string representation, Gambit-C uses 4 bytes per char and Bigloo 1 byte per char; this could be improved by recompiling Gambit-C so that strings use 1 or 2 bytes per char).
Of course, benchmarks are benchmarks. These measurements may change (sometimes dramatically) between different versions of the systems. So please take this with a big grain of salt.
Marc