After I saw Eli Barzilay's post to comp.lang.scheme about the performance of MzScheme compared to other Scheme systems (http:// groups.google.com/group/comp.lang.scheme/msg/473f91e7065b6f73? dmode=source) I was tempted to do some benchmarking of my own using the Gambit Scheme benchmark suite which goes beyond the classical Gabriel benchmarks (it has close to 60 Scheme benchmarks). Most benchmarks are straightforward R4RS code and so should work on just about any Scheme system.
I have updated the benchmarking infrastructure to accomodate other Scheme systems, and MzScheme and Bigloo in particular. I also wrote a script to run all benchmarks and automatically generate an HTML table of the results. The table can be viewed here:
http://www.iro.umontreal.ca/~gambit/bench.html
And the benchmark suite is available here:
http://www.iro.umontreal.ca/~gambit/bench.tar.gz
As you can see from the table, Gambit-C is faster than Bigloo and MzScheme on roughly 3 out of 4 benchmarks. I have used the same assumptions as Eli, which approximate the semantics of R6RS (see the last section of the result page for details).
It would be really interesting to extend this experiment to other Scheme systems. It should not be too difficult to modify the benchmarking scripts to test other systems. The hardest parts are installing the Scheme system, figuring out which compiler options are equivalent to the ones used by the other systems (i.e. approximate R6RS semantics), and work out unexpected problems.
The following systems are particularly interesting to compare to:
- Chicken - Scheme 48 - Larceny - Chez Scheme
Are there any volunteers?
Marc
Afficher les réponses par date
Isn't there a little problem that gambc40b18 isn't available *except* on your MacBook Pro ;-)?
Brad
On 27-Aug-06, at 8:17 AM, j.romildo@gmail.com wrote:
On Sun, Aug 27, 2006 at 12:08:16AM -0400, Bradley Lucier wrote:
Isn't there a little problem that gambc40b18 isn't available *except* on your MacBook Pro ;-)?
Is the release of gambc40b18 imminent?
Romildo
Yes, in fixnum days.
Marc
Based on some of the feedback I have updated the Gambit benchmark suite (http://www.iro.umontreal.ca/~gambit/bench.tar.gz) and the result page (http://www.iro.umontreal.ca/~gambit/bench.html). This time the benchmarks were run in three different situations. This was done by using compiler options and/or declarations.
- Safe with generic arithmetic - generic arithmetic operations - overflow detection on fixnum arithmetic (either produce a bignum or signal an exception) - immutable bindings for the definitions in the benchmark - immutable predefined bindings (for +, car, ...) - safe execution (i.e. an exception must be signalled on errors)
- Safe with fixnum/flonum specializations - arithmetic operations are specialized to fixnum or flonum arguments as appropriate (for example turning + into fl+) and the fixnum operations may wrap on overflow - immutable bindings for the definitions in the benchmark - immutable predefined bindings (for +, car, ...) - safe execution (i.e. an exception must be signalled on errors)
- Unsafe with fixnum/flonum specializations - like ``Safe with fixnum/flonum specializations'' but errors are not checked
I only ran MzScheme in the ``Safe with generic arithmetic'' situation because I didn't have time to figure out what was needed to get it to work in the other situations (actually I don't think MzScheme has an unsafe mode, but I could be wrong).
Anyway, in half of the benchmarks Gambit-C is faster than Bigloo when using fixnum/flonum specializations (in the safe and unsafe situations). For some benchmarks which use call/cc Gambit-C is 50 to 60 times faster than Bigloo. On floating point intensive code, Gambit-C can be 12 times slower than Bigloo but sometimes 3 times faster. The worst case for Gambit-C is the "string" benchmark which is 15 times slower (a large part of the slowdown is due to the string representation, Gambit-C uses 4 bytes per char and Bigloo 1 byte per char; this could be improved by recompiling Gambit-C so that strings use 1 or 2 bytes per char).
Of course, benchmarks are benchmarks. These measurements may change (sometimes dramatically) between different versions of the systems. So please take this with a big grain of salt.
Marc
Sun, 27 Aug 2006 18:38:30 -0400, feeley wrote:
representation, Gambit-C uses 4 bytes per char and Bigloo 1 byte per char; this could be improved by recompiling Gambit-C so that strings use 1 or 2 bytes per char
Is this documented anywhere? (I guess that this is the reason why my larger string/symbol-intensive programs will need so much more memory compared to any other Scheme implementation.)
Sven
On 28-Aug-06, at 2:58 AM, Sven.Hartrumpf@FernUni-Hagen.de wrote:
Sun, 27 Aug 2006 18:38:30 -0400, feeley wrote:
representation, Gambit-C uses 4 bytes per char and Bigloo 1 byte per char; this could be improved by recompiling Gambit-C so that strings use 1 or 2 bytes per char
Is this documented anywhere? (I guess that this is the reason why my larger string/symbol-intensive programs will need so much more memory compared to any other Scheme implementation.)
There are a few build parameters that are defined in include/ gambit.h . In particular:
/* * Range and size of Scheme characters. * * ___MAX_CHR must be 0xff, 0xffff or 0x10ffff. The value 0xff is * appropriate when text is limited to the ISO-8859-1 subset of * Unicode. The value 0xffff is for when text is limited to the BMP * (Basic Multilingual Plane) subset of Unicode. The value 0x10ffff * allows all Unicode characters in text. Note that the number of * bytes per character in a string depends on ___MAX_CHR as follows: * * ___MAX_CHR = 0xff => 1 byte per character * ___MAX_CHR = 0xffff => 2 bytes per character * ___MAX_CHR = 0x10ffff => 4 bytes per character */
#ifndef ___MAX_CHR #define ___MAX_CHR 0x10ffff #endif
Perhaps I should turn this into a "configure" option.
Marc