Hello, Marc & Will.
For what it's worth, when I last looked at the way Chez & Petite were run I noticed that it wasn't quite correct. The script I saw does this:
chez_exec () { echo "(load "$1.scm")" | /usr/bin/time scheme }
petite_chez_exec () { echo "(load "$1.scm")" | /usr/bin/time petite }
More correct would be something like this:
"(optimize-level 2) (module () (import-only scheme) (include "$1.scm"))"
The "(optimize-level 2)" will enable immutable Scheme bindings (and primitive inlining) while still producing safe code. The top-level module (especially with the absence of any exports) will signal to Chez that it is free to inline. The "(import-only scheme)" declaration is virtually identical to the "(optimize-level 2)" declaration, at least in Chez 7, and is in fact preferred over setting the optimize level--but Will is using 6.1 so I'd recommend using both since it will do no harm and 6.1 might be different. (See http://www.scheme.com/csug7/use.html#./use:h3 for more information on all of this.) Btw, a quick look at the Scheme code shows this change will create at least one problem because expressions sometimes appear before definitions (eg, the "(setup)" in boyer.scm before the definition of "main.").
Finally, make sure you aren't measuring compilation time in addition to running time. The above approach appears to be doing just that; for Chez at least compilation time can be significant relative to a small running time. Optimizations take time. :-)
If you're interested in more Chez times we have Chez 7.1 for Windows, Linux, Solaris, and Mac. (Our Sun box is rather ancient though.)
(Apologies if I misread something about the way the tests were set up.)
Hello. Will Clinger has done some "independent testing" of the performance of various Scheme systems using the Gambit benchmark suite in the "R6RS" mode (safe code, generic arithmetic, immutable bindings). The results are published on the following page:
http://www.ccs.neu.edu/home/will/Twobit/benchmarks2006.html
What's particularly interesting is that he benchmarked with more systems than I have access to (including Larceny, MIT-Scheme and Chez Scheme) and on two platforms (Sun Sparc and Intel/Linux).
The "geometric mean" results on the Sparc show that the code generated by Gambit is among the fastest of all the Scheme systems tested (it is only slightly slower than Chez Scheme and basically the same speed as Larceny, which are both native code compilers). Gambit is the fastest of the Scheme to C compilers (about 2 times faster than Bigloo and 4 times faster than Chicken). MzScheme and Scheme48 are about 10 times slower than Gambit. On Linux the situation is similar, and Gambit clearly dominates the other systems, about 50% faster than the next fastest, i.e. Larceny and Bigloo. Note that Chez Scheme was not available on the Linux machine.
So, both on Sparc and Intel, it appears that Gambit is the fastest open-source Scheme system.
Marc
Afficher les réponses par date
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 19-Oct-06, at 12:44 PM, MichaelL@frogware.com wrote:
Hello, Marc & Will.
For what it's worth, when I last looked at the way Chez & Petite were run I noticed that it wasn't quite correct. The script I saw does this:
chez_exec () { echo "(load \"$1.scm\")" | /usr/bin/time scheme } petite_chez_exec () { echo "(load \"$1.scm\")" | /usr/bin/time petite }
More correct would be something like this:
"(optimize-level 2) (module () (import-only scheme)
(include "$1.scm"))"
The "(optimize-level 2)" will enable immutable Scheme bindings (and primitive inlining) while still producing safe code. The top-level module (especially with the absence of any exports) will signal to Chez that it is free to inline. The "(import-only scheme)" declaration is virtually identical to the "(optimize-level 2)" declaration, at least in Chez 7, and is in fact preferred over setting the optimize level--but Will is using 6.1 so I'd recommend using both since it will do no harm and 6.1 might be different. (See http://www.scheme.com/csug7/use.html#./use:h3 for more information on all of this.) Btw, a quick look at the Scheme code shows this change will create at least one problem because expressions sometimes appear before definitions (eg, the "(setup)" in boyer.scm before the definition of "main.").
Finally, make sure you aren't measuring compilation time in addition to running time. The above approach appears to be doing just that; for Chez at least compilation time can be significant relative to a small running time. Optimizations take time. :-)
If you're interested in more Chez times we have Chez 7.1 for Windows, Linux, Solaris, and Mac. (Our Sun box is rather ancient though.)
(Apologies if I misread something about the way the tests were set up.)
I do not have access to Chez Scheme. However Will does, and he has modified the "bench" script to call up Chez Scheme this way:
r6rs) INSERTCODE="(define-syntax if-fixflo (syntax-rules () ((if- fixflo yes no) no)))" REPLCOMMANDS="(begin (optimize-level 2) (load "%s.scm") (main))" ;;
So the (optimize-level 2) declaration is being used, but not the (module () ...) wrapper. It is possible that Will could not use that trick because some benchmarks could no longer be compiled because expressions appear before definitions as you mention. I'll let Will look into this since I don't have access to Chez Scheme.
Concerning timing, the Chez Scheme prefix file uses the "(time <expr>)" form to measure the execution time, so the compile time is not included. In the new "bench" script /usr/bin/time is not used for Chez Scheme.
If you can run the benchmarks, I would be interested in seeing how Gambit compares to Chez Scheme 7.1 on Linux and Macintel. I can send you the latest benchmark suite.
Marc
So the (optimize-level 2) declaration is being used, but not the (module () ...) wrapper. It is possible that Will could not use that trick because some benchmarks could no longer be compiled because expressions appear before definitions as you mention. I'll let Will look into this since I don't have access to Chez Scheme.
I took a quick look at some files and it seems that very few have top-level expressions. In those that did it looked like the expressions could simply be moved to the bottom. I think it would be worth the change. In fact I'd personally be interested in the before & after numbers, so I might make the changes myself. I'll also take a look to see if any other obvious & simple changes would help.
If you can run the benchmarks, I would be interested in seeing how Gambit compares to Chez Scheme 7.1 on Linux and Macintel. I can send you the latest benchmark suite.
Ok, I should be able to run it before the end of next week. Btw, I have a PPC Mac. Presumably that's OK. In any event, Chez doesn't currently support Intel on Mac.
Can I grab the bench suite from your site? Or are you saying that isn't the latest?
Btw, none of this detracts from your statement that Gambit is the fastest Open Source Scheme. :-)