[gambit-list] More benchmarks

MichaelL at frogware.com MichaelL at frogware.com
Thu Oct 19 12:44:07 EDT 2006


Hello, Marc & Will. 

For what it's worth, when I last looked at the way Chez & Petite were run 
I noticed that it wasn't quite correct. The script I saw does this:

        chez_exec ()
        {
          echo "(load \"$1.scm\")" | /usr/bin/time scheme
        }

        petite_chez_exec ()
        {
          echo "(load \"$1.scm\")" | /usr/bin/time petite
        }

More correct would be something like this:

        "(optimize-level 2) (module () (import-only scheme) (include 
\"$1.scm\"))"

The "(optimize-level 2)" will enable immutable Scheme bindings (and 
primitive inlining) while still producing safe code. The top-level module 
(especially with the absence of any exports) will signal to Chez that it 
is free to inline. The "(import-only scheme)" declaration is virtually 
identical to the "(optimize-level 2)" declaration, at least in Chez 7, and 
is in fact preferred over setting the optimize level--but Will is using 
6.1 so I'd recommend using both since it will do no harm and 6.1 might be 
different. (See http://www.scheme.com/csug7/use.html#./use:h3 for more 
information on all of this.) Btw, a quick look at the Scheme code shows 
this change will create at least one problem because expressions sometimes 
appear before definitions (eg, the "(setup)" in boyer.scm before the 
definition of "main.").

Finally, make sure you aren't measuring compilation time in addition to 
running time. The above approach appears to be doing just that; for Chez 
at least compilation time can be significant relative to a small running 
time. Optimizations take time. :-)

If you're interested in more Chez times we have Chez 7.1 for Windows, 
Linux, Solaris, and Mac. (Our Sun box is rather ancient though.)

(Apologies if I misread something about the way the tests were set up.)

> Hello.  Will Clinger has done some "independent testing" of the 
> performance of various Scheme systems using the Gambit benchmark 
> suite in the "R6RS" mode (safe code, generic arithmetic, immutable 
> bindings).  The results are published on the following page:
> 
>    http://www.ccs.neu.edu/home/will/Twobit/benchmarks2006.html
> 
> What's particularly interesting is that he benchmarked with more 
> systems than I have access to (including Larceny, MIT-Scheme and Chez 
> Scheme) and on two platforms (Sun Sparc and Intel/Linux).
> 
> The "geometric mean" results on the Sparc show that the code 
> generated by Gambit is among the fastest of all the Scheme systems 
> tested (it is only slightly slower than Chez Scheme and basically the 
> same speed as Larceny, which are both native code compilers).  Gambit 
> is the fastest of the Scheme to C compilers (about 2 times faster 
> than Bigloo and 4 times faster than Chicken).  MzScheme and Scheme48 
> are about 10 times slower than Gambit.  On Linux the situation is 
> similar, and Gambit clearly dominates the other systems, about 50% 
> faster than the next fastest, i.e. Larceny and Bigloo.  Note that 
> Chez Scheme was not available on the Linux machine.
> 
> So, both on Sparc and Intel, it appears that Gambit is the fastest 
> open-source Scheme system.
> 
> Marc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.iro.umontreal.ca/pipermail/gambit-list/attachments/20061019/f8100aa2/attachment.htm>


More information about the Gambit-list mailing list