Le 2012-11-06 à 3:50 AM, Hugh Aguilar hughaguilar96@yahoo.com a écrit :
I am very impressed that Racket is as fast as it is. I had expected the compilers such as Gambit to be much faster than the VM-JIT system. If Racket had a 64-bit x86 assembler available, I might even consider using it instead of Gambit.
Message: 1 Date: Mon, 5 Nov 2012 08:09:19 -0500 From: Sam Tobin-Hochstadt samth@ccs.neu.edu To: Matthew Flatt mflatt@cs.utah.edu Cc: "users@racket-lang.org" users@racket-lang.org Subject: Re: [racket] translate from Racket to Common Lisp Message-ID: CAK=HD+Yyr1R4K6emrgSXqie2c_AmT1BzhLpP1X=1NDXRgzsjGg@mail.gmail.com Content-Type: text/plain; charset=UTF-8
It's impressive to note the change in relative performance for Racket over past 3 years since you published the benchmarks on the blog -- Racket has gone from slower than Gambit on the majority of benchmarks, sometimes by a significant margin, to faster on most of them, and never more than 2x slower (except ctak).
For your information, it seems that the change in relative performance has more to do with the change in C compiler over the past 3 years than anything else. The benchmarks were run on Mac OS X with the C compiler available with Xcode. Recently Apple dropped distributing the GNU gcc compiler with Xcode, and the "gcc" program that is installed is actually a version which uses the gcc compiler parser but LLVM as a back-end (i.e. llvm-gcc). That C compiler has bugs which prevent using gcc's computed gotos which are critical for the best performance. This is what Gambit's "configure" script has to say when using llvm-gcc:
configure: ************************************************************************** *** *** *** The LLVM GCC compiler that is being used may have a bug in the *** *** compilation of label values (i.e. &&label). This bug was *** *** discovered when compiling Gambit with Xcode on Mac OS X (see bug *** *** ID# 9956714). For this reason the use of computed gotos in the *** *** generated code has been disabled. This greatly reduces the run *** *** time performance. The system runs 5 times slower than when it is *** *** compiled with GNU GCC. It is highly recommended that you use *** *** another C compiler if possible. *** *** *** **************************************************************************
I have suggested to Matthew that he rerun the benchmarks on a linux machine where GNU gcc is the default C compiler. It will be interesting to see how that affects performance.
Marc
Afficher les réponses par date
I downloaded 64-bit Racket and compared it to my install of 64-bit gambit:
[Media-Mac-mini-3:~/programs/gambit/bench] lucier% gsc -v v4.6.6 20120915144211 i386-apple-darwin10.8.0 "./configure 'CC=/pkgs/gcc-4.7.2/bin/gcc -march=core2 -fschedule-insns -frename-registers' '--enable-single-host' '--enable-multiple-versions'" [Media-Mac-mini-3:~/programs/gambit/bench] lucier% uname -a Darwin Media-Mac-mini-3.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
This is a Mac Mini with a 2.4GHz Intel Core 2 Duo and 8GB of ram.
So this is a Gambit with special gcc options that I find generally makes code run faster, at the expense of compile time. And it uses the latest GNU GCC.
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
I applied Matthew Flatt's patch from
https://www.cs.utah.edu/%7Emflatt/benchmarks-20100126/log1/bench.patch
to Gambit's bench script, and then ran "table" from Gambit's bench directory.
This gave me the following four html files that summarize the cpu and real times to execute the programs; compile times are not reported (just because the "table" script does not report them).
The meanings of the various table titles can be found here:
http://www.iro.umontreal.ca/~gambit/bench.html
I repeat:
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
The systems seem comparable running programs in R6RS-* semantics, except for programs using call-cc.
Brad
I removed the Racket users list from this CC.
Viewable html files containing the results can be found at
http://www.math.purdue.edu/~lucier/bench-gambit-2012-11-06/
Brad
What does r6rs really involve in these benchmarks, since Gambit is only supporting r5rs?
Álvaro
On Wed, Nov 7, 2012 at 8:13 PM, Bradley Lucier lucier@math.purdue.eduwrote:
** I removed the Racket users list from this CC.
Viewable html files containing the results can be found at
http://www.math.purdue.edu/~lucier/bench-gambit-2012-11-06/
Brad
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Wed, 2012-11-07 at 21:15 +0100, Álvaro Castro-Castilla wrote:
What does r6rs really involve in these benchmarks, since Gambit is only supporting r5rs?
As I said in my previous e-mail, see
http://www.iro.umontreal.ca/~gambit/bench.html
for an explanation of the headings.
Brad
I see. Thank you Brad.
On Wed, Nov 7, 2012 at 9:20 PM, Bradley Lucier lucier@math.purdue.eduwrote:
** On Wed, 2012-11-07 at 21:15 +0100, Álvaro Castro-Castilla wrote:
What does r6rs really involve in these benchmarks, since Gambit is only supporting r5rs?
As I said in my previous e-mail, see
http://www.iro.umontreal.ca/~gambit/bench.html
for an explanation of the headings.
Brad
Le 2012-11-07 à 2:13 PM, Bradley Lucier lucier@math.purdue.edu a écrit :
I removed the Racket users list from this CC.
Viewable html files containing the results can be found at
http://www.math.purdue.edu/~lucier/bench-gambit-2012-11-06/
Brad
Thank you Brad and Matthew for running the benchmarks. There are some discrepancies between your results (on 64 bits) which are worrisome. The discrepancies may be due to a difference in compiler options or declarations that one of you is using that the other is not using. For example, these are the numbers you reported (R/G is the execution time ratio between Racket and Gambit):
benchmark Brad says Matthew says
dynamic R/G = 1.33 R/G = 0.48 (that's almost a factor of 3 difference!)
ctak R/G = 63.67 R/G = 30.44
puzzle R/G = 2.60 R/G = 1.32
paraffins R/G = 1.89 R/G = 1.38
peval R/G = 1.49 R/G = 1.13
Given that Brad has a better knowledge of Gambit, and Matthew has a better knowledge of Racket, it wouldn't be surprizing that they used Gambit and Racket differently.
I wonder if these discrepancies are due to
- different assumptions (perhaps Brad used declarations which assume that global variables will not be mutated if they are not set!, which Matthew is not using)
- different hardware (perhaps memory access is relatively faster on one of these machines)
- different heap sizes
- different C compilers
It would be interesting to know where these differences come from, so that we know for future benchmarking experiments.
Marc
On Nov 7, 2012, at 3:54 PM, Marc Feeley wrote:
There are some discrepancies between your results (on 64 bits) which are worrisome.
I believe that in general, while the benchmarks have the same name, the source code for the benchmarks could be different. That's true for fft, for example.
Brad
Hi Brad: this is a 6-year old version of Racket you're using. FYI.
Robby
On Tue, Nov 6, 2012 at 3:50 PM, Bradley Lucier lucier@math.purdue.edu wrote:
I downloaded 64-bit Racket and compared it to my install of 64-bit gambit:
[Media-Mac-mini-3:~/programs/gambit/bench] lucier% gsc -v v4.6.6 20120915144211 i386-apple-darwin10.8.0 "./configure 'CC=/pkgs/gcc-4.7.2/bin/gcc -march=core2 -fschedule-insns -frename-registers' '--enable-single-host' '--enable-multiple-versions'" [Media-Mac-mini-3:~/programs/gambit/bench] lucier% uname -a Darwin Media-Mac-mini-3.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
This is a Mac Mini with a 2.4GHz Intel Core 2 Duo and 8GB of ram.
So this is a Gambit with special gcc options that I find generally makes code run faster, at the expense of compile time. And it uses the latest GNU GCC.
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
I applied Matthew Flatt's patch from
https://www.cs.utah.edu/%7Emflatt/benchmarks-20100126/log1/bench.patch
to Gambit's bench script, and then ran "table" from Gambit's bench directory.
This gave me the following four html files that summarize the cpu and real times to execute the programs; compile times are not reported (just because the "table" script does not report them).
The meanings of the various table titles can be found here:
http://www.iro.umontreal.ca/~gambit/bench.html
I repeat:
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
The systems seem comparable running programs in R6RS-* semantics, except for programs using call-cc.
Brad
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
I thought that too, but I think the URL (bench.html) with the 6-year old version was just for the meaning of the columns. The new results were attached to the email, and presumably he used the most recent one.
Jay
On Wed, Nov 7, 2012 at 1:28 PM, Robby Findler robby@eecs.northwestern.edu wrote:
Hi Brad: this is a 6-year old version of Racket you're using. FYI.
Robby
On Tue, Nov 6, 2012 at 3:50 PM, Bradley Lucier lucier@math.purdue.edu wrote:
I downloaded 64-bit Racket and compared it to my install of 64-bit gambit:
[Media-Mac-mini-3:~/programs/gambit/bench] lucier% gsc -v v4.6.6 20120915144211 i386-apple-darwin10.8.0 "./configure 'CC=/pkgs/gcc-4.7.2/bin/gcc -march=core2 -fschedule-insns -frename-registers' '--enable-single-host' '--enable-multiple-versions'" [Media-Mac-mini-3:~/programs/gambit/bench] lucier% uname -a Darwin Media-Mac-mini-3.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
This is a Mac Mini with a 2.4GHz Intel Core 2 Duo and 8GB of ram.
So this is a Gambit with special gcc options that I find generally makes code run faster, at the expense of compile time. And it uses the latest GNU GCC.
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
I applied Matthew Flatt's patch from
https://www.cs.utah.edu/%7Emflatt/benchmarks-20100126/log1/bench.patch
to Gambit's bench script, and then ran "table" from Gambit's bench directory.
This gave me the following four html files that summarize the cpu and real times to execute the programs; compile times are not reported (just because the "table" script does not report them).
The meanings of the various table titles can be found here:
http://www.iro.umontreal.ca/~gambit/bench.html
I repeat:
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
The systems seem comparable running programs in R6RS-* semantics, except for programs using call-cc.
Brad
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Racket Users list: http://lists.racket-lang.org/users
Oh, yes indeed. My apologies for the confusion!!
Robby
On Wed, Nov 7, 2012 at 2:40 PM, Jay McCarthy jay.mccarthy@gmail.com wrote:
I thought that too, but I think the URL (bench.html) with the 6-year old version was just for the meaning of the columns. The new results were attached to the email, and presumably he used the most recent one.
Jay
On Wed, Nov 7, 2012 at 1:28 PM, Robby Findler robby@eecs.northwestern.edu wrote:
Hi Brad: this is a 6-year old version of Racket you're using. FYI.
Robby
On Tue, Nov 6, 2012 at 3:50 PM, Bradley Lucier lucier@math.purdue.edu wrote:
I downloaded 64-bit Racket and compared it to my install of 64-bit gambit:
[Media-Mac-mini-3:~/programs/gambit/bench] lucier% gsc -v v4.6.6 20120915144211 i386-apple-darwin10.8.0 "./configure 'CC=/pkgs/gcc-4.7.2/bin/gcc -march=core2 -fschedule-insns -frename-registers' '--enable-single-host' '--enable-multiple-versions'" [Media-Mac-mini-3:~/programs/gambit/bench] lucier% uname -a Darwin Media-Mac-mini-3.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
This is a Mac Mini with a 2.4GHz Intel Core 2 Duo and 8GB of ram.
So this is a Gambit with special gcc options that I find generally makes code run faster, at the expense of compile time. And it uses the latest GNU GCC.
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
I applied Matthew Flatt's patch from
https://www.cs.utah.edu/%7Emflatt/benchmarks-20100126/log1/bench.patch
to Gambit's bench script, and then ran "table" from Gambit's bench directory.
This gave me the following four html files that summarize the cpu and real times to execute the programs; compile times are not reported (just because the "table" script does not report them).
The meanings of the various table titles can be found here:
http://www.iro.umontreal.ca/~gambit/bench.html
I repeat:
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
The systems seem comparable running programs in R6RS-* semantics, except for programs using call-cc.
Brad
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Racket Users list: http://lists.racket-lang.org/users
-- Jay McCarthy jay@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay
"The glory of God is Intelligence" - D&C 93
I put files that open in web browsers of my benchmark results at
http://www.math.purdue.edu/~lucier/bench-gambit-2012-11-06/
with a readme that's basically my original e-mail with the version that mzscheme reports (5.3) added.
I looked at the sources for fft.scm in the Gambit benchmark suite and in the Racket benchmark suite, and they are different, so it's not clear that the two sets of results can be compared directly.
Brad
-
Bradley Lucier -
Jay McCarthy -
Marc Feeley -
Robby Findler -
Álvaro Castro-Castilla