Re: [gambit-list] [racket] translate from Racket to Common Lisp
Le 2012-11-06 à 3:50 AM, Hugh Aguilar <hughaguilar96@yahoo.com> a écrit :
I am very impressed that Racket is as fast as it is. I had expected the compilers such as Gambit to be much faster than the VM-JIT system. If Racket had a 64-bit x86 assembler available, I might even consider using it instead of Gambit.
Message: 1 Date: Mon, 5 Nov 2012 08:09:19 -0500 From: Sam Tobin-Hochstadt <samth@ccs.neu.edu> To: Matthew Flatt <mflatt@cs.utah.edu> Cc: "users@racket-lang.org" <users@racket-lang.org> Subject: Re: [racket] translate from Racket to Common Lisp Message-ID: <CAK=HD+Yyr1R4K6emrgSXqie2c_AmT1BzhLpP1X=1NDXRgzsjGg@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
It's impressive to note the change in relative performance for Racket over past 3 years since you published the benchmarks on the blog -- Racket has gone from slower than Gambit on the majority of benchmarks, sometimes by a significant margin, to faster on most of them, and never more than 2x slower (except ctak).
For your information, it seems that the change in relative performance has more to do with the change in C compiler over the past 3 years than anything else. The benchmarks were run on Mac OS X with the C compiler available with Xcode. Recently Apple dropped distributing the GNU gcc compiler with Xcode, and the "gcc" program that is installed is actually a version which uses the gcc compiler parser but LLVM as a back-end (i.e. llvm-gcc). That C compiler has bugs which prevent using gcc's computed gotos which are critical for the best performance. This is what Gambit's "configure" script has to say when using llvm-gcc: configure: ************************************************************************** *** *** *** The LLVM GCC compiler that is being used may have a bug in the *** *** compilation of label values (i.e. &&label). This bug was *** *** discovered when compiling Gambit with Xcode on Mac OS X (see bug *** *** ID# 9956714). For this reason the use of computed gotos in the *** *** generated code has been disabled. This greatly reduces the run *** *** time performance. The system runs 5 times slower than when it is *** *** compiled with GNU GCC. It is highly recommended that you use *** *** another C compiler if possible. *** *** *** ************************************************************************** I have suggested to Matthew that he rerun the benchmarks on a linux machine where GNU gcc is the default C compiler. It will be interesting to see how that affects performance. Marc
Afficher les réponses par date
I downloaded 64-bit Racket and compared it to my install of 64-bit gambit: [Media-Mac-mini-3:~/programs/gambit/bench] lucier% gsc -v v4.6.6 20120915144211 i386-apple-darwin10.8.0 "./configure 'CC=/pkgs/gcc-4.7.2/bin/gcc -march=core2 -fschedule-insns -frename-registers' '--enable-single-host' '--enable-multiple-versions'" [Media-Mac-mini-3:~/programs/gambit/bench] lucier% uname -a Darwin Media-Mac-mini-3.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386 This is a Mac Mini with a 2.4GHz Intel Core 2 Duo and 8GB of ram. So this is a Gambit with special gcc options that I find generally makes code run faster, at the expense of compile time. And it uses the latest GNU GCC. So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples. I applied Matthew Flatt's patch from https://www.cs.utah.edu/%7Emflatt/benchmarks-20100126/log1/bench.patch to Gambit's bench script, and then ran "table" from Gambit's bench directory. This gave me the following four html files that summarize the cpu and real times to execute the programs; compile times are not reported (just because the "table" script does not report them). The meanings of the various table titles can be found here: http://www.iro.umontreal.ca/~gambit/bench.html I repeat: So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples. The systems seem comparable running programs in R6RS-* semantics, except for programs using call-cc. Brad
I removed the Racket users list from this CC. Viewable html files containing the results can be found at http://www.math.purdue.edu/~lucier/bench-gambit-2012-11-06/ Brad
What does r6rs really involve in these benchmarks, since Gambit is only supporting r5rs? Álvaro On Wed, Nov 7, 2012 at 8:13 PM, Bradley Lucier <lucier@math.purdue.edu>wrote:
** I removed the Racket users list from this CC.
Viewable html files containing the results can be found at
http://www.math.purdue.edu/~lucier/bench-gambit-2012-11-06/
Brad
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Wed, 2012-11-07 at 21:15 +0100, Álvaro Castro-Castilla wrote:
What does r6rs really involve in these benchmarks, since Gambit is only supporting r5rs?
As I said in my previous e-mail, see http://www.iro.umontreal.ca/~gambit/bench.html for an explanation of the headings. Brad
I see. Thank you Brad. On Wed, Nov 7, 2012 at 9:20 PM, Bradley Lucier <lucier@math.purdue.edu>wrote:
** On Wed, 2012-11-07 at 21:15 +0100, Álvaro Castro-Castilla wrote:
What does r6rs really involve in these benchmarks, since Gambit is only supporting r5rs?
As I said in my previous e-mail, see
http://www.iro.umontreal.ca/~gambit/bench.html
for an explanation of the headings.
Brad
Le 2012-11-07 à 2:13 PM, Bradley Lucier <lucier@math.purdue.edu> a écrit :
I removed the Racket users list from this CC.
Viewable html files containing the results can be found at
http://www.math.purdue.edu/~lucier/bench-gambit-2012-11-06/
Brad
Thank you Brad and Matthew for running the benchmarks. There are some discrepancies between your results (on 64 bits) which are worrisome. The discrepancies may be due to a difference in compiler options or declarations that one of you is using that the other is not using. For example, these are the numbers you reported (R/G is the execution time ratio between Racket and Gambit): benchmark Brad says Matthew says dynamic R/G = 1.33 R/G = 0.48 (that's almost a factor of 3 difference!) ctak R/G = 63.67 R/G = 30.44 puzzle R/G = 2.60 R/G = 1.32 paraffins R/G = 1.89 R/G = 1.38 peval R/G = 1.49 R/G = 1.13 Given that Brad has a better knowledge of Gambit, and Matthew has a better knowledge of Racket, it wouldn't be surprizing that they used Gambit and Racket differently. I wonder if these discrepancies are due to - different assumptions (perhaps Brad used declarations which assume that global variables will not be mutated if they are not set!, which Matthew is not using) - different hardware (perhaps memory access is relatively faster on one of these machines) - different heap sizes - different C compilers It would be interesting to know where these differences come from, so that we know for future benchmarking experiments. Marc
On Nov 7, 2012, at 3:54 PM, Marc Feeley wrote:
There are some discrepancies between your results (on 64 bits) which are worrisome.
I believe that in general, while the benchmarks have the same name, the source code for the benchmarks could be different. That's true for fft, for example. Brad
Hi Brad: this is a 6-year old version of Racket you're using. FYI. Robby On Tue, Nov 6, 2012 at 3:50 PM, Bradley Lucier <lucier@math.purdue.edu> wrote:
I downloaded 64-bit Racket and compared it to my install of 64-bit gambit:
[Media-Mac-mini-3:~/programs/gambit/bench] lucier% gsc -v v4.6.6 20120915144211 i386-apple-darwin10.8.0 "./configure 'CC=/pkgs/gcc-4.7.2/bin/gcc -march=core2 -fschedule-insns -frename-registers' '--enable-single-host' '--enable-multiple-versions'" [Media-Mac-mini-3:~/programs/gambit/bench] lucier% uname -a Darwin Media-Mac-mini-3.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
This is a Mac Mini with a 2.4GHz Intel Core 2 Duo and 8GB of ram.
So this is a Gambit with special gcc options that I find generally makes code run faster, at the expense of compile time. And it uses the latest GNU GCC.
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
I applied Matthew Flatt's patch from
https://www.cs.utah.edu/%7Emflatt/benchmarks-20100126/log1/bench.patch
to Gambit's bench script, and then ran "table" from Gambit's bench directory.
This gave me the following four html files that summarize the cpu and real times to execute the programs; compile times are not reported (just because the "table" script does not report them).
The meanings of the various table titles can be found here:
http://www.iro.umontreal.ca/~gambit/bench.html
I repeat:
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
The systems seem comparable running programs in R6RS-* semantics, except for programs using call-cc.
Brad
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
I thought that too, but I think the URL (bench.html) with the 6-year old version was just for the meaning of the columns. The new results were attached to the email, and presumably he used the most recent one. Jay On Wed, Nov 7, 2012 at 1:28 PM, Robby Findler <robby@eecs.northwestern.edu> wrote:
Hi Brad: this is a 6-year old version of Racket you're using. FYI.
Robby
On Tue, Nov 6, 2012 at 3:50 PM, Bradley Lucier <lucier@math.purdue.edu> wrote:
I downloaded 64-bit Racket and compared it to my install of 64-bit gambit:
[Media-Mac-mini-3:~/programs/gambit/bench] lucier% gsc -v v4.6.6 20120915144211 i386-apple-darwin10.8.0 "./configure 'CC=/pkgs/gcc-4.7.2/bin/gcc -march=core2 -fschedule-insns -frename-registers' '--enable-single-host' '--enable-multiple-versions'" [Media-Mac-mini-3:~/programs/gambit/bench] lucier% uname -a Darwin Media-Mac-mini-3.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
This is a Mac Mini with a 2.4GHz Intel Core 2 Duo and 8GB of ram.
So this is a Gambit with special gcc options that I find generally makes code run faster, at the expense of compile time. And it uses the latest GNU GCC.
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
I applied Matthew Flatt's patch from
https://www.cs.utah.edu/%7Emflatt/benchmarks-20100126/log1/bench.patch
to Gambit's bench script, and then ran "table" from Gambit's bench directory.
This gave me the following four html files that summarize the cpu and real times to execute the programs; compile times are not reported (just because the "table" script does not report them).
The meanings of the various table titles can be found here:
http://www.iro.umontreal.ca/~gambit/bench.html
I repeat:
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
The systems seem comparable running programs in R6RS-* semantics, except for programs using call-cc.
Brad
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
____________________ Racket Users list: http://lists.racket-lang.org/users
-- Jay McCarthy <jay@cs.byu.edu> Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay "The glory of God is Intelligence" - D&C 93
Oh, yes indeed. My apologies for the confusion!! Robby On Wed, Nov 7, 2012 at 2:40 PM, Jay McCarthy <jay.mccarthy@gmail.com> wrote:
I thought that too, but I think the URL (bench.html) with the 6-year old version was just for the meaning of the columns. The new results were attached to the email, and presumably he used the most recent one.
Jay
On Wed, Nov 7, 2012 at 1:28 PM, Robby Findler <robby@eecs.northwestern.edu> wrote:
Hi Brad: this is a 6-year old version of Racket you're using. FYI.
Robby
On Tue, Nov 6, 2012 at 3:50 PM, Bradley Lucier <lucier@math.purdue.edu> wrote:
I downloaded 64-bit Racket and compared it to my install of 64-bit gambit:
[Media-Mac-mini-3:~/programs/gambit/bench] lucier% gsc -v v4.6.6 20120915144211 i386-apple-darwin10.8.0 "./configure 'CC=/pkgs/gcc-4.7.2/bin/gcc -march=core2 -fschedule-insns -frename-registers' '--enable-single-host' '--enable-multiple-versions'" [Media-Mac-mini-3:~/programs/gambit/bench] lucier% uname -a Darwin Media-Mac-mini-3.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
This is a Mac Mini with a 2.4GHz Intel Core 2 Duo and 8GB of ram.
So this is a Gambit with special gcc options that I find generally makes code run faster, at the expense of compile time. And it uses the latest GNU GCC.
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
I applied Matthew Flatt's patch from
https://www.cs.utah.edu/%7Emflatt/benchmarks-20100126/log1/bench.patch
to Gambit's bench script, and then ran "table" from Gambit's bench directory.
This gave me the following four html files that summarize the cpu and real times to execute the programs; compile times are not reported (just because the "table" script does not report them).
The meanings of the various table titles can be found here:
http://www.iro.umontreal.ca/~gambit/bench.html
I repeat:
So this is not comparing stock gambit with stock mzscheme; it is not comparing apples to apples.
The systems seem comparable running programs in R6RS-* semantics, except for programs using call-cc.
Brad
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
____________________ Racket Users list: http://lists.racket-lang.org/users
-- Jay McCarthy <jay@cs.byu.edu> Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay
"The glory of God is Intelligence" - D&C 93
I put files that open in web browsers of my benchmark results at http://www.math.purdue.edu/~lucier/bench-gambit-2012-11-06/ with a readme that's basically my original e-mail with the version that mzscheme reports (5.3) added. I looked at the sources for fft.scm in the Gambit benchmark suite and in the Racket benchmark suite, and they are different, so it's not clear that the two sets of results can be compared directly. Brad
participants (5)
-
Bradley Lucier -
Jay McCarthy -
Marc Feeley -
Robby Findler -
Álvaro Castro-Castilla