Re: [gambit-list] Fwd: Gambit-C vs. Bigloo vs. MzScheme
On Sat, 26 Aug 2006 21:31:16 -0400 Marc Feeley <feeley@iro.umontreal.ca> wrote:
Désolé j'ai oublié de te mettre en CC.
Marc
Begin forwarded message:
From: Marc Feeley <feeley@iro.umontreal.ca> Date: August 26, 2006 9:28:15 PM EDT (CA) To: Gambit List <Gambit-list@iro.umontreal.ca> Cc: Eli Barzilay <eli@barzilay.org>, Matthew Flatt <mflatt@cs.utah.edu> Subject: Gambit-C vs. Bigloo vs. MzScheme
After I saw Eli Barzilay's post to comp.lang.scheme about the performance of MzScheme compared to other Scheme systems (http:// groups.google.com/group/comp.lang.scheme/msg/473f91e7065b6f73? dmode=source) I was tempted to do some benchmarking of my own using the Gambit Scheme benchmark suite which goes beyond the classical Gabriel benchmarks (it has close to 60 Scheme benchmarks). Most benchmarks are straightforward R4RS code and so should work on just about any Scheme system.
I have updated the benchmarking infrastructure to accomodate other Scheme systems, and MzScheme and Bigloo in particular. I also wrote a script to run all benchmarks and automatically generate an HTML table of the results. The table can be viewed here:
http://www.iro.umontreal.ca/~gambit/bench.html
And the benchmark suite is available here:
http://www.iro.umontreal.ca/~gambit/bench.tar.gz
As you can see from the table, Gambit-C is faster than Bigloo and MzScheme on roughly 3 out of 4 benchmarks. I have used the same assumptions as Eli, which approximate the semantics of R6RS (see the last section of the result page for details).
It would be really interesting to extend this experiment to other Scheme systems. It should not be too difficult to modify the benchmarking scripts to test other systems. The hardest parts are installing the Scheme system, figuring out which compiler options are equivalent to the ones used by the other systems (i.e. approximate R6RS semantics), and work out unexpected problems.
The following systems are particularly interesting to compare to:
- Chicken - Scheme 48 - Larceny - Chez Scheme
Are there any volunteers?
Marc
Hello Marc, Ah, benchmarking... Definitely we can't escape benchmarks :-). I had more or less sweared not to benchmark any more because this is an activity that can consume all your time and for which the result is not necessarily very useful. However, because I didn't want to spend a bad Sunday :-) I have downloaded your benchmarks and make some measurements with Bigloo. Here is what I got: -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|----- General command line: bigloo -O6 OFP=-copt -fomit-frame-pointer fixnum= /+/+fx/, /-/-fx/, ... unsafe=bigloo -O6 -unsafe FLOAT= re-implementing FLOAT+, FLOAT-, ... ====================================================================== tak1: Marc's time: real: 11.22s sys: 0.02s user: 11.13s fixnum: real: 3.61s sys: 0.01s user: 3.57s fixnum, omit-frame-pointer: real: 2.80s sys: 0.01s user: 2.78s ====================================================================== fft: Marc's time: real: 11.71s sys: 0.03s user: 11.32s FLOAT fixed: real: 4.39s sys: 0.02s user: 4.10s FLOAT fixed, unsafe real: 3.44s sys: 0.01s user: 3.33s FLOAT fixed, unsafe, $OFP real: 3.02s sys: 0.01s user: 2.92s ====================================================================== pnpoly: Marc's time: real: 16.39s sys: 0.02s user: 15.79s FLOAT: real: 15.81s sys: 0.02s user: 14.50s FLOAT, unsafe: real: 1.86s sys: 0.01s user: 1.82s FLOAT, unsafe, $OFP: real: 1.49s sys: 0.00s user: 1.47s ====================================================================== sum: Marc's time: real: 5.44s sys: 0.01s user: 5.36s FLOAT, unsafe, $OFP: real: 5.29s sys: 0.01s user: 5.22s FIXNUM, unsafe, $OFP: real: 0.29s sys: 0.00s user: 0.27s ====================================================================== sumfp: Marc's time: real: 9.76s sys: 0.01s user: 9.57s FLOAT: real: 0.55s sys: 0.01s user: 0.52s FLOAT, unsafe, $OFP: real: 0.54s sys: 0.00s user: 0.52s -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|----- As you can notice, changing the compilation options and improving your prefix-bigloo.scm file has a huge impact. As you can notice some benchmarks now go 18 times faster! Here is a description of the modiication I have applied: 1- I have re-implemented your macros FLOATxxx that were badling implemented for Bigloo. They used to expanse into call to the generic functions. They now expanse to the floating point operator. The improvement for benchmark using floating applications is in between 2 and 5 times faster. 2- I have used the proper benchmarking compilation options. a- The produced C code must be compiled with -fomit-frame-pointer (when using GCC). b- The option -unsafe has to be used. 3- For fixnum benchmarks, I have replaced the call to the generic functions with calls to fixnum functions. Please, find enclosed the new version of prefix-bigloo.scm. I would appreciate if you could re-execute the Bigloo benchmarks with the new prefix and update your web pages with the new Bigloo results. Thanks in advance. -- Manuel -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|----- (module prefix (main main-entry) (extern (include "sys/times.h")) (extern (include "time.h")) (option (set! *unsafe-rgc* #t)) (option (set! *unsafe-library* #t)) (option (set! *unsafe-arity* #t)) (option (set! *unsafe-type* #t)) (option (set! *unsafe-struct* #t)) (option (set! *unsafe-range* #t)) (option (set! *unsafe-version* #t)) (option (set! *cc-options* (string-append "-fomit-frame-pointer " *cc-options*)))) (define (clock-to-msecs x) (quotient (* x 1000) (pragma::int "CLOCKS_PER_SEC"))) (define (process-times) (let () (pragma "struct tms buf") (let ((real::long (pragma::long "times(&buf)"))) (cons real (+ (pragma::long "buf.tms_utime") (pragma::long "buf.tms_stime")))))) (define (time* thunk) (let ((start (process-times))) (let ((result (thunk))) (let ((end (process-times))) (let ((cpu (clock-to-msecs (- (cdr end) (cdr start)))) (real (clock-to-msecs (- (car end) (car start))))) (display "cpu time: ") (display cpu) (display " real time: ") (display real) (newline) result))))) (define (run-benchmark name count ok? run-maker . args) (let* ((run (apply run-maker args)) (result (time* (lambda () (run-bench name count ok? run))))) (if (not (ok? result)) (begin (display "*** wrong result ***") (newline) (display "*** got: ") (write result) (newline))))) (define (run-bench name count ok? run) (let loop ((i 0) (result (list 'undefined))) (if (< i count) (loop (+ i 1) (run)) result))) (define (fatal-error . args) (for-each display args) (newline) (exit 1)) (define (main-entry args) (main)) ;------------------------------------------------------------------------------ ; Macros... (define-macro (def-macro form . body) `(define-macro ,form (let () ,@body))) ; Flonum arithmetic. (begin (def-macro (FLOATvector-const . lst) `',(list->vector lst)) (def-macro (FLOATvector? x) `(vector? ,x)) (def-macro (FLOATvector . lst) `(vector ,@lst)) (def-macro (FLOATmake-vector n . init) `(make-vector ,n ,@init)) (def-macro (FLOATvector-ref v i) `(vector-ref ,v ,i)) (def-macro (FLOATvector-set! v i x) `(vector-set! ,v ,i ,x)) (def-macro (FLOATvector-length v) `(vector-length ,v)) (def-macro (nuc-const . lst) `',(list->vector lst)) ) (begin (def-macro (+ . lst) (if (null? (cdr lst)) (car lst) `(+fx ,(car lst) (+ ,@(cdr lst))))) (def-macro (- . lst) (if (null? (cdr lst)) (car lst) `(-fx ,(car lst) (- ,@(cdr lst))))) (def-macro (* . lst) (if (null? (cdr lst)) (car lst) `(*fx ,(car lst) (* ,@(cdr lst))))) (def-macro (/ . lst) (if (null? (cdr lst)) (car lst) `(/fx ,(car lst) (/ ,@(cdr lst))))) (def-macro (FLOAT+ . lst) (if (null? (cdr lst)) (car lst) `(+fl ,(car lst) (FLOAT+ ,@(cdr lst))))) (def-macro (FLOAT- . lst) (if (null? (cdr lst)) (car lst) `(-fl ,(car lst) (FLOAT- ,@(cdr lst))))) (def-macro (FLOAT* . lst) (if (null? (cdr lst)) (car lst) `(*fl ,(car lst) (FLOAT* ,@(cdr lst))))) (def-macro (FLOAT/ . lst) (if (null? (cdr lst)) (car lst) `(/fl ,(car lst) (FLOAT/ ,@(cdr lst))))) (def-macro (FLOAT= a b) `(=fl ,a ,b)) (def-macro (FLOAT< a b) `(<fl ,a ,b)) (def-macro (FLOAT<= a b) `(<= ,a ,b)) (def-macro (FLOAT> a b) `(> ,a ,b)) (def-macro (FLOAT>= a b) `(>= ,a ,b)) (def-macro (FLOATnegative? a) `(negativefl? ,a)) (def-macro (FLOATpositive? a) `(positivefl? ,a)) (def-macro (FLOATzero? a) `(zerofl? ,a)) (def-macro (FLOATabs a) `(abs ,a)) (def-macro (FLOATsin a) `(sin ,a)) (def-macro (FLOATcos a) `(cos ,a)) (def-macro (FLOATatan a) `(atan ,a)) (def-macro (FLOATsqrt a) `(sqrt ,a)) (def-macro (FLOATmin a) `(min ,a)) (def-macro (FLOATmax a) `(max ,a)) (def-macro (FLOATround a) `(round ,a)) (def-macro (FLOATinexact->exact a) `(inexact->exact ,a)) ) ; Generic arithmetic. (begin (def-macro (GENERIC+ . lst) `(+ ,@lst)) (def-macro (GENERIC- . lst) `(- ,@lst)) (def-macro (GENERIC* . lst) `(* ,@lst)) (def-macro (GENERIC/ . lst) `(/ ,@lst)) (def-macro (GENERICquotient . lst) `(quotient ,@lst)) (def-macro (GENERICremainder . lst) `(remainder ,@lst)) (def-macro (GENERICmodulo . lst) `(modulo ,@lst)) (def-macro (GENERIC= . lst) `(= ,@lst)) (def-macro (GENERIC< . lst) `(< ,@lst)) (def-macro (GENERIC<= . lst) `(<= ,@lst)) (def-macro (GENERIC> . lst) `(> ,@lst)) (def-macro (GENERIC>= . lst) `(>= ,@lst)) (def-macro (GENERICexpt . lst) `(expt ,@lst)) ) (begin (def-macro (bitwise-or x y) `(bit-or ,x ,y)) (def-macro (bitwise-and x y) `(bit-and ,x ,y)) (def-macro (bitwise-not x) `(bit-not ,x)) ) -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|----- ps: benchmarks using vector of float could be improved by using Bigloo typed vectors. I have not tried to apply this because as I said, I don't want to allocate to much time to benchmarking. However, if you want accurate Bigloo measurement. You should try to add the following to prefix-bigloo.scm: (module prefix (main main-entry) (extern (include "sys/times.h")) (extern (include "time.h")) ... (type (tvector f64vector (double)))) (def-macro (FLOATvector? x) `(f64vector? ,x)) (def-macro (FLOATvector-ref v i) `(f64vector-ref ,v ,i)) ...
Manuel thank you for taking the time to look into the benchmarks and determining what would produce the fastest execution with Bigloo. By the way, wouldn't it also make sense to specialize =, <, <= and other operators to fixnums? As you discovered, the benchmark suite was written so that it is easy to specialize the arithmetic operators, for example, wherever the benchmarks call FLOAT+ you can either call plain + or the flonum addition and where the benchmarks call + you can either call plain + or the fixnum addition (i.e. when there is no prefix on the name, the fixnum operators will work). With appropriate macro definitions in the system specific prefix file you can force particular specializations. The code you sent for Bigloo will be useful for experiments where the code is compiled in "unsafe" mode with no overflow detection (i.e. which models the setting where the user is willing to have the fastest possible code at all costs, possibly crashing the program if there is any bug in the code). This was not the setting Eli and I were interested in benchmarking. Eli and I are trying to model the (expected) semantics of R6RS Scheme: - safe execution (i.e. an exception has to be signalled on errors) - overflow detection for fixnum arithmetic (either produce a bignum or signal an exception) - immutable bindings for the definitions in the benchmark (as if in an R6RS module) - immutable predefined bindings That is why I did not specialize the arithmetic operators in the prefix files for Bigloo (and Gambit and MzScheme). Correct me if I'm wrong but the only improvement in this setting is the addition of the -fomit-frame-pointer option, i.e. using this in the "bench" script: bigloo -O6 $usecc -copt -O3 -copt -fomit-frame-pointer $1.scm -o $1 What would I have to add to detect fixnum arithmetic overflows? Would adding "-copt -ftrapv" make sense? Does Bigloo support bignums? Marc On 27-Aug-06, at 3:49 AM, Manuel Serrano wrote:
On Sat, 26 Aug 2006 21:31:16 -0400 Marc Feeley <feeley@iro.umontreal.ca> wrote:
Désolé j'ai oublié de te mettre en CC.
Marc
Begin forwarded message:
From: Marc Feeley <feeley@iro.umontreal.ca> Date: August 26, 2006 9:28:15 PM EDT (CA) To: Gambit List <Gambit-list@iro.umontreal.ca> Cc: Eli Barzilay <eli@barzilay.org>, Matthew Flatt <mflatt@cs.utah.edu> Subject: Gambit-C vs. Bigloo vs. MzScheme
After I saw Eli Barzilay's post to comp.lang.scheme about the performance of MzScheme compared to other Scheme systems (http:// groups.google.com/group/comp.lang.scheme/msg/473f91e7065b6f73? dmode=source) I was tempted to do some benchmarking of my own using the Gambit Scheme benchmark suite which goes beyond the classical Gabriel benchmarks (it has close to 60 Scheme benchmarks). Most benchmarks are straightforward R4RS code and so should work on just about any Scheme system.
I have updated the benchmarking infrastructure to accomodate other Scheme systems, and MzScheme and Bigloo in particular. I also wrote a script to run all benchmarks and automatically generate an HTML table of the results. The table can be viewed here:
http://www.iro.umontreal.ca/~gambit/bench.html
And the benchmark suite is available here:
http://www.iro.umontreal.ca/~gambit/bench.tar.gz
As you can see from the table, Gambit-C is faster than Bigloo and MzScheme on roughly 3 out of 4 benchmarks. I have used the same assumptions as Eli, which approximate the semantics of R6RS (see the last section of the result page for details).
It would be really interesting to extend this experiment to other Scheme systems. It should not be too difficult to modify the benchmarking scripts to test other systems. The hardest parts are installing the Scheme system, figuring out which compiler options are equivalent to the ones used by the other systems (i.e. approximate R6RS semantics), and work out unexpected problems.
The following systems are particularly interesting to compare to:
- Chicken - Scheme 48 - Larceny - Chez Scheme
Are there any volunteers?
Marc
Hello Marc,
Ah, benchmarking... Definitely we can't escape benchmarks :-). I had more or less sweared not to benchmark any more because this is an activity that can consume all your time and for which the result is not necessarily very useful. However, because I didn't want to spend a bad Sunday :-) I have downloaded your benchmarks and make some measurements with Bigloo. Here is what I got:
-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|---- -|----- General command line: bigloo -O6
OFP=-copt -fomit-frame-pointer fixnum= /+/+fx/, /-/-fx/, ... unsafe=bigloo -O6 -unsafe FLOAT= re-implementing FLOAT+, FLOAT-, ...
====================================================================== tak1:
Marc's time: real: 11.22s sys: 0.02s user: 11.13s fixnum: real: 3.61s sys: 0.01s user: 3.57s fixnum, omit-frame-pointer: real: 2.80s sys: 0.01s user: 2.78s
====================================================================== fft:
Marc's time: real: 11.71s sys: 0.03s user: 11.32s FLOAT fixed: real: 4.39s sys: 0.02s user: 4.10s FLOAT fixed, unsafe real: 3.44s sys: 0.01s user: 3.33s FLOAT fixed, unsafe, $OFP real: 3.02s sys: 0.01s user: 2.92s
====================================================================== pnpoly:
Marc's time: real: 16.39s sys: 0.02s user: 15.79s FLOAT: real: 15.81s sys: 0.02s user: 14.50s FLOAT, unsafe: real: 1.86s sys: 0.01s user: 1.82s FLOAT, unsafe, $OFP: real: 1.49s sys: 0.00s user: 1.47s
====================================================================== sum:
Marc's time: real: 5.44s sys: 0.01s user: 5.36s FLOAT, unsafe, $OFP: real: 5.29s sys: 0.01s user: 5.22s FIXNUM, unsafe, $OFP: real: 0.29s sys: 0.00s user: 0.27s
====================================================================== sumfp:
Marc's time: real: 9.76s sys: 0.01s user: 9.57s FLOAT: real: 0.55s sys: 0.01s user: 0.52s FLOAT, unsafe, $OFP: real: 0.54s sys: 0.00s user: 0.52s -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|---- -|-----
As you can notice, changing the compilation options and improving your prefix-bigloo.scm file has a huge impact. As you can notice some benchmarks now go 18 times faster!
Here is a description of the modiication I have applied:
1- I have re-implemented your macros FLOATxxx that were badling implemented for Bigloo. They used to expanse into call to the generic functions. They now expanse to the floating point operator. The improvement for benchmark using floating applications is in between 2 and 5 times faster.
2- I have used the proper benchmarking compilation options. a- The produced C code must be compiled with -fomit-frame-pointer (when using GCC). b- The option -unsafe has to be used.
3- For fixnum benchmarks, I have replaced the call to the generic functions with calls to fixnum functions.
Please, find enclosed the new version of prefix-bigloo.scm. I would appreciate if you could re-execute the Bigloo benchmarks with the new prefix and update your web pages with the new Bigloo results. Thanks in advance.
-- Manuel
-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|---- -|----- (module prefix (main main-entry) (extern (include "sys/times.h")) (extern (include "time.h")) (option (set! *unsafe-rgc* #t)) (option (set! *unsafe-library* #t)) (option (set! *unsafe-arity* #t)) (option (set! *unsafe-type* #t)) (option (set! *unsafe-struct* #t)) (option (set! *unsafe-range* #t)) (option (set! *unsafe-version* #t)) (option (set! *cc-options* (string-append "-fomit-frame-pointer " *cc-options*))))
(define (clock-to-msecs x) (quotient (* x 1000) (pragma::int "CLOCKS_PER_SEC")))
(define (process-times) (let () (pragma "struct tms buf") (let ((real::long (pragma::long "times(&buf)"))) (cons real (+ (pragma::long "buf.tms_utime") (pragma::long "buf.tms_stime"))))))
(define (time* thunk) (let ((start (process-times))) (let ((result (thunk))) (let ((end (process-times))) (let ((cpu (clock-to-msecs (- (cdr end) (cdr start)))) (real (clock-to-msecs (- (car end) (car start))))) (display "cpu time: ") (display cpu) (display " real time: ") (display real) (newline) result)))))
(define (run-benchmark name count ok? run-maker . args) (let* ((run (apply run-maker args)) (result (time* (lambda () (run-bench name count ok? run))))) (if (not (ok? result)) (begin (display "*** wrong result ***") (newline) (display "*** got: ") (write result) (newline)))))
(define (run-bench name count ok? run) (let loop ((i 0) (result (list 'undefined))) (if (< i count) (loop (+ i 1) (run)) result)))
(define (fatal-error . args) (for-each display args) (newline) (exit 1))
(define (main-entry args) (main))
;--------------------------------------------------------------------- ---------
; Macros...
(define-macro (def-macro form . body) `(define-macro ,form (let () ,@body)))
; Flonum arithmetic.
(begin (def-macro (FLOATvector-const . lst) `',(list->vector lst)) (def-macro (FLOATvector? x) `(vector? ,x)) (def-macro (FLOATvector . lst) `(vector ,@lst)) (def-macro (FLOATmake-vector n . init) `(make-vector ,n ,@init)) (def-macro (FLOATvector-ref v i) `(vector-ref ,v ,i)) (def-macro (FLOATvector-set! v i x) `(vector-set! ,v ,i ,x)) (def-macro (FLOATvector-length v) `(vector-length ,v))
(def-macro (nuc-const . lst) `',(list->vector lst)) )
(begin (def-macro (+ . lst) (if (null? (cdr lst)) (car lst) `(+fx ,(car lst) (+ ,@(cdr lst))))) (def-macro (- . lst) (if (null? (cdr lst)) (car lst) `(-fx ,(car lst) (- ,@(cdr lst))))) (def-macro (* . lst) (if (null? (cdr lst)) (car lst) `(*fx ,(car lst) (* ,@(cdr lst))))) (def-macro (/ . lst) (if (null? (cdr lst)) (car lst) `(/fx ,(car lst) (/ ,@(cdr lst)))))
(def-macro (FLOAT+ . lst) (if (null? (cdr lst)) (car lst) `(+fl ,(car lst) (FLOAT+ ,@(cdr lst))))) (def-macro (FLOAT- . lst) (if (null? (cdr lst)) (car lst) `(-fl ,(car lst) (FLOAT- ,@(cdr lst))))) (def-macro (FLOAT* . lst) (if (null? (cdr lst)) (car lst) `(*fl ,(car lst) (FLOAT* ,@(cdr lst))))) (def-macro (FLOAT/ . lst) (if (null? (cdr lst)) (car lst) `(/fl ,(car lst) (FLOAT/ ,@(cdr lst))))) (def-macro (FLOAT= a b) `(=fl ,a ,b)) (def-macro (FLOAT< a b) `(<fl ,a ,b)) (def-macro (FLOAT<= a b) `(<= ,a ,b)) (def-macro (FLOAT> a b) `(> ,a ,b)) (def-macro (FLOAT>= a b) `(>= ,a ,b)) (def-macro (FLOATnegative? a) `(negativefl? ,a)) (def-macro (FLOATpositive? a) `(positivefl? ,a)) (def-macro (FLOATzero? a) `(zerofl? ,a)) (def-macro (FLOATabs a) `(abs ,a)) (def-macro (FLOATsin a) `(sin ,a)) (def-macro (FLOATcos a) `(cos ,a)) (def-macro (FLOATatan a) `(atan ,a)) (def-macro (FLOATsqrt a) `(sqrt ,a)) (def-macro (FLOATmin a) `(min ,a)) (def-macro (FLOATmax a) `(max ,a)) (def-macro (FLOATround a) `(round ,a)) (def-macro (FLOATinexact->exact a) `(inexact->exact ,a)) )
; Generic arithmetic.
(begin (def-macro (GENERIC+ . lst) `(+ ,@lst)) (def-macro (GENERIC- . lst) `(- ,@lst)) (def-macro (GENERIC* . lst) `(* ,@lst)) (def-macro (GENERIC/ . lst) `(/ ,@lst)) (def-macro (GENERICquotient . lst) `(quotient ,@lst)) (def-macro (GENERICremainder . lst) `(remainder ,@lst)) (def-macro (GENERICmodulo . lst) `(modulo ,@lst)) (def-macro (GENERIC= . lst) `(= ,@lst)) (def-macro (GENERIC< . lst) `(< ,@lst)) (def-macro (GENERIC<= . lst) `(<= ,@lst)) (def-macro (GENERIC> . lst) `(> ,@lst)) (def-macro (GENERIC>= . lst) `(>= ,@lst)) (def-macro (GENERICexpt . lst) `(expt ,@lst)) )
(begin (def-macro (bitwise-or x y) `(bit-or ,x ,y)) (def-macro (bitwise-and x y) `(bit-and ,x ,y)) (def-macro (bitwise-not x) `(bit-not ,x)) ) -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|---- -|-----
ps: benchmarks using vector of float could be improved by using Bigloo typed vectors. I have not tried to apply this because as I said, I don't want to allocate to much time to benchmarking. However, if you want accurate Bigloo measurement. You should try to add the following to prefix-bigloo.scm:
(module prefix (main main-entry) (extern (include "sys/times.h")) (extern (include "time.h")) ... (type (tvector f64vector (double))))
(def-macro (FLOATvector? x) `(f64vector? ,x)) (def-macro (FLOATvector-ref v i) `(f64vector-ref ,v ,i)) ...
Hey Marc -- did you try the benchmarks with mzscheme3m (which uses the newer gc)? We haven't made that the default collector yet, but I believe that is Matthew's plan. (If you're building from source, you'll need to do "make 3m" and "make install-3m" instead of just "make" and "make install" to get 3m). Robby On Aug 27, 2006, at 8:10 AM, Marc Feeley wrote:
Manuel thank you for taking the time to look into the benchmarks and determining what would produce the fastest execution with Bigloo. By the way, wouldn't it also make sense to specialize =, <, <= and other operators to fixnums? As you discovered, the benchmark suite was written so that it is easy to specialize the arithmetic operators, for example, wherever the benchmarks call FLOAT+ you can either call plain + or the flonum addition and where the benchmarks call + you can either call plain + or the fixnum addition (i.e. when there is no prefix on the name, the fixnum operators will work). With appropriate macro definitions in the system specific prefix file you can force particular specializations.
The code you sent for Bigloo will be useful for experiments where the code is compiled in "unsafe" mode with no overflow detection (i.e. which models the setting where the user is willing to have the fastest possible code at all costs, possibly crashing the program if there is any bug in the code). This was not the setting Eli and I were interested in benchmarking. Eli and I are trying to model the (expected) semantics of R6RS Scheme:
- safe execution (i.e. an exception has to be signalled on errors) - overflow detection for fixnum arithmetic (either produce a bignum or signal an exception) - immutable bindings for the definitions in the benchmark (as if in an R6RS module) - immutable predefined bindings
That is why I did not specialize the arithmetic operators in the prefix files for Bigloo (and Gambit and MzScheme).
Correct me if I'm wrong but the only improvement in this setting is the addition of the -fomit-frame-pointer option, i.e. using this in the "bench" script:
bigloo -O6 $usecc -copt -O3 -copt -fomit-frame-pointer $1.scm - o $1
What would I have to add to detect fixnum arithmetic overflows? Would adding "-copt -ftrapv" make sense? Does Bigloo support bignums?
Marc
On 27-Aug-06, at 3:49 AM, Manuel Serrano wrote:
On Sat, 26 Aug 2006 21:31:16 -0400 Marc Feeley <feeley@iro.umontreal.ca> wrote:
Désolé j'ai oublié de te mettre en CC.
Marc
Begin forwarded message:
From: Marc Feeley <feeley@iro.umontreal.ca> Date: August 26, 2006 9:28:15 PM EDT (CA) To: Gambit List <Gambit-list@iro.umontreal.ca> Cc: Eli Barzilay <eli@barzilay.org>, Matthew Flatt <mflatt@cs.utah.edu> Subject: Gambit-C vs. Bigloo vs. MzScheme
After I saw Eli Barzilay's post to comp.lang.scheme about the performance of MzScheme compared to other Scheme systems (http:// groups.google.com/group/comp.lang.scheme/msg/473f91e7065b6f73? dmode=source) I was tempted to do some benchmarking of my own using the Gambit Scheme benchmark suite which goes beyond the classical Gabriel benchmarks (it has close to 60 Scheme benchmarks). Most benchmarks are straightforward R4RS code and so should work on just about any Scheme system.
I have updated the benchmarking infrastructure to accomodate other Scheme systems, and MzScheme and Bigloo in particular. I also wrote a script to run all benchmarks and automatically generate an HTML table of the results. The table can be viewed here:
http://www.iro.umontreal.ca/~gambit/bench.html
And the benchmark suite is available here:
http://www.iro.umontreal.ca/~gambit/bench.tar.gz
As you can see from the table, Gambit-C is faster than Bigloo and MzScheme on roughly 3 out of 4 benchmarks. I have used the same assumptions as Eli, which approximate the semantics of R6RS (see the last section of the result page for details).
It would be really interesting to extend this experiment to other Scheme systems. It should not be too difficult to modify the benchmarking scripts to test other systems. The hardest parts are installing the Scheme system, figuring out which compiler options are equivalent to the ones used by the other systems (i.e. approximate R6RS semantics), and work out unexpected problems.
The following systems are particularly interesting to compare to:
- Chicken - Scheme 48 - Larceny - Chez Scheme
Are there any volunteers?
Marc
Hello Marc,
Ah, benchmarking... Definitely we can't escape benchmarks :-). I had more or less sweared not to benchmark any more because this is an activity that can consume all your time and for which the result is not necessarily very useful. However, because I didn't want to spend a bad Sunday :-) I have downloaded your benchmarks and make some measurements with Bigloo. Here is what I got:
-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|--- - -|----- General command line: bigloo -O6
OFP=-copt -fomit-frame-pointer fixnum= /+/+fx/, /-/-fx/, ... unsafe=bigloo -O6 -unsafe FLOAT= re-implementing FLOAT+, FLOAT-, ...
===================================================================== = tak1:
Marc's time: real: 11.22s sys: 0.02s user: 11.13s fixnum: real: 3.61s sys: 0.01s user: 3.57s fixnum, omit-frame-pointer: real: 2.80s sys: 0.01s user: 2.78s
===================================================================== = fft:
Marc's time: real: 11.71s sys: 0.03s user: 11.32s FLOAT fixed: real: 4.39s sys: 0.02s user: 4.10s FLOAT fixed, unsafe real: 3.44s sys: 0.01s user: 3.33s FLOAT fixed, unsafe, $OFP real: 3.02s sys: 0.01s user: 2.92s
===================================================================== = pnpoly:
Marc's time: real: 16.39s sys: 0.02s user: 15.79s FLOAT: real: 15.81s sys: 0.02s user: 14.50s FLOAT, unsafe: real: 1.86s sys: 0.01s user: 1.82s FLOAT, unsafe, $OFP: real: 1.49s sys: 0.00s user: 1.47s
===================================================================== = sum:
Marc's time: real: 5.44s sys: 0.01s user: 5.36s FLOAT, unsafe, $OFP: real: 5.29s sys: 0.01s user: 5.22s FIXNUM, unsafe, $OFP: real: 0.29s sys: 0.00s user: 0.27s
===================================================================== = sumfp:
Marc's time: real: 9.76s sys: 0.01s user: 9.57s FLOAT: real: 0.55s sys: 0.01s user: 0.52s FLOAT, unsafe, $OFP: real: 0.54s sys: 0.00s user: 0.52s -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|--- - -|-----
As you can notice, changing the compilation options and improving your prefix-bigloo.scm file has a huge impact. As you can notice some benchmarks now go 18 times faster!
Here is a description of the modiication I have applied:
1- I have re-implemented your macros FLOATxxx that were badling implemented for Bigloo. They used to expanse into call to the generic functions. They now expanse to the floating point operator. The improvement for benchmark using floating applications is in between 2 and 5 times faster.
2- I have used the proper benchmarking compilation options. a- The produced C code must be compiled with -fomit-frame-pointer (when using GCC). b- The option -unsafe has to be used.
3- For fixnum benchmarks, I have replaced the call to the generic functions with calls to fixnum functions.
Please, find enclosed the new version of prefix-bigloo.scm. I would appreciate if you could re-execute the Bigloo benchmarks with the new prefix and update your web pages with the new Bigloo results. Thanks in advance.
-- Manuel
-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|--- - -|----- (module prefix (main main-entry) (extern (include "sys/times.h")) (extern (include "time.h")) (option (set! *unsafe-rgc* #t)) (option (set! *unsafe-library* #t)) (option (set! *unsafe-arity* #t)) (option (set! *unsafe-type* #t)) (option (set! *unsafe-struct* #t)) (option (set! *unsafe-range* #t)) (option (set! *unsafe-version* #t)) (option (set! *cc-options* (string-append "-fomit-frame-pointer " *cc-options*))))
(define (clock-to-msecs x) (quotient (* x 1000) (pragma::int "CLOCKS_PER_SEC")))
(define (process-times) (let () (pragma "struct tms buf") (let ((real::long (pragma::long "times(&buf)"))) (cons real (+ (pragma::long "buf.tms_utime") (pragma::long "buf.tms_stime"))))))
(define (time* thunk) (let ((start (process-times))) (let ((result (thunk))) (let ((end (process-times))) (let ((cpu (clock-to-msecs (- (cdr end) (cdr start)))) (real (clock-to-msecs (- (car end) (car start))))) (display "cpu time: ") (display cpu) (display " real time: ") (display real) (newline) result)))))
(define (run-benchmark name count ok? run-maker . args) (let* ((run (apply run-maker args)) (result (time* (lambda () (run-bench name count ok? run))))) (if (not (ok? result)) (begin (display "*** wrong result ***") (newline) (display "*** got: ") (write result) (newline)))))
(define (run-bench name count ok? run) (let loop ((i 0) (result (list 'undefined))) (if (< i count) (loop (+ i 1) (run)) result)))
(define (fatal-error . args) (for-each display args) (newline) (exit 1))
(define (main-entry args) (main))
;-------------------------------------------------------------------- - ---------
; Macros...
(define-macro (def-macro form . body) `(define-macro ,form (let () ,@body)))
; Flonum arithmetic.
(begin (def-macro (FLOATvector-const . lst) `',(list->vector lst)) (def-macro (FLOATvector? x) `(vector? ,x)) (def-macro (FLOATvector . lst) `(vector ,@lst)) (def-macro (FLOATmake-vector n . init) `(make-vector ,n ,@init)) (def-macro (FLOATvector-ref v i) `(vector-ref ,v ,i)) (def-macro (FLOATvector-set! v i x) `(vector-set! ,v ,i ,x)) (def-macro (FLOATvector-length v) `(vector-length ,v))
(def-macro (nuc-const . lst) `',(list->vector lst)) )
(begin (def-macro (+ . lst) (if (null? (cdr lst)) (car lst) `(+fx ,(car lst) (+ ,@(cdr lst))))) (def-macro (- . lst) (if (null? (cdr lst)) (car lst) `(-fx ,(car lst) (- ,@(cdr lst))))) (def-macro (* . lst) (if (null? (cdr lst)) (car lst) `(*fx ,(car lst) (* ,@(cdr lst))))) (def-macro (/ . lst) (if (null? (cdr lst)) (car lst) `(/fx ,(car lst) (/ ,@(cdr lst)))))
(def-macro (FLOAT+ . lst) (if (null? (cdr lst)) (car lst) `(+fl ,(car lst) (FLOAT+ ,@(cdr lst))))) (def-macro (FLOAT- . lst) (if (null? (cdr lst)) (car lst) `(-fl ,(car lst) (FLOAT- ,@(cdr lst))))) (def-macro (FLOAT* . lst) (if (null? (cdr lst)) (car lst) `(*fl ,(car lst) (FLOAT* ,@(cdr lst))))) (def-macro (FLOAT/ . lst) (if (null? (cdr lst)) (car lst) `(/fl ,(car lst) (FLOAT/ ,@(cdr lst))))) (def-macro (FLOAT= a b) `(=fl ,a ,b)) (def-macro (FLOAT< a b) `(<fl ,a ,b)) (def-macro (FLOAT<= a b) `(<= ,a ,b)) (def-macro (FLOAT> a b) `(> ,a ,b)) (def-macro (FLOAT>= a b) `(>= ,a ,b)) (def-macro (FLOATnegative? a) `(negativefl? ,a)) (def-macro (FLOATpositive? a) `(positivefl? ,a)) (def-macro (FLOATzero? a) `(zerofl? ,a)) (def-macro (FLOATabs a) `(abs ,a)) (def-macro (FLOATsin a) `(sin ,a)) (def-macro (FLOATcos a) `(cos ,a)) (def-macro (FLOATatan a) `(atan ,a)) (def-macro (FLOATsqrt a) `(sqrt ,a)) (def-macro (FLOATmin a) `(min ,a)) (def-macro (FLOATmax a) `(max ,a)) (def-macro (FLOATround a) `(round ,a)) (def-macro (FLOATinexact->exact a) `(inexact->exact ,a)) )
; Generic arithmetic.
(begin (def-macro (GENERIC+ . lst) `(+ ,@lst)) (def-macro (GENERIC- . lst) `(- ,@lst)) (def-macro (GENERIC* . lst) `(* ,@lst)) (def-macro (GENERIC/ . lst) `(/ ,@lst)) (def-macro (GENERICquotient . lst) `(quotient ,@lst)) (def-macro (GENERICremainder . lst) `(remainder ,@lst)) (def-macro (GENERICmodulo . lst) `(modulo ,@lst)) (def-macro (GENERIC= . lst) `(= ,@lst)) (def-macro (GENERIC< . lst) `(< ,@lst)) (def-macro (GENERIC<= . lst) `(<= ,@lst)) (def-macro (GENERIC> . lst) `(> ,@lst)) (def-macro (GENERIC>= . lst) `(>= ,@lst)) (def-macro (GENERICexpt . lst) `(expt ,@lst)) )
(begin (def-macro (bitwise-or x y) `(bit-or ,x ,y)) (def-macro (bitwise-and x y) `(bit-and ,x ,y)) (def-macro (bitwise-not x) `(bit-not ,x)) ) -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|--- - -|-----
ps: benchmarks using vector of float could be improved by using Bigloo typed vectors. I have not tried to apply this because as I said, I don't want to allocate to much time to benchmarking. However, if you want accurate Bigloo measurement. You should try to add the following to prefix-bigloo.scm:
(module prefix (main main-entry) (extern (include "sys/times.h")) (extern (include "time.h")) ... (type (tvector f64vector (double))))
(def-macro (FLOATvector? x) `(f64vector? ,x)) (def-macro (FLOATvector-ref v i) `(f64vector-ref ,v ,i)) ...
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca http://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
participants (3)
-
Manuel Serrano -
Marc Feeley -
Robby Findler