[gambit-list] Gambit-C vs. Bigloo vs. MzScheme
Robby Findler
robby at cs.uchicago.edu
Sun Aug 27 09:24:40 EDT 2006
Hey Marc -- did you try the benchmarks with mzscheme3m (which uses
the newer gc)? We haven't made that the default collector yet, but I
believe that is Matthew's plan.
(If you're building from source, you'll need to do "make 3m" and
"make install-3m" instead of just "make" and "make install" to get 3m).
Robby
On Aug 27, 2006, at 8:10 AM, Marc Feeley wrote:
> Manuel thank you for taking the time to look into the benchmarks and
> determining what would produce the fastest execution with Bigloo. By
> the way, wouldn't it also make sense to specialize =, <, <= and other
> operators to fixnums? As you discovered, the benchmark suite was
> written so that it is easy to specialize the arithmetic operators,
> for example, wherever the benchmarks call FLOAT+ you can either call
> plain + or the flonum addition and where the benchmarks call + you
> can either call plain + or the fixnum addition (i.e. when there is no
> prefix on the name, the fixnum operators will work). With
> appropriate macro definitions in the system specific prefix file you
> can force particular specializations.
>
> The code you sent for Bigloo will be useful for experiments where the
> code is compiled in "unsafe" mode with no overflow detection (i.e.
> which models the setting where the user is willing to have the
> fastest possible code at all costs, possibly crashing the program if
> there is any bug in the code). This was not the setting Eli and I
> were interested in benchmarking. Eli and I are trying to model the
> (expected) semantics of R6RS Scheme:
>
> - safe execution (i.e. an exception has to be signalled on errors)
> - overflow detection for fixnum arithmetic (either produce a
> bignum or signal an exception)
> - immutable bindings for the definitions in the benchmark (as if
> in an R6RS module)
> - immutable predefined bindings
>
> That is why I did not specialize the arithmetic operators in the
> prefix files for Bigloo (and Gambit and MzScheme).
>
> Correct me if I'm wrong but the only improvement in this setting is
> the addition of the -fomit-frame-pointer option, i.e. using this in
> the "bench" script:
>
> bigloo -O6 $usecc -copt -O3 -copt -fomit-frame-pointer $1.scm -
> o $1
>
> What would I have to add to detect fixnum arithmetic overflows?
> Would adding "-copt -ftrapv" make sense? Does Bigloo support bignums?
>
> Marc
>
>
> On 27-Aug-06, at 3:49 AM, Manuel Serrano wrote:
>
>>
>> On Sat, 26 Aug 2006 21:31:16 -0400
>> Marc Feeley <feeley at iro.umontreal.ca> wrote:
>>
>>> Désolé j'ai oublié de te mettre en CC.
>>>
>>> Marc
>>>
>>>
>>> Begin forwarded message:
>>>
>>>> From: Marc Feeley <feeley at iro.umontreal.ca>
>>>> Date: August 26, 2006 9:28:15 PM EDT (CA)
>>>> To: Gambit List <Gambit-list at iro.umontreal.ca>
>>>> Cc: Eli Barzilay <eli at barzilay.org>, Matthew Flatt
>>>> <mflatt at cs.utah.edu>
>>>> Subject: Gambit-C vs. Bigloo vs. MzScheme
>>>>
>>>> After I saw Eli Barzilay's post to comp.lang.scheme about the
>>>> performance of MzScheme compared to other Scheme systems (http://
>>>> groups.google.com/group/comp.lang.scheme/msg/473f91e7065b6f73?
>>>> dmode=source) I was tempted to do some benchmarking of my own
>>>> using the Gambit Scheme benchmark suite which goes beyond the
>>>> classical Gabriel benchmarks (it has close to 60 Scheme
>>>> benchmarks). Most benchmarks are straightforward R4RS code and so
>>>> should work on just about any Scheme system.
>>>>
>>>> I have updated the benchmarking infrastructure to accomodate other
>>>> Scheme systems, and MzScheme and Bigloo in particular. I also
>>>> wrote a script to run all benchmarks and automatically generate an
>>>> HTML table of the results. The table can be viewed here:
>>>>
>>>> http://www.iro.umontreal.ca/~gambit/bench.html
>>>>
>>>> And the benchmark suite is available here:
>>>>
>>>> http://www.iro.umontreal.ca/~gambit/bench.tar.gz
>>>>
>>>> As you can see from the table, Gambit-C is faster than Bigloo and
>>>> MzScheme on roughly 3 out of 4 benchmarks. I have used the same
>>>> assumptions as Eli, which approximate the semantics of R6RS (see
>>>> the last section of the result page for details).
>>>>
>>>> It would be really interesting to extend this experiment to other
>>>> Scheme systems. It should not be too difficult to modify the
>>>> benchmarking scripts to test other systems. The hardest parts are
>>>> installing the Scheme system, figuring out which compiler options
>>>> are equivalent to the ones used by the other systems (i.e.
>>>> approximate R6RS semantics), and work out unexpected problems.
>>>>
>>>> The following systems are particularly interesting to compare to:
>>>>
>>>> - Chicken
>>>> - Scheme 48
>>>> - Larceny
>>>> - Chez Scheme
>>>>
>>>> Are there any volunteers?
>>>>
>>>> Marc
>>>
>>
>> Hello Marc,
>>
>> Ah, benchmarking... Definitely we can't escape benchmarks :-). I
>> had more or
>> less sweared not to benchmark any more because this is an
>> activity that can consume all your time and for which the result is
>> not necessarily very useful. However, because I didn't want to
>> spend a
>> bad Sunday :-) I have downloaded your benchmarks and make some
>> measurements
>> with Bigloo. Here is what I got:
>>
>> -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|---
>> -
>> -|-----
>> General command line: bigloo -O6
>>
>> OFP=-copt -fomit-frame-pointer
>> fixnum= /+/+fx/, /-/-fx/, ...
>> unsafe=bigloo -O6 -unsafe
>> FLOAT= re-implementing FLOAT+, FLOAT-, ...
>>
>> =====================================================================
>> =
>> tak1:
>>
>> Marc's time: real: 11.22s sys: 0.02s user: 11.13s
>> fixnum: real: 3.61s sys: 0.01s user: 3.57s
>> fixnum, omit-frame-pointer: real: 2.80s sys: 0.01s user: 2.78s
>>
>> =====================================================================
>> =
>> fft:
>>
>> Marc's time: real: 11.71s sys: 0.03s user: 11.32s
>> FLOAT fixed: real: 4.39s sys: 0.02s user: 4.10s
>> FLOAT fixed, unsafe real: 3.44s sys: 0.01s user: 3.33s
>> FLOAT fixed, unsafe, $OFP real: 3.02s sys: 0.01s user: 2.92s
>>
>> =====================================================================
>> =
>> pnpoly:
>>
>> Marc's time: real: 16.39s sys: 0.02s user: 15.79s
>> FLOAT: real: 15.81s sys: 0.02s user: 14.50s
>> FLOAT, unsafe: real: 1.86s sys: 0.01s user: 1.82s
>> FLOAT, unsafe, $OFP: real: 1.49s sys: 0.00s user: 1.47s
>>
>>
>> =====================================================================
>> =
>> sum:
>>
>> Marc's time: real: 5.44s sys: 0.01s user: 5.36s
>> FLOAT, unsafe, $OFP: real: 5.29s sys: 0.01s user: 5.22s
>> FIXNUM, unsafe, $OFP: real: 0.29s sys: 0.00s user: 0.27s
>>
>> =====================================================================
>> =
>> sumfp:
>>
>> Marc's time: real: 9.76s sys: 0.01s user: 9.57s
>> FLOAT: real: 0.55s sys: 0.01s user: 0.52s
>> FLOAT, unsafe, $OFP: real: 0.54s sys: 0.00s user: 0.52s
>> -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|---
>> -
>> -|-----
>>
>> As you can notice, changing the compilation options and improving
>> your
>> prefix-bigloo.scm file has a huge impact. As you can notice some
>> benchmarks now go 18 times faster!
>>
>> Here is a description of the modiication I have applied:
>>
>> 1- I have re-implemented your macros FLOATxxx that were badling
>> implemented
>> for Bigloo. They used to expanse into call to the generic
>> functions. They
>> now expanse to the floating point operator. The improvement for
>> benchmark
>> using floating applications is in between 2 and 5 times faster.
>>
>> 2- I have used the proper benchmarking compilation options.
>> a- The produced C code must be compiled with -fomit-frame-pointer
>> (when
>> using GCC).
>> b- The option -unsafe has to be used.
>>
>> 3- For fixnum benchmarks, I have replaced the call to the generic
>> functions
>> with calls to fixnum functions.
>>
>> Please, find enclosed the new version of prefix-bigloo.scm. I would
>> appreciate if you could re-execute the Bigloo benchmarks with the new
>> prefix and update your web pages with the new Bigloo results. Thanks
>> in advance.
>>
>> --
>> Manuel
>>
>> -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|---
>> -
>> -|-----
>> (module prefix (main main-entry)
>> (extern (include "sys/times.h"))
>> (extern (include "time.h"))
>> (option (set! *unsafe-rgc* #t))
>> (option (set! *unsafe-library* #t))
>> (option (set! *unsafe-arity* #t))
>> (option (set! *unsafe-type* #t))
>> (option (set! *unsafe-struct* #t))
>> (option (set! *unsafe-range* #t))
>> (option (set! *unsafe-version* #t))
>> (option (set! *cc-options*
>> (string-append "-fomit-frame-pointer " *cc-options*))))
>>
>> (define (clock-to-msecs x)
>> (quotient (* x 1000) (pragma::int "CLOCKS_PER_SEC")))
>>
>> (define (process-times)
>> (let ()
>> (pragma "struct tms buf")
>> (let ((real::long (pragma::long "times(&buf)")))
>> (cons real
>> (+ (pragma::long "buf.tms_utime")
>> (pragma::long "buf.tms_stime"))))))
>>
>> (define (time* thunk)
>> (let ((start (process-times)))
>> (let ((result (thunk)))
>> (let ((end (process-times)))
>> (let ((cpu (clock-to-msecs (- (cdr end) (cdr start))))
>> (real (clock-to-msecs (- (car end) (car start)))))
>> (display "cpu time: ")
>> (display cpu)
>> (display " real time: ")
>> (display real)
>> (newline)
>> result)))))
>>
>> (define (run-benchmark name count ok? run-maker . args)
>> (let* ((run (apply run-maker args))
>> (result (time* (lambda () (run-bench name count ok? run)))))
>> (if (not (ok? result))
>> (begin
>> (display "*** wrong result ***")
>> (newline)
>> (display "*** got: ")
>> (write result)
>> (newline)))))
>>
>> (define (run-bench name count ok? run)
>> (let loop ((i 0) (result (list 'undefined)))
>> (if (< i count)
>> (loop (+ i 1) (run))
>> result)))
>>
>> (define (fatal-error . args)
>> (for-each display args)
>> (newline)
>> (exit 1))
>>
>> (define (main-entry args)
>> (main))
>>
>> ;--------------------------------------------------------------------
>> -
>> ---------
>>
>> ; Macros...
>>
>> (define-macro (def-macro form . body)
>> `(define-macro ,form (let () , at body)))
>>
>> ; Flonum arithmetic.
>>
>> (begin
>> (def-macro (FLOATvector-const . lst) `',(list->vector lst))
>> (def-macro (FLOATvector? x) `(vector? ,x))
>> (def-macro (FLOATvector . lst) `(vector , at lst))
>> (def-macro (FLOATmake-vector n . init) `(make-vector ,n , at init))
>> (def-macro (FLOATvector-ref v i) `(vector-ref ,v ,i))
>> (def-macro (FLOATvector-set! v i x) `(vector-set! ,v ,i ,x))
>> (def-macro (FLOATvector-length v) `(vector-length ,v))
>>
>> (def-macro (nuc-const . lst)
>> `',(list->vector lst))
>> )
>>
>> (begin
>> (def-macro (+ . lst)
>> (if (null? (cdr lst)) (car lst) `(+fx ,(car lst) (+ ,@(cdr
>> lst)))))
>> (def-macro (- . lst)
>> (if (null? (cdr lst)) (car lst) `(-fx ,(car lst) (- ,@(cdr
>> lst)))))
>> (def-macro (* . lst)
>> (if (null? (cdr lst)) (car lst) `(*fx ,(car lst) (* ,@(cdr
>> lst)))))
>> (def-macro (/ . lst)
>> (if (null? (cdr lst)) (car lst) `(/fx ,(car lst) (/ ,@(cdr
>> lst)))))
>>
>> (def-macro (FLOAT+ . lst)
>> (if (null? (cdr lst)) (car lst) `(+fl ,(car lst) (FLOAT+ ,@(cdr
>> lst)))))
>> (def-macro (FLOAT- . lst)
>> (if (null? (cdr lst)) (car lst) `(-fl ,(car lst) (FLOAT- ,@(cdr
>> lst)))))
>> (def-macro (FLOAT* . lst)
>> (if (null? (cdr lst)) (car lst) `(*fl ,(car lst) (FLOAT* ,@(cdr
>> lst)))))
>> (def-macro (FLOAT/ . lst)
>> (if (null? (cdr lst)) (car lst) `(/fl ,(car lst) (FLOAT/ ,@(cdr
>> lst)))))
>> (def-macro (FLOAT= a b) `(=fl ,a ,b))
>> (def-macro (FLOAT< a b) `(<fl ,a ,b))
>> (def-macro (FLOAT<= a b) `(<= ,a ,b))
>> (def-macro (FLOAT> a b) `(> ,a ,b))
>> (def-macro (FLOAT>= a b) `(>= ,a ,b))
>> (def-macro (FLOATnegative? a) `(negativefl? ,a))
>> (def-macro (FLOATpositive? a) `(positivefl? ,a))
>> (def-macro (FLOATzero? a) `(zerofl? ,a))
>> (def-macro (FLOATabs a) `(abs ,a))
>> (def-macro (FLOATsin a) `(sin ,a))
>> (def-macro (FLOATcos a) `(cos ,a))
>> (def-macro (FLOATatan a) `(atan ,a))
>> (def-macro (FLOATsqrt a) `(sqrt ,a))
>> (def-macro (FLOATmin a) `(min ,a))
>> (def-macro (FLOATmax a) `(max ,a))
>> (def-macro (FLOATround a) `(round ,a))
>> (def-macro (FLOATinexact->exact a) `(inexact->exact ,a))
>> )
>>
>> ; Generic arithmetic.
>>
>> (begin
>> (def-macro (GENERIC+ . lst) `(+ , at lst))
>> (def-macro (GENERIC- . lst) `(- , at lst))
>> (def-macro (GENERIC* . lst) `(* , at lst))
>> (def-macro (GENERIC/ . lst) `(/ , at lst))
>> (def-macro (GENERICquotient . lst) `(quotient , at lst))
>> (def-macro (GENERICremainder . lst) `(remainder , at lst))
>> (def-macro (GENERICmodulo . lst) `(modulo , at lst))
>> (def-macro (GENERIC= . lst) `(= , at lst))
>> (def-macro (GENERIC< . lst) `(< , at lst))
>> (def-macro (GENERIC<= . lst) `(<= , at lst))
>> (def-macro (GENERIC> . lst) `(> , at lst))
>> (def-macro (GENERIC>= . lst) `(>= , at lst))
>> (def-macro (GENERICexpt . lst) `(expt , at lst))
>> )
>>
>> (begin
>> (def-macro (bitwise-or x y) `(bit-or ,x ,y))
>> (def-macro (bitwise-and x y) `(bit-and ,x ,y))
>> (def-macro (bitwise-not x) `(bit-not ,x))
>> )
>> -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|---
>> -
>> -|-----
>>
>> ps: benchmarks using vector of float could be improved by using
>> Bigloo
>> typed vectors. I have not tried to apply this because as I said, I
>> don't want
>> to allocate to much time to benchmarking. However, if you want
>> accurate
>> Bigloo measurement. You should try to add the following to
>> prefix-bigloo.scm:
>>
>>
>>
>> (module prefix (main main-entry)
>> (extern (include "sys/times.h"))
>> (extern (include "time.h"))
>> ...
>> (type (tvector f64vector (double))))
>>
>> (def-macro (FLOATvector? x) `(f64vector? ,x))
>> (def-macro (FLOATvector-ref v i) `(f64vector-ref ,v ,i))
>> ...
>
> _______________________________________________
> Gambit-list mailing list
> Gambit-list at iro.umontreal.ca
> http://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
>
More information about the Gambit-list
mailing list