[gambit-list] Gambit-C vs. Bigloo vs. MzScheme

Robby Findler robby at cs.uchicago.edu
Sun Aug 27 09:24:40 EDT 2006


Hey Marc -- did you try the benchmarks with mzscheme3m (which uses  
the newer gc)? We haven't made that the default collector yet, but I  
believe that is Matthew's plan.

(If you're building from source, you'll need to do "make 3m" and  
"make install-3m" instead of just "make" and "make install" to get 3m).

Robby

On Aug 27, 2006, at 8:10 AM, Marc Feeley wrote:

> Manuel thank you for taking the time to look into the benchmarks and
> determining what would produce the fastest execution with Bigloo.  By
> the way, wouldn't it also make sense to specialize =, <, <= and other
> operators to fixnums?  As you discovered, the benchmark suite was
> written so that it is easy to specialize the arithmetic operators,
> for example, wherever the benchmarks call FLOAT+ you can either call
> plain + or the flonum addition and where the benchmarks call + you
> can either call plain + or the fixnum addition (i.e. when there is no
> prefix on the name, the fixnum operators will work).  With
> appropriate macro definitions in the system specific prefix file you
> can force particular specializations.
>
> The code you sent for Bigloo will be useful for experiments where the
> code is compiled in "unsafe" mode with no overflow detection (i.e.
> which models the setting where the user is willing to have the
> fastest possible code at all costs, possibly crashing the program if
> there is any bug in the code).  This was not the setting Eli and I
> were interested in benchmarking.  Eli and I are trying to model the
> (expected) semantics of R6RS Scheme:
>
>     - safe execution (i.e. an exception has to be signalled on errors)
>     - overflow detection for fixnum arithmetic (either produce a
> bignum or signal an exception)
>     - immutable bindings for the definitions in the benchmark (as if
> in an R6RS module)
>     - immutable predefined bindings
>
> That is why I did not specialize the arithmetic operators in the
> prefix files for Bigloo (and Gambit and MzScheme).
>
> Correct me if I'm wrong but the only improvement in this setting is
> the addition of the -fomit-frame-pointer option, i.e. using this in
> the "bench" script:
>
>      bigloo -O6 $usecc -copt -O3 -copt -fomit-frame-pointer $1.scm - 
> o $1
>
> What would I have to add to detect fixnum arithmetic overflows?
> Would adding "-copt -ftrapv" make sense?  Does Bigloo support bignums?
>
> Marc
>
>
> On 27-Aug-06, at 3:49 AM, Manuel Serrano wrote:
>
>>
>> On Sat, 26 Aug 2006 21:31:16 -0400
>> Marc Feeley <feeley at iro.umontreal.ca> wrote:
>>
>>> Désolé j'ai oublié de te mettre en CC.
>>>
>>> Marc
>>>
>>>
>>> Begin forwarded message:
>>>
>>>> From: Marc Feeley <feeley at iro.umontreal.ca>
>>>> Date: August 26, 2006 9:28:15 PM EDT (CA)
>>>> To: Gambit List <Gambit-list at iro.umontreal.ca>
>>>> Cc: Eli Barzilay <eli at barzilay.org>, Matthew Flatt
>>>> <mflatt at cs.utah.edu>
>>>> Subject: Gambit-C vs. Bigloo vs. MzScheme
>>>>
>>>> After I saw Eli Barzilay's post to comp.lang.scheme about the
>>>> performance of MzScheme compared to other Scheme systems (http://
>>>> groups.google.com/group/comp.lang.scheme/msg/473f91e7065b6f73?
>>>> dmode=source) I was tempted to do some benchmarking of my own
>>>> using the Gambit Scheme benchmark suite which goes beyond the
>>>> classical Gabriel benchmarks (it has close to 60 Scheme
>>>> benchmarks).  Most benchmarks are straightforward R4RS code and so
>>>> should work on just about any Scheme system.
>>>>
>>>> I have updated the benchmarking infrastructure to accomodate other
>>>> Scheme systems, and MzScheme and Bigloo in particular.  I also
>>>> wrote a script to run all benchmarks and automatically generate an
>>>> HTML table of the results.  The table can be viewed here:
>>>>
>>>>    http://www.iro.umontreal.ca/~gambit/bench.html
>>>>
>>>> And the benchmark suite is available here:
>>>>
>>>>    http://www.iro.umontreal.ca/~gambit/bench.tar.gz
>>>>
>>>> As you can see from the table, Gambit-C is faster than Bigloo and
>>>> MzScheme on roughly 3 out of 4 benchmarks.  I have used the same
>>>> assumptions as Eli, which approximate the semantics of R6RS (see
>>>> the last section of the result page for details).
>>>>
>>>> It would be really interesting to extend this experiment to other
>>>> Scheme systems.  It should not be too difficult to modify the
>>>> benchmarking scripts to test other systems.  The hardest parts are
>>>> installing the Scheme system, figuring out which compiler options
>>>> are equivalent to the ones used by the other systems (i.e.
>>>> approximate R6RS semantics), and work out unexpected problems.
>>>>
>>>> The following systems are particularly interesting to compare to:
>>>>
>>>>   - Chicken
>>>>   - Scheme 48
>>>>   - Larceny
>>>>   - Chez Scheme
>>>>
>>>> Are there any volunteers?
>>>>
>>>> Marc
>>>
>>
>> Hello Marc,
>>
>> Ah, benchmarking... Definitely we can't escape benchmarks :-). I
>> had more or
>> less sweared not to benchmark any more because this is an
>> activity that can consume all your time and for which the result is
>> not necessarily very useful. However, because I didn't want to  
>> spend a
>> bad Sunday :-) I have downloaded your benchmarks and make some
>> measurements
>> with Bigloo. Here is what I got:
>>
>> -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|--- 
>> -
>> -|-----
>> General command line: bigloo -O6
>>
>> OFP=-copt -fomit-frame-pointer
>> fixnum= /+/+fx/, /-/-fx/, ...
>> unsafe=bigloo -O6 -unsafe
>> FLOAT= re-implementing FLOAT+, FLOAT-, ...
>>
>> ===================================================================== 
>> =
>> tak1:
>>
>> Marc's time: 		   	real: 11.22s  sys: 0.02s  user: 11.13s
>> fixnum: 		   	real: 3.61s  sys: 0.01s  user: 3.57s
>> fixnum, omit-frame-pointer: 	real: 2.80s  sys: 0.01s  user: 2.78s
>>
>> ===================================================================== 
>> =
>> fft:
>>
>> Marc's time:			real: 11.71s  sys: 0.03s  user: 11.32s
>> FLOAT fixed:			real: 4.39s  sys: 0.02s  user: 4.10s
>> FLOAT fixed, unsafe		real: 3.44s  sys: 0.01s  user: 3.33s
>> FLOAT fixed, unsafe, $OFP	real: 3.02s  sys: 0.01s  user: 2.92s
>>
>> ===================================================================== 
>> =
>> pnpoly:
>>
>> Marc's time:			real: 16.39s  sys: 0.02s  user: 15.79s
>> FLOAT:				real: 15.81s  sys: 0.02s  user: 14.50s
>> FLOAT, unsafe:			real: 1.86s  sys: 0.01s  user: 1.82s
>> FLOAT, unsafe, $OFP:		real: 1.49s  sys: 0.00s  user: 1.47s
>>
>>
>> ===================================================================== 
>> =
>> sum:
>>
>> Marc's time:			real: 5.44s  sys: 0.01s  user: 5.36s
>> FLOAT, unsafe, $OFP:		real: 5.29s  sys: 0.01s  user: 5.22s
>> FIXNUM, unsafe, $OFP:		real: 0.29s  sys: 0.00s  user: 0.27s
>>
>> ===================================================================== 
>> =
>> sumfp:
>>
>> Marc's time:			real: 9.76s  sys: 0.01s  user: 9.57s
>> FLOAT:				real: 0.55s  sys: 0.01s  user: 0.52s
>> FLOAT, unsafe, $OFP:		real: 0.54s  sys: 0.00s  user: 0.52s
>> -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|--- 
>> -
>> -|-----
>>
>> As you can notice, changing the compilation options and improving  
>> your
>> prefix-bigloo.scm file has a huge impact. As you can notice some
>> benchmarks now go 18 times faster!
>>
>> Here is a description of the modiication I have applied:
>>
>> 1- I have re-implemented your macros FLOATxxx that were badling
>> implemented
>> for Bigloo. They used to expanse into call to the generic
>> functions. They
>> now expanse to the floating point operator. The improvement for
>> benchmark
>> using floating applications is in between 2 and 5 times faster.
>>
>> 2- I have used the proper benchmarking compilation options.
>>   a- The produced C code must be compiled with -fomit-frame-pointer
>> (when
>>      using GCC).
>>   b- The option -unsafe has to be used.
>>
>> 3- For fixnum benchmarks, I have replaced the call to the generic
>> functions
>> with calls to fixnum functions.
>>
>> Please, find enclosed the new version of prefix-bigloo.scm.  I would
>> appreciate if you could re-execute the Bigloo benchmarks with the new
>> prefix and update your web pages with the new Bigloo results. Thanks
>> in advance.
>>
>> -- 
>> Manuel
>>
>> -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|--- 
>> -
>> -|-----
>> (module prefix (main main-entry)
>>   (extern (include "sys/times.h"))
>>   (extern (include "time.h"))
>>   (option (set! *unsafe-rgc*     #t))
>>   (option (set! *unsafe-library* #t))
>>   (option (set! *unsafe-arity*   #t))
>>   (option (set! *unsafe-type*    #t))
>>   (option (set! *unsafe-struct*  #t))
>>   (option (set! *unsafe-range*   #t))
>>   (option (set! *unsafe-version* #t))
>>   (option (set! *cc-options*
>> 		(string-append "-fomit-frame-pointer " *cc-options*))))
>>
>> (define (clock-to-msecs x)
>>   (quotient (* x 1000) (pragma::int "CLOCKS_PER_SEC")))
>>
>> (define (process-times)
>>   (let ()
>>     (pragma "struct tms buf")
>>     (let ((real::long (pragma::long "times(&buf)")))
>>       (cons real
>>             (+ (pragma::long "buf.tms_utime")
>>                (pragma::long "buf.tms_stime"))))))
>>
>> (define (time* thunk)
>>    (let ((start (process-times)))
>>      (let ((result (thunk)))
>>        (let ((end (process-times)))
>>          (let ((cpu (clock-to-msecs (- (cdr end) (cdr start))))
>>                (real (clock-to-msecs (- (car end) (car start)))))
>>            (display "cpu time: ")
>>            (display cpu)
>>            (display " real time: ")
>>            (display real)
>>            (newline)
>>            result)))))
>>
>> (define (run-benchmark name count ok? run-maker . args)
>>   (let* ((run (apply run-maker args))
>>          (result (time* (lambda () (run-bench name count ok? run)))))
>>     (if (not (ok? result))
>>       (begin
>>         (display "*** wrong result ***")
>>         (newline)
>>         (display "*** got: ")
>>         (write result)
>>         (newline)))))
>>
>> (define (run-bench name count ok? run)
>>   (let loop ((i 0) (result (list 'undefined)))
>>     (if (< i count)
>>       (loop (+ i 1) (run))
>>       result)))
>>
>> (define (fatal-error . args)
>>   (for-each display args)
>>   (newline)
>>   (exit 1))
>>
>> (define (main-entry args)
>>   (main))
>>
>> ;-------------------------------------------------------------------- 
>> -
>> ---------
>>
>> ; Macros...
>>
>> (define-macro (def-macro form . body)
>>   `(define-macro ,form (let () , at body)))
>>
>> ; Flonum arithmetic.
>>
>> (begin
>> (def-macro (FLOATvector-const . lst)   `',(list->vector lst))
>> (def-macro (FLOATvector? x)            `(vector? ,x))
>> (def-macro (FLOATvector . lst)         `(vector , at lst))
>> (def-macro (FLOATmake-vector n . init) `(make-vector ,n , at init))
>> (def-macro (FLOATvector-ref v i)       `(vector-ref ,v ,i))
>> (def-macro (FLOATvector-set! v i x)    `(vector-set! ,v ,i ,x))
>> (def-macro (FLOATvector-length v)      `(vector-length ,v))
>>
>> (def-macro (nuc-const . lst)
>>   `',(list->vector lst))
>> )
>>
>> (begin
>> (def-macro (+ . lst)
>>    (if (null? (cdr lst)) (car lst) `(+fx ,(car lst) (+ ,@(cdr  
>> lst)))))
>> (def-macro (- . lst)
>>    (if (null? (cdr lst)) (car lst) `(-fx ,(car lst) (- ,@(cdr  
>> lst)))))
>> (def-macro (* . lst)
>>    (if (null? (cdr lst)) (car lst) `(*fx ,(car lst) (* ,@(cdr  
>> lst)))))
>> (def-macro (/ . lst)
>>    (if (null? (cdr lst)) (car lst) `(/fx ,(car lst) (/ ,@(cdr  
>> lst)))))
>>
>> (def-macro (FLOAT+ . lst)
>>    (if (null? (cdr lst)) (car lst) `(+fl ,(car lst) (FLOAT+ ,@(cdr
>> lst)))))
>> (def-macro (FLOAT- . lst)
>>    (if (null? (cdr lst)) (car lst) `(-fl ,(car lst) (FLOAT- ,@(cdr
>> lst)))))
>> (def-macro (FLOAT* . lst)
>>    (if (null? (cdr lst)) (car lst) `(*fl ,(car lst) (FLOAT* ,@(cdr
>> lst)))))
>> (def-macro (FLOAT/ . lst)
>>    (if (null? (cdr lst)) (car lst) `(/fl ,(car lst) (FLOAT/ ,@(cdr
>> lst)))))
>> (def-macro (FLOAT= a b)  `(=fl ,a ,b))
>> (def-macro (FLOAT< a b)  `(<fl ,a ,b))
>> (def-macro (FLOAT<= a b) `(<= ,a ,b))
>> (def-macro (FLOAT> a b)  `(> ,a ,b))
>> (def-macro (FLOAT>= a b) `(>= ,a ,b))
>> (def-macro (FLOATnegative? a) `(negativefl? ,a))
>> (def-macro (FLOATpositive? a) `(positivefl? ,a))
>> (def-macro (FLOATzero? a)     `(zerofl? ,a))
>> (def-macro (FLOATabs a) `(abs ,a))
>> (def-macro (FLOATsin a) `(sin ,a))
>> (def-macro (FLOATcos a) `(cos ,a))
>> (def-macro (FLOATatan a) `(atan ,a))
>> (def-macro (FLOATsqrt a) `(sqrt ,a))
>> (def-macro (FLOATmin a) `(min ,a))
>> (def-macro (FLOATmax a) `(max ,a))
>> (def-macro (FLOATround a) `(round ,a))
>> (def-macro (FLOATinexact->exact a) `(inexact->exact ,a))
>> )
>>
>> ; Generic arithmetic.
>>
>> (begin
>> (def-macro (GENERIC+ . lst) `(+ , at lst))
>> (def-macro (GENERIC- . lst) `(- , at lst))
>> (def-macro (GENERIC* . lst) `(* , at lst))
>> (def-macro (GENERIC/ . lst) `(/ , at lst))
>> (def-macro (GENERICquotient . lst)  `(quotient , at lst))
>> (def-macro (GENERICremainder . lst) `(remainder , at lst))
>> (def-macro (GENERICmodulo . lst)    `(modulo , at lst))
>> (def-macro (GENERIC= . lst)  `(= , at lst))
>> (def-macro (GENERIC< . lst)  `(< , at lst))
>> (def-macro (GENERIC<= . lst) `(<= , at lst))
>> (def-macro (GENERIC> . lst)  `(> , at lst))
>> (def-macro (GENERIC>= . lst) `(>= , at lst))
>> (def-macro (GENERICexpt . lst) `(expt , at lst))
>> )
>>
>> (begin
>> (def-macro (bitwise-or x y) `(bit-or ,x ,y))
>> (def-macro (bitwise-and x y) `(bit-and ,x ,y))
>> (def-macro (bitwise-not x) `(bit-not ,x))
>> )
>> -----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|--- 
>> -
>> -|-----
>>
>> ps: benchmarks using vector of float could be improved by using  
>> Bigloo
>> typed vectors. I have not tried to apply this because as I said, I
>> don't want
>> to allocate to much time to benchmarking. However, if you want
>> accurate
>> Bigloo measurement. You should try to add the following to
>> prefix-bigloo.scm:
>>
>>
>>
>> (module prefix (main main-entry)
>>   (extern (include "sys/times.h"))
>>   (extern (include "time.h"))
>>   ...
>>   (type (tvector f64vector (double))))
>>
>> (def-macro (FLOATvector? x)            `(f64vector? ,x))
>> (def-macro (FLOATvector-ref v i)       `(f64vector-ref ,v ,i))
>> ...
>
> _______________________________________________
> Gambit-list mailing list
> Gambit-list at iro.umontreal.ca
> http://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
>




More information about the Gambit-list mailing list