[gambit-list] Fwd: Gambit-C vs. Bigloo vs. MzScheme

Manuel Serrano Manuel.Serrano at sophia.inria.fr
Sun Aug 27 05:11:56 EDT 2006

On Sat, 26 Aug 2006 21:31:16 -0400
Marc Feeley <feeley at iro.umontreal.ca> wrote:

> Désolé j'ai oublié de te mettre en CC.
> Marc
> Begin forwarded message:
> > From: Marc Feeley <feeley at iro.umontreal.ca>
> > Date: August 26, 2006 9:28:15 PM EDT (CA)
> > To: Gambit List <Gambit-list at iro.umontreal.ca>
> > Cc: Eli Barzilay <eli at barzilay.org>, Matthew Flatt  
> > <mflatt at cs.utah.edu>
> > Subject: Gambit-C vs. Bigloo vs. MzScheme
> >
> > After I saw Eli Barzilay's post to comp.lang.scheme about the  
> > performance of MzScheme compared to other Scheme systems (http:// 
> > groups.google.com/group/comp.lang.scheme/msg/473f91e7065b6f73? 
> > dmode=source) I was tempted to do some benchmarking of my own
> > using the Gambit Scheme benchmark suite which goes beyond the
> > classical Gabriel benchmarks (it has close to 60 Scheme
> > benchmarks).  Most benchmarks are straightforward R4RS code and so
> > should work on just about any Scheme system.
> >
> > I have updated the benchmarking infrastructure to accomodate other  
> > Scheme systems, and MzScheme and Bigloo in particular.  I also  
> > wrote a script to run all benchmarks and automatically generate an  
> > HTML table of the results.  The table can be viewed here:
> >
> >    http://www.iro.umontreal.ca/~gambit/bench.html
> >
> > And the benchmark suite is available here:
> >
> >    http://www.iro.umontreal.ca/~gambit/bench.tar.gz
> >
> > As you can see from the table, Gambit-C is faster than Bigloo and  
> > MzScheme on roughly 3 out of 4 benchmarks.  I have used the same  
> > assumptions as Eli, which approximate the semantics of R6RS (see  
> > the last section of the result page for details).
> >
> > It would be really interesting to extend this experiment to other  
> > Scheme systems.  It should not be too difficult to modify the  
> > benchmarking scripts to test other systems.  The hardest parts are  
> > installing the Scheme system, figuring out which compiler options  
> > are equivalent to the ones used by the other systems (i.e.  
> > approximate R6RS semantics), and work out unexpected problems.
> >
> > The following systems are particularly interesting to compare to:
> >
> >   - Chicken
> >   - Scheme 48
> >   - Larceny
> >   - Chez Scheme
> >
> > Are there any volunteers?
> >
> > Marc

Hello Marc, 

Ah, benchmarking... Definitely we can't escape benchmarks :-). I had more or
less sweared not to benchmark any more because this is an
activity that can consume all your time and for which the result is
not necessarily very useful. However, because I didn't want to spend a
bad Sunday :-) I have downloaded your benchmarks and make some measurements
with Bigloo. Here is what I got:

General command line: bigloo -O6

OFP=-copt -fomit-frame-pointer
fixnum= /+/+fx/, /-/-fx/, ...
unsafe=bigloo -O6 -unsafe
FLOAT= re-implementing FLOAT+, FLOAT-, ...


Marc's time: 		   	real: 11.22s  sys: 0.02s  user: 11.13s
fixnum: 		   	real: 3.61s  sys: 0.01s  user: 3.57s
fixnum, omit-frame-pointer: 	real: 2.80s  sys: 0.01s  user: 2.78s


Marc's time:			real: 11.71s  sys: 0.03s  user: 11.32s
FLOAT fixed:			real: 4.39s  sys: 0.02s  user: 4.10s
FLOAT fixed, unsafe		real: 3.44s  sys: 0.01s  user: 3.33s
FLOAT fixed, unsafe, $OFP	real: 3.02s  sys: 0.01s  user: 2.92s


Marc's time:			real: 16.39s  sys: 0.02s  user: 15.79s
FLOAT:				real: 15.81s  sys: 0.02s  user: 14.50s
FLOAT, unsafe:			real: 1.86s  sys: 0.01s  user: 1.82s
FLOAT, unsafe, $OFP:		real: 1.49s  sys: 0.00s  user: 1.47s


Marc's time:			real: 5.44s  sys: 0.01s  user: 5.36s
FLOAT, unsafe, $OFP:		real: 5.29s  sys: 0.01s  user: 5.22s
FIXNUM, unsafe, $OFP:		real: 0.29s  sys: 0.00s  user: 0.27s


Marc's time:			real: 9.76s  sys: 0.01s  user: 9.57s
FLOAT:				real: 0.55s  sys: 0.01s  user: 0.52s
FLOAT, unsafe, $OFP:		real: 0.54s  sys: 0.00s  user: 0.52s

As you can notice, changing the compilation options and improving your
prefix-bigloo.scm file has a huge impact. As you can notice some
benchmarks now go 18 times faster!

Here is a description of the modiication I have applied:

1- I have re-implemented your macros FLOATxxx that were badling implemented
for Bigloo. They used to expanse into call to the generic functions. They
now expanse to the floating point operator. The improvement for benchmark
using floating applications is in between 2 and 5 times faster.

2- I have used the proper benchmarking compilation options. 
  a- The produced C code must be compiled with -fomit-frame-pointer (when
     using GCC).
  b- The option -unsafe has to be used.

3- For fixnum benchmarks, I have replaced the call to the generic functions
with calls to fixnum functions.

Please, find enclosed the new version of prefix-bigloo.scm.  I would
appreciate if you could re-execute the Bigloo benchmarks with the new
prefix and update your web pages with the new Bigloo results. Thanks
in advance.


(module prefix (main main-entry)
  (extern (include "sys/times.h"))
  (extern (include "time.h"))
  (option (set! *unsafe-rgc*     #t))
  (option (set! *unsafe-library* #t))
  (option (set! *unsafe-arity*   #t))
  (option (set! *unsafe-type*    #t))
  (option (set! *unsafe-struct*  #t))
  (option (set! *unsafe-range*   #t))
  (option (set! *unsafe-version* #t))
  (option (set! *cc-options*
		(string-append "-fomit-frame-pointer " *cc-options*))))

(define (clock-to-msecs x)
  (quotient (* x 1000) (pragma::int "CLOCKS_PER_SEC")))

(define (process-times)
  (let ()
    (pragma "struct tms buf")
    (let ((real::long (pragma::long "times(&buf)")))
      (cons real
            (+ (pragma::long "buf.tms_utime")
               (pragma::long "buf.tms_stime"))))))

(define (time* thunk)
   (let ((start (process-times)))
     (let ((result (thunk)))
       (let ((end (process-times)))
         (let ((cpu (clock-to-msecs (- (cdr end) (cdr start))))
               (real (clock-to-msecs (- (car end) (car start)))))
           (display "cpu time: ")
           (display cpu)
           (display " real time: ")
           (display real)

(define (run-benchmark name count ok? run-maker . args)
  (let* ((run (apply run-maker args))
         (result (time* (lambda () (run-bench name count ok? run)))))
    (if (not (ok? result))
        (display "*** wrong result ***")
        (display "*** got: ")
        (write result)

(define (run-bench name count ok? run)
  (let loop ((i 0) (result (list 'undefined)))
    (if (< i count)
      (loop (+ i 1) (run))

(define (fatal-error . args)
  (for-each display args)
  (exit 1))

(define (main-entry args)


; Macros...

(define-macro (def-macro form . body)
  `(define-macro ,form (let () , at body)))

; Flonum arithmetic.

(def-macro (FLOATvector-const . lst)   `',(list->vector lst))
(def-macro (FLOATvector? x)            `(vector? ,x))
(def-macro (FLOATvector . lst)         `(vector , at lst))
(def-macro (FLOATmake-vector n . init) `(make-vector ,n , at init))
(def-macro (FLOATvector-ref v i)       `(vector-ref ,v ,i))
(def-macro (FLOATvector-set! v i x)    `(vector-set! ,v ,i ,x))
(def-macro (FLOATvector-length v)      `(vector-length ,v))

(def-macro (nuc-const . lst)
  `',(list->vector lst))

(def-macro (+ . lst)
   (if (null? (cdr lst)) (car lst) `(+fx ,(car lst) (+ ,@(cdr lst)))))
(def-macro (- . lst) 
   (if (null? (cdr lst)) (car lst) `(-fx ,(car lst) (- ,@(cdr lst)))))
(def-macro (* . lst) 
   (if (null? (cdr lst)) (car lst) `(*fx ,(car lst) (* ,@(cdr lst)))))
(def-macro (/ . lst)
   (if (null? (cdr lst)) (car lst) `(/fx ,(car lst) (/ ,@(cdr lst)))))
(def-macro (FLOAT+ . lst)
   (if (null? (cdr lst)) (car lst) `(+fl ,(car lst) (FLOAT+ ,@(cdr lst)))))
(def-macro (FLOAT- . lst) 
   (if (null? (cdr lst)) (car lst) `(-fl ,(car lst) (FLOAT- ,@(cdr lst)))))
(def-macro (FLOAT* . lst) 
   (if (null? (cdr lst)) (car lst) `(*fl ,(car lst) (FLOAT* ,@(cdr lst)))))
(def-macro (FLOAT/ . lst)
   (if (null? (cdr lst)) (car lst) `(/fl ,(car lst) (FLOAT/ ,@(cdr lst)))))
(def-macro (FLOAT= a b)  `(=fl ,a ,b))
(def-macro (FLOAT< a b)  `(<fl ,a ,b))
(def-macro (FLOAT<= a b) `(<= ,a ,b))
(def-macro (FLOAT> a b)  `(> ,a ,b))
(def-macro (FLOAT>= a b) `(>= ,a ,b))
(def-macro (FLOATnegative? a) `(negativefl? ,a))
(def-macro (FLOATpositive? a) `(positivefl? ,a))
(def-macro (FLOATzero? a)     `(zerofl? ,a))
(def-macro (FLOATabs a) `(abs ,a))
(def-macro (FLOATsin a) `(sin ,a))
(def-macro (FLOATcos a) `(cos ,a))
(def-macro (FLOATatan a) `(atan ,a))
(def-macro (FLOATsqrt a) `(sqrt ,a))
(def-macro (FLOATmin a) `(min ,a))
(def-macro (FLOATmax a) `(max ,a))
(def-macro (FLOATround a) `(round ,a))
(def-macro (FLOATinexact->exact a) `(inexact->exact ,a))

; Generic arithmetic.

(def-macro (GENERIC+ . lst) `(+ , at lst))
(def-macro (GENERIC- . lst) `(- , at lst))
(def-macro (GENERIC* . lst) `(* , at lst))
(def-macro (GENERIC/ . lst) `(/ , at lst))
(def-macro (GENERICquotient . lst)  `(quotient , at lst))
(def-macro (GENERICremainder . lst) `(remainder , at lst))
(def-macro (GENERICmodulo . lst)    `(modulo , at lst))
(def-macro (GENERIC= . lst)  `(= , at lst))
(def-macro (GENERIC< . lst)  `(< , at lst))
(def-macro (GENERIC<= . lst) `(<= , at lst))
(def-macro (GENERIC> . lst)  `(> , at lst))
(def-macro (GENERIC>= . lst) `(>= , at lst))
(def-macro (GENERICexpt . lst) `(expt , at lst))

(def-macro (bitwise-or x y) `(bit-or ,x ,y))
(def-macro (bitwise-and x y) `(bit-and ,x ,y))
(def-macro (bitwise-not x) `(bit-not ,x))

ps: benchmarks using vector of float could be improved by using Bigloo
typed vectors. I have not tried to apply this because as I said, I don't want
to allocate to much time to benchmarking. However, if you want accurate
Bigloo measurement. You should try to add the following to 

(module prefix (main main-entry)
  (extern (include "sys/times.h"))
  (extern (include "time.h"))
  (type (tvector f64vector (double))))

(def-macro (FLOATvector? x)            `(f64vector? ,x))
(def-macro (FLOATvector-ref v i)       `(f64vector-ref ,v ,i))

More information about the Gambit-list mailing list