[gambit-list] fast binary serialization

lowly coder lowlycoder at huoyanjinjing.com
Mon Feb 16 21:40:25 EST 2009


I'm really glad you responded -- I wasn't aware gambit-list blocked
attachments




$ cat bench.scm:
(define (gen-lst min max)
  (define (helper min max cur)
    (if (< min max)
        (helper min (- max 1) (cons max cur))
        cur))
  (helper (- min 1) (- max 1) '()))

(define (pad-with value amount lst)
  (define (helper amount lst)
    (if (<= amount 0) lst
        (helper (- amount 1) (cons value lst))))
  (helper amount lst))

(define (pad-to value ttl-length lst)
  (pad-with value (- ttl-length (length lst)) lst))

(define (convert-to-base value base)
  (define (helper value cur)
    (if (= value 0) cur
        (helper (quotient value base) (cons (remainder value base) cur))))
  (helper value '()))

(define (base-to-int lst base)
  (define (helper lst cur)
    (if (null? lst) cur
        (helper (cdr lst) (+ (* cur base) (car lst)))))
  (helper lst 0))

(define (int-as-length8-uvector value)
  (list->u8vector (pad-to 0 8 (convert-to-base value 256))))

(define (length8-uvector-as-int vec)
  (base-to-int (u8vector->list vec) 256))


;(pp (base-to-int (convert-to-base 123456 10) 10))

;(pp (list->u8vector '(1 2 3 4 5 255)))

(define (make-timer)
  (let ((start-time 0))
    (define (dispatch cmd)
      (cond ((equal? cmd 'start)
              (set! start-time (time->seconds (current-time))))
             ((equal? cmd 'stop)
              (- (time->seconds (current-time)) start-time))))
    dispatch))

(define my-timer (make-timer))

                                        ;(pp (gen-lst 0 10))

(define (serialize-text object filename)
  (call-with-output-file filename
    (lambda (file)
      (pp object file))))

(define (serialize-binary object filename)
  (call-with-output-file filename
    (lambda (file)
      (let ((as-vec (object->u8vector object)))
        (write-subu8vector (int-as-length8-uvector (u8vector-length as-vec))
0 8 file)
        (write-subu8vector as-vec 0 (u8vector-length as-vec) file)))))

(define (unserialize-binary filename)
  (call-with-input-file filename
    (lambda (file)
      (let* ((vec (make-u8vector 8)))
        (read-subu8vector vec 0 8 file)
        (let* ((len (length8-uvector-as-int vec))
               (obj-vec (make-u8vector len)))
          (read-subu8vector obj-vec 0 len file)
          (u8vector->object obj-vec))))))




(define (bench number-of-entries)

  (define object (gen-lst 0 number-of-entries))

  (serialize-text object "plain_text")
  (serialize-binary object "binary")

  (let ((plain-time 0)
        (binary-time 0))


    (my-timer 'start)
    (unserialize-binary "binary")
    (set! binary-time (my-timer 'stop))

    (my-timer 'start)
    (read-all (open-input-file "plain_text"))
    (set! plain-time (my-timer 'stop))

    (pp `(number-of-entries: ,number-of-entries (binary: ,binary-time)
(plain: ,plain-time)))))


(bench 1)
(bench 10)
(bench 100)
(bench 1000)
(bench 10000)
(bench 100000)
(bench 1000000)
(bench 10000000)



$ gsi bench.scm
(number-of-entries: 1 (binary: 5.1021575927734375e-5) (plain:
3.0040740966796875e-5))
(number-of-entries: 10 (binary: 3.0040740966796875e-5) (plain:
2.4080276489257812e-5))
(number-of-entries: 100 (binary: 4.792213439941406e-5) (plain:
5.698204040527344e-5))
(number-of-entries: 1000 (binary: 2.8395652770996094e-4) (plain:
4.1794776916503906e-4))
(number-of-entries: 10000 (binary: .004720926284790039) (plain:
.009029150009155273))
(number-of-entries: 100000 (binary: .07220315933227539) (plain:
.052423954010009766))
(number-of-entries: 1000000 (binary: 1.0091660022735596) (plain:
.6231379508972168))
(number-of-entries: 10000000 (binary: 9.397350072860718) (plain:
17.698192834854126))



On Mon, Feb 16, 2009 at 6:22 PM, Adrien Piérard
<pierarda at iro.umontreal.ca>wrote:

> For most of them, plain_text appears to be faster; and only at 10 million
>> entries, does the binary format become slightly faster. What am I doing
>> wrong?
>>
>
> You are not showing us the code. That is just *wrong*.
>
> Are you converting each number independently, or the whole list at once?
>
> P!
>
> --
> Français, English, 日本語, 한국어
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.iro.umontreal.ca/pipermail/gambit-list/attachments/20090216/fcd45fd6/attachment.htm>


More information about the Gambit-list mailing list