[gambit-list] file read performance

Marc Feeley feeley at iro.umontreal.ca
Fri Sep 8 13:27:15 EDT 2006


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 4-Sep-06, at 4:35 AM, phil at phildawes.net wrote:

> Hi Gambit List,
>
> What's the quickest way of reading a file from disk into memory?
>
> I tried 'read-all' with 'read-line', but unfortunately even with  
> compiling I get an order of magnitude worse performance than doing  
> the same in python.
> I'm using the following code, compiled using gsc and gcc -O3, and I  
> get 500ms to read the 'bib' file supplied in the bench.tgz package.
>
> ----------------------
>
> (define (f)
>   (call-with-input-file "bib"
>     (lambda (p) (read-all p read-line))))
>
> (time (f))
>
> ------------------------
>
> In python, the same takes about 50ms
> e.g.
> for line in file("bib").readlines():
>     # do something
>
> Is there anything I can do to speed this up?
> (I've tried setting
> #define ___MAX_CHR 0xff
> )
>
> Many thanks,
>
> Phil

It depends what you mean by "reading into memory".  You might want to  
slirp the content of the file into a single u8vector (as a sequence  
of bytes), or you might want to get the list of strings corresponding  
to the lines in the file (which requires a byte to character  
conversion).  On my MacBook Pro, the former takes about 60  
milliseconds for the "bib" file and the latter about 200 milliseconds  
(I also get about 50 ms for the Python code).  I tried to improve the  
time with a specialized buffering but that only improved the timing  
by 20%.  Using 1 byte characters instead of 4 byte characters helps,  
but it is still roughly 2.5 times slower than slirping the bytes into  
a u8vector.  If you iterate using read-line (instead of using read- 
all) you save some more because the list of lines does not have to be  
kept in memory all the time.  See the code below.

I think the largest cost here is the allocation of the strings and  
conversion from bytes to characters.  I'm not sure if the conversion  
to characters is a cost that Python is paying (can the text be  
encoded in something other than ISO-8859-1? that's something that you  
can easily change with Gambit's I/O system).

Marc


(declare (standard-bindings) (block) (fixnum) (not safe))

(define (read-all-file-v1 filename)
   (let* ((size (file-size filename))
          (buffer (make-u8vector size)))
     (call-with-input-file filename
       (lambda (in)
         (let ((n (read-subu8vector buffer 0 size in)))
           buffer)))))

(define (read-all-file-v2 filename)
   (call-with-input-file filename
     (lambda (in)
       (read-all in read-line))))

(define (read-all-file-v3 filename)
   (call-with-input-file filename
     (lambda (in)
       (let* ((size 4096)
              (buffer (make-string (+ size 1))))
         (let loop1 ((rev-lines '())
                     (incomplete-line #f))
           (let ((n (read-substring buffer 0 size in)))
             (string-set! buffer n #\newline)
             (if (= n 0)
                 (reverse (if incomplete-line
                              (cons incomplete-line rev-lines)
                              rev-lines))
                 (let loop2 ((i 0)
                             (rev-lines rev-lines)
                             (incomplete-line incomplete-line))
                   (let loop3 ((j i))
                     (if (char=? (string-ref buffer j) #\newline)
                         (if (= j n)
                             (loop1 rev-lines
                                    (cond ((= i j)
                                           incomplete-line)
                                          (incomplete-line
                                           (string-append
                                            incomplete-line
                                            (substring buffer i j)))
                                          (else
                                           (substring buffer i j))))
                             (loop2 (+ j 1)
                                    (cons (if incomplete-line
                                              (string-append
                                               incomplete-line
                                               (substring buffer i j))
                                              (substring buffer i j))
                                          rev-lines)
                                    #f))
                         (loop3 (+ j 1))))))))))))

(define (read-all-file-v4 filename)
   (call-with-input-file filename
     (lambda (in)
       (let loop ()
         (let ((line (read-line in)))
           (if (string? line)
               (begin
                 ; do something with line
                 (loop))))))))

(define filename "bench/src/bib")

; Execution time with character size = 4 bytes  1 byte
(time (read-all-file-v1 filename)) ;    57 ms    57 ms
(time (read-all-file-v2 filename)) ;   229 ms   170 ms
(time (read-all-file-v3 filename)) ;   181 ms   146 ms
(time (read-all-file-v4 filename)) ;   162 ms   130 ms

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFFAafz//V9Zc2T/v4RAr97AJ43/61utp7eFKE7M2f3AuaUBNs/aACgvw+j
7m+CVSFm+kqVjkRtIRgm5Lg=
=+0j4
-----END PGP SIGNATURE-----



More information about the Gambit-list mailing list