[gambit-list] file read performance
Marc Feeley
feeley at iro.umontreal.ca
Fri Sep 8 13:27:15 EDT 2006
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 4-Sep-06, at 4:35 AM, phil at phildawes.net wrote:
> Hi Gambit List,
>
> What's the quickest way of reading a file from disk into memory?
>
> I tried 'read-all' with 'read-line', but unfortunately even with
> compiling I get an order of magnitude worse performance than doing
> the same in python.
> I'm using the following code, compiled using gsc and gcc -O3, and I
> get 500ms to read the 'bib' file supplied in the bench.tgz package.
>
> ----------------------
>
> (define (f)
> (call-with-input-file "bib"
> (lambda (p) (read-all p read-line))))
>
> (time (f))
>
> ------------------------
>
> In python, the same takes about 50ms
> e.g.
> for line in file("bib").readlines():
> # do something
>
> Is there anything I can do to speed this up?
> (I've tried setting
> #define ___MAX_CHR 0xff
> )
>
> Many thanks,
>
> Phil
It depends what you mean by "reading into memory". You might want to
slirp the content of the file into a single u8vector (as a sequence
of bytes), or you might want to get the list of strings corresponding
to the lines in the file (which requires a byte to character
conversion). On my MacBook Pro, the former takes about 60
milliseconds for the "bib" file and the latter about 200 milliseconds
(I also get about 50 ms for the Python code). I tried to improve the
time with a specialized buffering but that only improved the timing
by 20%. Using 1 byte characters instead of 4 byte characters helps,
but it is still roughly 2.5 times slower than slirping the bytes into
a u8vector. If you iterate using read-line (instead of using read-
all) you save some more because the list of lines does not have to be
kept in memory all the time. See the code below.
I think the largest cost here is the allocation of the strings and
conversion from bytes to characters. I'm not sure if the conversion
to characters is a cost that Python is paying (can the text be
encoded in something other than ISO-8859-1? that's something that you
can easily change with Gambit's I/O system).
Marc
(declare (standard-bindings) (block) (fixnum) (not safe))
(define (read-all-file-v1 filename)
(let* ((size (file-size filename))
(buffer (make-u8vector size)))
(call-with-input-file filename
(lambda (in)
(let ((n (read-subu8vector buffer 0 size in)))
buffer)))))
(define (read-all-file-v2 filename)
(call-with-input-file filename
(lambda (in)
(read-all in read-line))))
(define (read-all-file-v3 filename)
(call-with-input-file filename
(lambda (in)
(let* ((size 4096)
(buffer (make-string (+ size 1))))
(let loop1 ((rev-lines '())
(incomplete-line #f))
(let ((n (read-substring buffer 0 size in)))
(string-set! buffer n #\newline)
(if (= n 0)
(reverse (if incomplete-line
(cons incomplete-line rev-lines)
rev-lines))
(let loop2 ((i 0)
(rev-lines rev-lines)
(incomplete-line incomplete-line))
(let loop3 ((j i))
(if (char=? (string-ref buffer j) #\newline)
(if (= j n)
(loop1 rev-lines
(cond ((= i j)
incomplete-line)
(incomplete-line
(string-append
incomplete-line
(substring buffer i j)))
(else
(substring buffer i j))))
(loop2 (+ j 1)
(cons (if incomplete-line
(string-append
incomplete-line
(substring buffer i j))
(substring buffer i j))
rev-lines)
#f))
(loop3 (+ j 1))))))))))))
(define (read-all-file-v4 filename)
(call-with-input-file filename
(lambda (in)
(let loop ()
(let ((line (read-line in)))
(if (string? line)
(begin
; do something with line
(loop))))))))
(define filename "bench/src/bib")
; Execution time with character size = 4 bytes 1 byte
(time (read-all-file-v1 filename)) ; 57 ms 57 ms
(time (read-all-file-v2 filename)) ; 229 ms 170 ms
(time (read-all-file-v3 filename)) ; 181 ms 146 ms
(time (read-all-file-v4 filename)) ; 162 ms 130 ms
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)
iD8DBQFFAafz//V9Zc2T/v4RAr97AJ43/61utp7eFKE7M2f3AuaUBNs/aACgvw+j
7m+CVSFm+kqVjkRtIRgm5Lg=
=+0j4
-----END PGP SIGNATURE-----
More information about the Gambit-list
mailing list