Hi Marc, Thanks for looking at this (and giving me insight into what's happening). I think you're right about python - AFAICS python strings contain encoding information and are not translated into unicode when loading from file. I think 'print' does something similar for output - it decodes the string into the output encoding used by the terminal/whatever, so if that's the same as the encoding used in the file then it never gets translated. Python has a seperate 'unicode' type. Translating each line of the file into unicode generates timings closer to the gambit version: t = time.time() map(unicode,file("bib").readlines()) print >>sys.stderr, time.time() - t Many thanks, Phil Marc Feeley wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 4-Sep-06, at 4:35 AM, phil@phildawes.net wrote:

...

Hi Gambit List,

What's the quickest way of reading a file from disk into memory?

I tried 'read-all' with 'read-line', but unfortunately even with compiling I get an order of magnitude worse performance than doing the same in python. I'm using the following code, compiled using gsc and gcc -O3, and I get 500ms to read the 'bib' file supplied in the bench.tgz package.

----------------------

(define (f) (call-with-input-file "bib" (lambda (p) (read-all p read-line))))

(time (f))

------------------------

In python, the same takes about 50ms e.g. for line in file("bib").readlines(): # do something

Is there anything I can do to speed this up? (I've tried setting #define ___MAX_CHR 0xff )

Many thanks,

Phil

It depends what you mean by "reading into memory". You might want to slirp the content of the file into a single u8vector (as a sequence of bytes), or you might want to get the list of strings corresponding to the lines in the file (which requires a byte to character conversion). On my MacBook Pro, the former takes about 60 milliseconds for the "bib" file and the latter about 200 milliseconds (I also get about 50 ms for the Python code). I tried to improve the time with a specialized buffering but that only improved the timing by 20%. Using 1 byte characters instead of 4 byte characters helps, but it is still roughly 2.5 times slower than slirping the bytes into a u8vector. If you iterate using read-line (instead of using read-all) you save some more because the list of lines does not have to be kept in memory all the time. See the code below.

I think the largest cost here is the allocation of the strings and conversion from bytes to characters. I'm not sure if the conversion to characters is a cost that Python is paying (can the text be encoded in something other than ISO-8859-1? that's something that you can easily change with Gambit's I/O system).

Marc

(declare (standard-bindings) (block) (fixnum) (not safe))

(define (read-all-file-v1 filename) (let* ((size (file-size filename)) (buffer (make-u8vector size))) (call-with-input-file filename (lambda (in) (let ((n (read-subu8vector buffer 0 size in))) buffer)))))

(define (read-all-file-v2 filename) (call-with-input-file filename (lambda (in) (read-all in read-line))))

(define (read-all-file-v3 filename) (call-with-input-file filename (lambda (in) (let* ((size 4096) (buffer (make-string (+ size 1)))) (let loop1 ((rev-lines '()) (incomplete-line #f)) (let ((n (read-substring buffer 0 size in))) (string-set! buffer n #\newline) (if (= n 0) (reverse (if incomplete-line (cons incomplete-line rev-lines) rev-lines)) (let loop2 ((i 0) (rev-lines rev-lines) (incomplete-line incomplete-line)) (let loop3 ((j i)) (if (char=? (string-ref buffer j) #\newline) (if (= j n) (loop1 rev-lines (cond ((= i j) incomplete-line) (incomplete-line (string-append incomplete-line (substring buffer i j))) (else (substring buffer i j)))) (loop2 (+ j 1) (cons (if incomplete-line (string-append incomplete-line (substring buffer i j)) (substring buffer i j)) rev-lines) #f)) (loop3 (+ j 1))))))))))))

(define (read-all-file-v4 filename) (call-with-input-file filename (lambda (in) (let loop () (let ((line (read-line in))) (if (string? line) (begin ; do something with line (loop))))))))

(define filename "bench/src/bib")

; Execution time with character size = 4 bytes 1 byte (time (read-all-file-v1 filename)) ; 57 ms 57 ms (time (read-all-file-v2 filename)) ; 229 ms 170 ms (time (read-all-file-v3 filename)) ; 181 ms 146 ms (time (read-all-file-v4 filename)) ; 162 ms 130 ms

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin)

iD8DBQFFAafz//V9Zc2T/v4RAr97AJ43/61utp7eFKE7M2f3AuaUBNs/aACgvw+j 7m+CVSFm+kqVjkRtIRgm5Lg= =+0j4 -----END PGP SIGNATURE-----