[gambit-list] file read performance

Phil Dawes phil at phildawes.net
Sat Sep 9 05:08:45 EDT 2006


Hi Marc,

Thanks for looking at this (and giving me insight into what's happening).

I think you're right about python - AFAICS python strings contain
encoding information and are not translated into unicode when loading
from file. I think 'print' does something similar for output - it
decodes the string into the output encoding used by the
terminal/whatever, so if that's the same as the encoding used in the
file then it never gets translated.

Python has a seperate 'unicode' type. Translating each line of the file
into unicode generates timings closer to the gambit version:

t = time.time()
map(unicode,file("bib").readlines())
print >>sys.stderr, time.time() - t

Many thanks,

Phil



Marc Feeley wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 4-Sep-06, at 4:35 AM, phil at phildawes.net wrote:
> 
>> Hi Gambit List,
>>
>> What's the quickest way of reading a file from disk into memory?
>>
>> I tried 'read-all' with 'read-line', but unfortunately even with 
>> compiling I get an order of magnitude worse performance than doing the 
>> same in python.
>> I'm using the following code, compiled using gsc and gcc -O3, and I 
>> get 500ms to read the 'bib' file supplied in the bench.tgz package.
>>
>> ----------------------
>>
>> (define (f)
>>   (call-with-input-file "bib"
>>     (lambda (p) (read-all p read-line))))
>>
>> (time (f))
>>
>> ------------------------
>>
>> In python, the same takes about 50ms
>> e.g.
>> for line in file("bib").readlines():
>>     # do something
>>
>> Is there anything I can do to speed this up?
>> (I've tried setting
>> #define ___MAX_CHR 0xff
>> )
>>
>> Many thanks,
>>
>> Phil
> 
> It depends what you mean by "reading into memory".  You might want to 
> slirp the content of the file into a single u8vector (as a sequence of 
> bytes), or you might want to get the list of strings corresponding to 
> the lines in the file (which requires a byte to character conversion).  
> On my MacBook Pro, the former takes about 60 milliseconds for the "bib" 
> file and the latter about 200 milliseconds (I also get about 50 ms for 
> the Python code).  I tried to improve the time with a specialized 
> buffering but that only improved the timing by 20%.  Using 1 byte 
> characters instead of 4 byte characters helps, but it is still roughly 
> 2.5 times slower than slirping the bytes into a u8vector.  If you 
> iterate using read-line (instead of using read-all) you save some more 
> because the list of lines does not have to be kept in memory all the 
> time.  See the code below.
> 
> I think the largest cost here is the allocation of the strings and 
> conversion from bytes to characters.  I'm not sure if the conversion to 
> characters is a cost that Python is paying (can the text be encoded in 
> something other than ISO-8859-1? that's something that you can easily 
> change with Gambit's I/O system).
> 
> Marc
> 
> 
> (declare (standard-bindings) (block) (fixnum) (not safe))
> 
> (define (read-all-file-v1 filename)
>   (let* ((size (file-size filename))
>          (buffer (make-u8vector size)))
>     (call-with-input-file filename
>       (lambda (in)
>         (let ((n (read-subu8vector buffer 0 size in)))
>           buffer)))))
> 
> (define (read-all-file-v2 filename)
>   (call-with-input-file filename
>     (lambda (in)
>       (read-all in read-line))))
> 
> (define (read-all-file-v3 filename)
>   (call-with-input-file filename
>     (lambda (in)
>       (let* ((size 4096)
>              (buffer (make-string (+ size 1))))
>         (let loop1 ((rev-lines '())
>                     (incomplete-line #f))
>           (let ((n (read-substring buffer 0 size in)))
>             (string-set! buffer n #\newline)
>             (if (= n 0)
>                 (reverse (if incomplete-line
>                              (cons incomplete-line rev-lines)
>                              rev-lines))
>                 (let loop2 ((i 0)
>                             (rev-lines rev-lines)
>                             (incomplete-line incomplete-line))
>                   (let loop3 ((j i))
>                     (if (char=? (string-ref buffer j) #\newline)
>                         (if (= j n)
>                             (loop1 rev-lines
>                                    (cond ((= i j)
>                                           incomplete-line)
>                                          (incomplete-line
>                                           (string-append
>                                            incomplete-line
>                                            (substring buffer i j)))
>                                          (else
>                                           (substring buffer i j))))
>                             (loop2 (+ j 1)
>                                    (cons (if incomplete-line
>                                              (string-append
>                                               incomplete-line
>                                               (substring buffer i j))
>                                              (substring buffer i j))
>                                          rev-lines)
>                                    #f))
>                         (loop3 (+ j 1))))))))))))
> 
> (define (read-all-file-v4 filename)
>   (call-with-input-file filename
>     (lambda (in)
>       (let loop ()
>         (let ((line (read-line in)))
>           (if (string? line)
>               (begin
>                 ; do something with line
>                 (loop))))))))
> 
> (define filename "bench/src/bib")
> 
> ; Execution time with character size = 4 bytes  1 byte
> (time (read-all-file-v1 filename)) ;    57 ms    57 ms
> (time (read-all-file-v2 filename)) ;   229 ms   170 ms
> (time (read-all-file-v3 filename)) ;   181 ms   146 ms
> (time (read-all-file-v4 filename)) ;   162 ms   130 ms
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (Darwin)
> 
> iD8DBQFFAafz//V9Zc2T/v4RAr97AJ43/61utp7eFKE7M2f3AuaUBNs/aACgvw+j
> 7m+CVSFm+kqVjkRtIRgm5Lg=
> =+0j4
> -----END PGP SIGNATURE-----
> 






More information about the Gambit-list mailing list