[gambit-list] fast csv parser (libcsv wrapped for gambit)

Phil Dawes pdawes at users.sf.net
Fri Feb 9 13:36:53 EST 2007

Hi All,

Ok - that prompted me to do a little speed testing on a large file (6000 
odd records). Note that Marc's comma splitter doesn't actually do the 
parsing properly because many records are split over multiple lines, so 
I'm guessing this is an upper performance limit on pure scheme?.

wrapped libcsv  ~240ms
Marc comma splitter  ~510ms
Phil Bewig csv parser  ~1008ms

I've pasted the timing code below.



(load "feeley-csv")
(load "csv")
(load "bewig-csv")

(define (test-feeley-speed)
   (let ((it (csv-make-iterator-feeley "data.csv")))
     (let loop ((e (it)))
       (if (not (eof-object? e))
           (loop (it))))))

(define (test-libcsv-speed)
   (let ((it (csv-make-iterator "data.csv")))
     (let loop ((e (it)))
       (if (not (equal? e '()))
           (loop (it))))))

(define (test-bewig-speed)
   (read-all (open-input-file "data.csv") read-csv-record)

(define (tester fn)
   (for-each (lambda (e) (##gc) (time (fn))) '(1 2 3 4 5)))

(tester test-feeley-speed)     ; I get ~510 ms
(tester test-libcsv-speed)     ; I get ~240 ms
(tester test-bewig-speed)      ; I get ~1008 ms

Phil Bewig wrote:
> I replied to Phil privately, but since others have joined the 
> conversation, I'll respond publicly.  I use the code at 
> pbewig.googlepages.com/ProcessingFieldedTextFiles.pdf 
> <http://pbewig.googlepages.com/ProcessingFieldedTextFiles.pdf> to 
> process comma-delimited files and other text-formatted databases.  I've 
> never timed it, or compared it to other code, but it's always been fast 
> enough for what I wanted to do.  Let me know if you find it useful.
> Phil (another Phil, not the original poster)
> On 2/9/07, *Phil Dawes* <pdawes at users.sf.net 
> <mailto:pdawes at users.sf.net>> wrote:
>     Marc Feeley wrote:
>      > Interesting.  But why would you do it in C (in 600 lines of code)
>     when
>      > you can do it in 20 lines of Scheme?
>      >
>     Ignorance maybe!
>     There's a little more to csv parsing than just splitting on ",", but
>     having said that your code runs a lot quicker than I was expecting.
>     I had tried http://www.neilvandyke.org/csv-scm/ but this was really slow
>     so turned my sights to wrapping a c library.
>     Thanks!
>     Phil
>     _______________________________________________
>     Gambit-list mailing list
>     Gambit-list at iro.umontreal.ca <mailto:Gambit-list at iro.umontreal.ca>
>     https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
>     <https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list>

More information about the Gambit-list mailing list