Hi All,
Ok - that prompted me to do a little speed testing on a large file (6000 odd records). Note that Marc's comma splitter doesn't actually do the parsing properly because many records are split over multiple lines, so I'm guessing this is an upper performance limit on pure scheme?.
wrapped libcsv ~240ms Marc comma splitter ~510ms Phil Bewig csv parser ~1008ms
I've pasted the timing code below.
Cheers,
Phil
(load "feeley-csv") (load "csv") (load "bewig-csv")
(define (test-feeley-speed) (let ((it (csv-make-iterator-feeley "data.csv"))) (let loop ((e (it))) (if (not (eof-object? e)) (loop (it))))))
(define (test-libcsv-speed) (let ((it (csv-make-iterator "data.csv"))) (let loop ((e (it))) (if (not (equal? e '())) (loop (it))))))
(define (test-bewig-speed) (read-all (open-input-file "data.csv") read-csv-record) #t)
(define (tester fn) (for-each (lambda (e) (##gc) (time (fn))) '(1 2 3 4 5)))
(tester test-feeley-speed) ; I get ~510 ms (tester test-libcsv-speed) ; I get ~240 ms (tester test-bewig-speed) ; I get ~1008 ms
Phil Bewig wrote:
I replied to Phil privately, but since others have joined the conversation, I'll respond publicly. I use the code at pbewig.googlepages.com/ProcessingFieldedTextFiles.pdf http://pbewig.googlepages.com/ProcessingFieldedTextFiles.pdf to process comma-delimited files and other text-formatted databases. I've never timed it, or compared it to other code, but it's always been fast enough for what I wanted to do. Let me know if you find it useful.
Phil (another Phil, not the original poster)
On 2/9/07, *Phil Dawes* <pdawes@users.sf.net mailto:pdawes@users.sf.net> wrote:
Marc Feeley wrote: > Interesting. But why would you do it in C (in 600 lines of code) when > you can do it in 20 lines of Scheme? > Ignorance maybe! There's a little more to csv parsing than just splitting on ",", but having said that your code runs a lot quicker than I was expecting. I had tried http://www.neilvandyke.org/csv-scm/ but this was really slow so turned my sights to wrapping a c library. Thanks! Phil _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca <mailto:Gambit-list@iro.umontreal.ca> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list <https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list>