[gambit-list] fast csv parser (libcsv wrapped for gambit)

Phil Dawes pdawes at users.sf.net
Sat Feb 10 03:44:19 EST 2007


Bradley Lucier wrote:
> On Feb 9, 2007, at 5:08 PM, Phil Dawes wrote:
> 
>> That brings the Phil Bewig parser down to ~950 ms
> 
> Ah, not much.
> 
> What happens when you configure gambit with --enable-char-size=1 (the  
> default is 4). (I presume the C you're using has one-byte chars.)
> 

I haven't tried this, but have done a similar thing for text reading 
before without gaining much speed improvement (because you still incur 
byte->char translation). I suspect a faster approach is to do the 
reading in binary mode (i.e. u8vectors)?

> Or you could give a URL for your data for others to play with.
> 

Good idea - here's the data:
http://www.phildawes.net/2007/gambit-csv/data.csv.gz
(I can't remember where I first got it from, but it would have been from 
the web somewhere so hopefully I'm not breaking any copyright).

> (Next I would suggest doing line reads and stepping through the  
> characters locally without going through the trampoline required for  
> each call to read-char.  It seems that doing your own line buffering  
> may be natural for this problem.)

I've done something similar in python before, using a fast regex library 
to first mask out escaped chars, then quoted strings. Then you can look 
at the last char in a line to see if it should be joined to the next 
line (i.e. the newline is within a quote).
Note that libcsv doesn't actually copy the strings, it just callbacks 
with pointers to each field within the input buffer. (although my ffi 
code then does the copy and CONS' it into a list).

Cheers,

Phil






More information about the Gambit-list mailing list