[gambit-list] fast csv parser (libcsv wrapped for gambit)
pdawes at users.sf.net
Sat Feb 10 03:44:19 EST 2007
Bradley Lucier wrote:
> On Feb 9, 2007, at 5:08 PM, Phil Dawes wrote:
>> That brings the Phil Bewig parser down to ~950 ms
> Ah, not much.
> What happens when you configure gambit with --enable-char-size=1 (the
> default is 4). (I presume the C you're using has one-byte chars.)
I haven't tried this, but have done a similar thing for text reading
before without gaining much speed improvement (because you still incur
byte->char translation). I suspect a faster approach is to do the
reading in binary mode (i.e. u8vectors)?
> Or you could give a URL for your data for others to play with.
Good idea - here's the data:
(I can't remember where I first got it from, but it would have been from
the web somewhere so hopefully I'm not breaking any copyright).
> (Next I would suggest doing line reads and stepping through the
> characters locally without going through the trampoline required for
> each call to read-char. It seems that doing your own line buffering
> may be natural for this problem.)
I've done something similar in python before, using a fast regex library
to first mask out escaped chars, then quoted strings. Then you can look
at the last char in a line to see if it should be joined to the next
line (i.e. the newline is within a quote).
Note that libcsv doesn't actually copy the strings, it just callbacks
with pointers to each field within the input buffer. (although my ffi
code then does the copy and CONS' it into a list).
More information about the Gambit-list