Bradley Lucier wrote:
On Feb 9, 2007, at 5:08 PM, Phil Dawes wrote:
That brings the Phil Bewig parser down to ~950 ms
Ah, not much.
What happens when you configure gambit with --enable-char-size=1 (the default is 4). (I presume the C you're using has one-byte chars.)
I haven't tried this, but have done a similar thing for text reading before without gaining much speed improvement (because you still incur byte->char translation). I suspect a faster approach is to do the reading in binary mode (i.e. u8vectors)?
Or you could give a URL for your data for others to play with.
Good idea - here's the data: http://www.phildawes.net/2007/gambit-csv/data.csv.gz (I can't remember where I first got it from, but it would have been from the web somewhere so hopefully I'm not breaking any copyright).
(Next I would suggest doing line reads and stepping through the characters locally without going through the trampoline required for each call to read-char. It seems that doing your own line buffering may be natural for this problem.)
I've done something similar in python before, using a fast regex library to first mask out escaped chars, then quoted strings. Then you can look at the last char in a line to see if it should be joined to the next line (i.e. the newline is within a quote). Note that libcsv doesn't actually copy the strings, it just callbacks with pointers to each field within the input buffer. (although my ffi code then does the copy and CONS' it into a list).
Cheers,
Phil