Speed on XML-parsing
Hi, I have some large xml-files which I have to parse, and I have used latest Gambit + ssax on my 64-bit (now :-) computer, Linux. My program parses my file in 150 seconds. After adding (declare (not safe... ) and so on, and compiling with -02 I get it down to 140 seconds, tweaking ssax itself the same way gives me 115 seconds. The parser is so far doing nothing - it just runs through the data. Doing the same thing in Java takes about 3 secs, using expat from C makes it 2 secs. But my math works blazingly fast in Gambit. Is there any thing I can do to make the xml-parsing faster? I have found some papers that claim ssax to be as quick as expat (http://okmij.org/ftp/Scheme/SSAX-benchmark-1.html) - as I have some thousand of these files to parse, a speedup would really make a difference. Clues on this one? Cheers, Petter
On Aug 15, 2008, at 7:08 PM, Hans Petter Egesund wrote:
Clues on this one?
SSAX isn't easy to build (my own private definition, for anything I can't figure out how to build in 30 minutes or less--ah, it seems I've proved incompetent at this before: https://webmail.iro.umontreal.ca/pipermail/gambit-list/2006-October/ 000882.html ), so it's going to be hard to offer suggestions. If you have heavy-duty string processing you might enable Gambit with --enable-char-size=1; the default is 4. If you follow the thread spread between subthreads starting https://webmail.iro.umontreal.ca/pipermail/gambit-list/2007-February/ 001112.html and https://webmail.iro.umontreal.ca/pipermail/gambit-list/2007-February/ 001125.html you'll see that IO using read-char and write-char is slow; that's hard to fix. You can increase the minimum heap size with -:m100000 (to give it 100MB). Good luck. Brad
Adjusting gambit to accept .c files not in the current directory will allow SSAX to build. There's a patch for that in bug #41. On Fri, Aug 15, 2008 at 5:45 PM, Bradley Lucier <lucier@math.purdue.edu> wrote:
On Aug 15, 2008, at 7:08 PM, Hans Petter Egesund wrote:
Clues on this one?
SSAX isn't easy to build (my own private definition, for anything I can't figure out how to build in 30 minutes or less--ah, it seems I've proved incompetent at this before:
https://webmail.iro.umontreal.ca/pipermail/gambit-list/2006-October/ 000882.html
), so it's going to be hard to offer suggestions.
If you have heavy-duty string processing you might enable Gambit with --enable-char-size=1; the default is 4.
If you follow the thread spread between subthreads starting
https://webmail.iro.umontreal.ca/pipermail/gambit-list/2007-February/ 001112.html
and
https://webmail.iro.umontreal.ca/pipermail/gambit-list/2007-February/ 001125.html
you'll see that IO using read-char and write-char is slow; that's hard to fix.
You can increase the minimum heap size with -:m100000 (to give it 100MB).
Good luck.
Brad _______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
On Aug 15, 2008, at 10:00 PM, Frederick LeMaster wrote:
Adjusting gambit to accept .c files not in the current directory will allow SSAX to build. There's a patch for that in bug #41.
That's not enough for any of the SSAX sources I found (CVS, the gambit one from 2006) to build using the Makefiles. Brad
On Sat, Aug 16, 2008 at 1:45 AM, Bradley Lucier <lucier@math.purdue.edu> wrote:
On Aug 15, 2008, at 7:08 PM, Hans Petter Egesund wrote:
Clues on this one?
If you follow the thread spread between subthreads starting... you'll see that IO using read-char and write-char is slow; that's hard to fix.
Yes. I've had this problem with a number of schemes. Depending on the size of your file, reading the whole thing as a string and *then* running the rest of your program on a string-port (instead of a plain input-port) can give you a fair bit of speed increase. There are deeper hacks you can use to avoid allocator churn while reading the file, but these can be very dependent on the specfic implementation. Another thing I did on one project was develop a preprocessor which converted my XML files to SXML format. Since READ is highly optimized in most Scheme readers it made the actual data processing go very fast. This had the added advantage that I was able to break my dependency on SSAX. Just some thoughts. Remember, memory is cheap these days. david -- GPG Public key at http://cyber-rush.org/drr/gpg-public-key.txt
Guess I was lucky when I installed ssax, I just had to add a couple of parameters to the makefile. I will try to slurp the file, and then read from a string-port, as a first try. Thanks. Petter Den 16. aug. 2008 kl. 08.32 skrev David Rush:
On Sat, Aug 16, 2008 at 1:45 AM, Bradley Lucier <lucier@math.purdue.edu> wrote:
On Aug 15, 2008, at 7:08 PM, Hans Petter Egesund wrote:
Clues on this one?
If you follow the thread spread between subthreads starting... you'll see that IO using read-char and write-char is slow; that's hard to fix.
Yes. I've had this problem with a number of schemes. Depending on the size of your file, reading the whole thing as a string and *then* running the rest of your program on a string-port (instead of a plain input-port) can give you a fair bit of speed increase. There are deeper hacks you can use to avoid allocator churn while reading the file, but these can be very dependent on the specfic implementation.
Another thing I did on one project was develop a preprocessor which converted my XML files to SXML format. Since READ is highly optimized in most Scheme readers it made the actual data processing go very fast. This had the added advantage that I was able to break my dependency on SSAX.
Just some thoughts. Remember, memory is cheap these days.
david -- GPG Public key at http://cyber-rush.org/drr/gpg-public-key.txt
participants (4)
-
Bradley Lucier -
David Rush -
Frederick LeMaster -
Hans Petter Egesund