Yes, this is right.

Strange enough ssax gives me a warning on my valid (?!) xml-file. No other parsers complains, neither do xmllint.

Petter

Den 16. aug. 2008 kl. 14.33 skrev Mikael More:

That is both correct and not correct. It uses input-port-byte- position when generating an error response, not otherwise. It typically passes you errors in case you fed the deserializer with invalid XML. Here's an utility function to deserialize an XML string into SXML.

(define readxml (lambda (#!optional (namespace-prefix-assig "")) (lambda (port) (ssax:xml->sxml port namespace-prefix-assig))))

(define (xml-string->sxml s #!optional (namespace-prefix-assig "")) (call-with-input-string s (readxml namespace-prefix-assig)))

Though, I do admit that it would be nice if someone fixed SSAX-SXML in this respect.

M

2008/8/16 Hans Petter Egesund petter.egesund@gmail.com Hi, more on slow xml...

I can't get ssax to parse from a string, it seems the library is using input-port-byte-position which needs a device input port. So the attempt so slurp the file into memory, and then do the parsing ends here?!

I will tell if I come up with something interesting.

Petter

---

Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list

---

Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list

Strange enough the parser always gives me a warning on char 1024, regardless which file I am parsing.

I did a small hack in ssax and recompiled, to run my test. (I will install the Christans' fix later, it is probably a more long-run solution).

I am now able to read from a string-port, and parsing time is now down to 50 secs, which is better, at least.

Petter

Den 16. aug. 2008 kl. 15.22 skrev Mikael More:

Can you paste the part it warns about here?

If you want it to pass you the line no, read the XML from a file input.

M

2008/8/16 Hans Petter Egesund petter.egesund@gmail.com Yes, this is right.

Strange enough ssax gives me a warning on my valid (?!) xml-file. No other parsers complains, neither do xmllint.

Petter

Den 16. aug. 2008 kl. 14.33 skrev Mikael More:

...

That is both correct and not correct. It uses input-port-byte- position when generating an error response, not otherwise. It typically passes you errors in case you fed the deserializer with invalid XML. Here's an utility function to deserialize an XML string into SXML.

(define readxml (lambda (#!optional (namespace-prefix-assig "")) (lambda (port) (ssax:xml->sxml port namespace-prefix-assig))))

(define (xml-string->sxml s #!optional (namespace-prefix-assig "")) (call-with-input-string s (readxml namespace-prefix-assig)))

Though, I do admit that it would be nice if someone fixed SSAX- SXML in this respect.

M

2008/8/16 Hans Petter Egesund petter.egesund@gmail.com Hi, more on slow xml...

I can't get ssax to parse from a string, it seems the library is using input-port-byte-position which needs a device input port. So the attempt so slurp the file into memory, and then do the parsing ends here?!

I will tell if I come up with something interesting.

Petter

---

Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list

---

Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list

---

Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list

---

Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list

Thanks. I will put it somewhere in the ssax-module and do a new recompile.

Petter

Den 16. aug. 2008 kl. 16.09 skrev Marc Feeley:

If you are really into optimizing the I/O, I suggest you implement your own special purpose string-port I/O layer. Basically you redefine read-char to read from a string. The code below shows that this can be over 20 times faster than Gambit's built-in string ports. The main reasons for the difference are function inlining and locking (to support multithreading).

(define (convoluted-string-length1 str) ;; uses standard string ports (declare (standard-bindings) (fixnum) (not safe)) (let ((port (open-input-string str))) (let loop ((i 0)) (let ((c (read-char port))) (if (char? c) (loop (+ i 1)) i)))))

(define-macro (macro-open-input-string str) `(let ((str ,str)) (declare (standard-bindings) (fixnum) (not safe)) (cons str 0)))

(define-macro (macro-read-char my-port) `(let ((my-port ,my-port)) (declare (standard-bindings) (fixnum) (not safe)) (let ((str (car my-port)) (pos (cdr my-port))) (if (< pos (string-length str)) (let ((c (string-ref str pos))) (set-cdr! my-port (+ 1 pos)) c) #!eof))))

(define-macro (open-input-string str) `(macro-open-input-string ,str)) (define-macro (read-char port) `(macro-read-char ,port))

;; insert your code after this point, for example:

(define (convoluted-string-length2 str) ;; uses "fast" string ports (declare (standard-bindings) (fixnum) (not safe)) (let ((port (open-input-string str))) (let loop ((i 0)) (let ((c (read-char port))) (if (char? c) (loop (+ i 1)) i)))))

(define (test) (let ((s (make-string 4000000 #!))) (pretty-print (time (convoluted-string-length1 s))) (pretty-print (time (convoluted-string-length2 s)))))

(test)

;; (time (convoluted-string-length1 s)) ;; 704 ms real time ;; 703 ms cpu time (661 user, 42 system) ;; 1 collection accounting for 50 ms real time (32 user, 19 system) ;; 36023288 bytes allocated ;; no minor faults ;; no major faults ;; 4000000 ;; (time (convoluted-string-length2 s)) ;; 26 ms real time ;; 25 ms cpu time (25 user, 0 system) ;; no collections ;; 24 bytes allocated ;; no minor faults ;; no major faults ;; 4000000

Mikael More wrote:

Marc,

just to ensure I got your point. You say that one could create a string-port more than 20 times faster than Gambit's built-in string ports. I'm interested in the reason - is this because:

a) If one writes one in scheme code, Gambit may perform function inlining? b) You presume that the string-port implementation one writes does not need to be thread-safe, within the context of Gambit's threading system, as is Gambit's built in string port?

It's both because of read-char requiring a real cross-module Scheme call (it is not being inlined) but also because it is implemented thread safe, i.e. using a (Gambit) mutex internally.

See my demo at http://scheme.mine.nu/gambit/experimental/read-char/ (which just a rehash of what Brad Lucier showed on the mailing list in february 2007 in the "fast csv parser" thread) where I showed that to someone on IRC.

You really want to clone http://scheme.mine.nu/gambit/experimental/read-char/.git and look at the history since there you find the benchmark results.

Any more reasons?

Also, what further reasons do you see as to why a SSAX-SXML XML to SXML deserialization that takes 3 seconds in Java takes 50 seconds in Gambit?

You could take a look at the Java code. Probably it isn't reading from the input in single chars with mutex locking for each of them.

Christian.

Mikael More wrote:

I see your point - if one writes a string-port, with completely custom open, read, write, close operations, one can achieve a lot higher speeds than if using Gambit's built-in I/O.

Though, are the higher speeds Marc addressed reachable while still using Gambit built-in I/O operations, such as read, write, read-char, display, etc.?

In order to make for instance SSAX-SXML use a custom string port implementation without

• modifying its code (i.e. changing all I/O operations there are in it to

custom ones)

That's easy, just alias the new read-char in place of the built-in one -- you've already using a separate ssax-sxml# namespace after all.

• dismaking its compatibility with Gambit's internal I/O functionality

(i.e. file ports, TCP ports, etc.)

Did you look at my example? You could just dump it in place of the built-in read-char *iff* you are sure you don't need thread-safe access to those ports.

one needs to run this port implementation atop/behind/under/using Gambit's built-in I/O system. So, is it possible to increase the speed of the string port implementation

My example was even using file input ports, not string ports.

a lot, while still running on Gambit's built-in I/O?

(My read-char example *is* using all of Gambit's built-in I/O except that it does not do the mutex locking.)

The Gambit mutex implementation could maybe be sped up, or complemented by some faster variant, in the current threading model (i.e. running in one system thread only), I did play with a Scheme-level spinlock implementation some time ago (which did atomic increments/decrements of a boxed integer through disabling interrupts / by using the C ffi, and loop running thread-yield! until the mutex is granted), but it only worked if all threads were running under the same priority since thread-yield! wouldn't yield to a thread with a lower priority; anyway should the thread system incorporate multiple system threads another mutex implementation (iff mutexes can be passed between system threads) will have to be made again so... (I could imagine to (help) work on this but don't currently see when this would be).

If not, perhaps one would benefit from writing a custom buffered IO layer atop Gambit-s built-in IO, and then patch SSAX-SXML to use it. That ought to give blazingly high speeds, correct?

If you want to make it guaranteed correct, you should find out (by reading the SSAX-SXML sources, or at least studying the api precisely and hoping you're taking correct conclusions from the study) to find out whether they don't somehow directly or indirectly allow different threads to read from the same port, or/and introduce the port mutex locking at a coarser granularity than character based.

I remember the discussion of the SSAX-SXML author boosting with his library being as fast as expat, but it was pointed out that this was only true if expat was run in character-reading mode too (smile), at which point he said that it's enjoyable to read in characters since this allows to precisely finish reading from a stream when an xml document has finished. So if you want to keep this philosophy, block-wise reading of input data is out. I guess you could still achieve that goal if you take the mutex when starting to read from a stream, then release it when the last piece of the document has been read. Where exactly you would introduce those lock/unlock calls into the work flow, I don't know as I haven't really worked with SSAX-SXML yet.

Christian.

Hi, it might have been a to optimistic approach, but I did just copy the new macros into ssax (on the top of the file SSAX-code.scm which seems to do the job).

It still compiles, but now I get a segmentation fault, when running,

As far as I can see this is not due to a thread-problem - ssax does not use many thread, nor do I.

It probably needs more investigation. Any opinions about where to start looking?

Petter

Den 16. aug. 2008 kl. 16.09 skrev Marc Feeley:

If you are really into optimizing the I/O, I suggest you implement your own special purpose string-port I/O layer. Basically you redefine read-char to read from a string. The code below shows that this can be over 20 times faster than Gambit's built-in string ports. The main reasons for the difference are function inlining and locking (to support multithreading).

(define (convoluted-string-length1 str) ;; uses standard string ports (declare (standard-bindings) (fixnum) (not safe)) (let ((port (open-input-string str))) (let loop ((i 0)) (let ((c (read-char port))) (if (char? c) (loop (+ i 1)) i)))))

(define-macro (macro-open-input-string str) `(let ((str ,str)) (declare (standard-bindings) (fixnum) (not safe)) (cons str 0)))

(define-macro (macro-read-char my-port) `(let ((my-port ,my-port)) (declare (standard-bindings) (fixnum) (not safe)) (let ((str (car my-port)) (pos (cdr my-port))) (if (< pos (string-length str)) (let ((c (string-ref str pos))) (set-cdr! my-port (+ 1 pos)) c) #!eof))))

(define-macro (open-input-string str) `(macro-open-input-string ,str)) (define-macro (read-char port) `(macro-read-char ,port))

;; insert your code after this point, for example:

(define (convoluted-string-length2 str) ;; uses "fast" string ports (declare (standard-bindings) (fixnum) (not safe)) (let ((port (open-input-string str))) (let loop ((i 0)) (let ((c (read-char port))) (if (char? c) (loop (+ i 1)) i)))))

(define (test) (let ((s (make-string 4000000 #!))) (pretty-print (time (convoluted-string-length1 s))) (pretty-print (time (convoluted-string-length2 s)))))

(test)

;; (time (convoluted-string-length1 s)) ;; 704 ms real time ;; 703 ms cpu time (661 user, 42 system) ;; 1 collection accounting for 50 ms real time (32 user, 19 system) ;; 36023288 bytes allocated ;; no minor faults ;; no major faults ;; 4000000 ;; (time (convoluted-string-length2 s)) ;; 26 ms real time ;; 25 ms cpu time (25 user, 0 system) ;; no collections ;; 24 bytes allocated ;; no minor faults ;; no major faults ;; 4000000

OK, I got sucked into this ... If you have 8 gigs of ram for the compile on x86-64 you can try this version of SSAX. Since I don't have any test data and I haven't ever tried expat I haven't done much more than compile this.

I started with SSAX-SXML by Kirill Lisovsky (updated by Dominique Boucher) from the dumping grounds at

http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/Dumping_Grounds

I uncommented this line

./stx/libmisc.scm:(define (self x) x)

because it was needed later. I also commented the calls to "gambitize" things because I don't think they help so much. (They don't eliminate any inter-module procedure calls.)

./libs/input-parse.scm:;;(gambitize (read-char port)) ./libs/input-parse.scm:;; (gambitize (peek-char port)) ./libs/input-parse.scm:;; (gambitize (eof-object? port))

The Makefile has some mistakes (which can't be fixed in a machine/OS-independent way, that's why Gambit has the configure command), but it has a useful list of files needed to build ssax-sxml.o1.

I include a file ssax-sxml.scm that I used to build SSAX on Ubuntu 8.0.4 on x86-64; if somebody's mail client munges this message, at least you'll have the included file. (The c->.o1 compile took about 6.3GB of RAM.) Here is an explanation of what it does.

First, I load the Gambit internal headers, which are always available at ~~/lib/, together with the names of all the standard procedures in r5rs:

(##include "~~/lib/gambit#.scm") (##include "~~/lib/_gambit#.scm") (##include "~~/lib/r5rs#.scm")

Then I set some declarations, mainly to reduce the size of the .c file so recent versions of gcc can compile it. (Supposedly, the introduction of "factored definition-use chains" will reduce the memory requirement for compiling Gambit-generated C code in gcc 4.4.*, but that change hasn't been committed to SVN gcc yet.):

(declare (standard-bindings)(extended-bindings)(block)(fixnum)) (declare (not safe)) (declare (inlining-limit 134))

Then I want to redefine read-char and peek-char to local versions that just contain the fast path from _io.scm. I also want to redefine eof-object? because that isn't expanded by gsc. (Marc: Can you fix this last thing without me filing a bug report?) So I remove read-char, peek-char, and eof-object? from the list of standard procedures the compiler knows about, and I tell gsc to not do runtime expansion of eof-object?

(declare (not standard-bindings read-char peek-char eof-object?)) (declare (not run-time-bindings eof-object?))

Then I redefine peek-char, read-char, and eof-object? as I decided to:

(define (ssax#peek-char port)

(##declare (not interrupts-enabled))

(macro-port-mutex-lock! port) ;; get exclusive access to port

(let ((char-rlo (macro-character-port-rlo port)) (char-rhi (macro-character-port-rhi port))) (if (##fixnum.< char-rlo char-rhi)

;; the next character is in the character read buffer

(let ((c (##string-ref (macro-character-port-rbuf port) char-rlo))) (macro-port-mutex-unlock! port) c)

(if (macro-character-port-peek-eof? port)

(begin (macro-port-mutex-unlock! port) #!eof) (begin (macro-port-mutex-unlock! port) (##peek-char port))))))

(define (peek-char #!optional (port (macro-absent-obj))) (macro-force-vars (port) (let ((p (if (##eq? port (macro-absent-obj)) (macro-current-input-port) port))) (macro-check-character-input-port p 1 (peek-char p) (ssax#peek-char p)))))

(define (ssax#read-char port)

(##declare (not interrupts-enabled))

(macro-port-mutex-lock! port) ;; get exclusive access to port

(let loop ()

(let ((char-rlo (macro-character-port-rlo port)) (char-rhi (macro-character-port-rhi port))) (if (##fixnum.< char-rlo char-rhi)

;; the next character is in the character read buffer

(let ((c (##string-ref (macro-character-port-rbuf port) char-rlo))) (if (##not (##char=? c #\newline))

;; frequent simple case, just advance rlo

(begin (macro-character-port-rlo-set! port (##fixnum.+ char-rlo 1)) (macro-port-mutex-unlock! port) c)

(begin (macro-port-mutex-unlock! port) (##read-char port))))))))

(define (read-char #!optional (port (macro-absent-obj))) (macro-force-vars (port) (let ((p (if (##eq? port (macro-absent-obj)) (macro-current-input-port) port))) (macro-check-character-input-port p 1 (read-char p) (ssax#read-char p)))))

(define (ssax#eof-object? x) (##eq? x #!eof))

(define (eof-object? x) (macro-force-vars (x) (ssax#eof-object? x)))

Then I just include the list of .scm files included in ssax-sxml.o1:

(include "libs/gambit/common.scm") (include "libs/gambit/myenv.scm") (include "libs/srfi-13-local.scm") (include "libs/util.scm") (include "libs/gambit/parse-error.scm") (include "libs/input-parse.scm") (include "libs/look-for-str.scm") (include "ssax/char-encoding.scm") (include "ssax/SSAX-code.scm") (include "ssax/SXML-tree-trans.scm") (include "sxml-tools/sxpathlib.scm") (include "multi-parser/id/srfi-12.scm") (include "multi-parser/id/mime.scm") (include "multi-parser/id/http.scm") (include "multi-parser/id/access-remote.scm") (include "multi-parser/id/id.scm") (include "sxml-tools/xlink-parser.scm") (include "multi-parser/ssax-prim.scm") (include "multi-parser/multi-parser.scm") (include "html-prag/htmlprag.scm") (include "sxml-tools/sxml-tools.scm") (include "sxml-tools/sxpath-ext.scm") (include "sxml-tools/xpath-parser.scm") (include "sxml-tools/txpath.scm") (include "sxml-tools/sxpath.scm") (include "sxml-tools/xpath-ast.scm") (include "sxml-tools/xpath-context.scm") (include "sxml-tools/xlink.scm") (include "sxml-tools/ddo-axes.scm") (include "sxml-tools/ddo-txpath.scm") (include "sxml-tools/lazy-xpath.scm") (include "ssax/lazy-ssax.scm") (include "sxml-tools/modif.scm") (include "sxml-tools/serializer.scm") (include "sxml-tools/guides.scm") (include "stx/libmisc.scm") (include "stx/stx-engine.scm")

Then we get

frying-pan:~/programs/gambc-v4_2_3/ssax-sxml> time gsc -keep-c -expansion ssax-sxml > ! expansion.scm 150.589u 4.680s 2:36.47 99.2% 0+0k 16+46104io 0pf+0w frying-pan:~/programs/gambc-v4_2_3/ssax-sxml> ll total 13320 drwxr-xr-x 2 lucier lucier 4096 2008-08-16 16:54 apidoc/ -rw------- 1 lucier lucier 7623 2006-05-29 09:27 doc.txt -rw------- 1 lucier lucier 1092 2006-05-29 09:27 example.sch -rw------- 1 lucier lucier 5447 2006-05-29 09:27 example.scm -rw-r--r-- 1 lucier lucier 3065182 2008-08-16 17:22 expansion.scm drwxr-xr-x 2 lucier lucier 4096 2008-08-16 16:55 html-prag/ drwxr-xr-x 3 lucier lucier 4096 2008-08-16 16:55 libs/ -rw------- 1 lucier lucier 8361 2008-05-02 21:12 Makefile drwxr-xr-x 3 lucier lucier 4096 2008-08-16 16:55 multi-parser/ -rw------- 1 lucier lucier 10081 2006-05-29 09:27 prj-ssax-sxml.s drwxr-xr-x 2 lucier lucier 4096 2008-08-16 16:55 ssax/ -rw-r--r-- 1 lucier lucier 7505995 2008-08-16 17:20 ssax-sxml.c -rwxr-xr-x 1 lucier lucier 2956306 2008-08-16 17:22 ssax-sxml.o1* -rw-r--r-- 1 lucier lucier 3887 2008-08-16 16:56 ssax-sxml.scm drwxr-xr-x 2 lucier lucier 4096 2008-08-16 16:55 stx/ drwxr-xr-x 3 lucier lucier 4096 2008-08-16 16:55 sxml-tools/ -rw------- 1 lucier lucier 234 2006-05-29 09:27 test-sxml.scm drwxr-xr-x 2 lucier lucier 4096 2006-05-29 09:27 XML/

If there is some data around that I can benchmark this with, I'll try it. It seems to pass the tests in test-sxml.scm.

Brad

Hi,

my computer (64-bit ADM, 8GB ram, Ubuntu) is still trying to compile the file. It has been running for 6 hours or so, and is swapping a lot.

I did a: time gsc -keep-c -expansion ssax-sxml > ! expansion.scm

Very much like you?

Petter

Den 17. aug. 2008 kl. 04.53 skrev Bradley Lucier:

Sorry, there was a bug in ssax#read-char, a corrected version is included here. There doesn't seem to be a short and easily identifiable "fast-path" in ##write-char, so I just defined ssax#write-char, which is a copy of ##write-char, and defined write-char in terms of it in ssax-sxml.scm.

Pretty strange that the tests in test-sxml.scm passed ...

I ran Marc's example of "build-your-own" string ports on my machine and got:

frying-pan:~/programs/gambc-v4_2_3/ssax-sxml> gsi test-io-3 (time (convoluted-string-length1 s)) 420 ms real time 420 ms cpu time (400 user, 20 system) 1 collection accounting for 17 ms real time (12 user, 4 system) 40016080 bytes allocated 8913 minor faults no major faults 4000000 (time (convoluted-string-length2 s)) 13 ms real time 12 ms cpu time (12 user, 0 system) no collections 32 bytes allocated no minor faults no major faults 4000000

Pretty convincing. I then ran

(define (convoluted-string-length2 str) (let ((port (open-input-string str))) (let loop ((i 0)) (let ((c (read-char port))) (if (char? c) (loop (+ i 1)) i)))))

(define (convoluted-string-length3 port) (let loop ((i 0)) (let ((c (read-char port))) (if (char? c) (loop (+ i 1)) i))))

(define (test) (let ((s (make-string 4000000 #!))) (pretty-print (time (convoluted-string-length2 s))) (let ((port (open-input-string s))) (pretty-print (time (convoluted-string-length3 port))) )))

(test)

with the prefix I made for ssax-sxml.scm and got

frying-pan:~/programs/gambc-v4_2_3/ssax-sxml> gsi test-io (time (convoluted-string-length2 s)) 121 ms real time 124 ms cpu time (112 user, 12 system) 1 collection accounting for 17 ms real time (12 user, 4 system) 40016080 bytes allocated 8912 minor faults no major faults 4000000 (time (convoluted-string-length3 port)) 74 ms real time 76 ms cpu time (76 user, 0 system) no collections -16 bytes allocated no minor faults no major faults 4000000

So open-input-string on a length 4000000 string takes 50ms and allocates 40016080 bytes.

I wanted to test the effects of these changes on the "Gambit Benchmarks", so I added the ssax-sxml.scm prefix to prefix-gambit.scm and ran

./bench -s r6rs-fixflo-unsafe gambit 'cat compiler slatex tail wc dynamic sum1'

all the benchmarks that contain read-char, peek-char, write-char, or eof-object?. Here are the results, first the cputime without the local i/o procedures, then with:

cat: 880 ms 160 ms compiler: 824 ms 820 ms slatex: 356 ms 232 ms tail: 692 ms 380 ms wc: 400 ms 100 ms dynamic: 732 ms 696 ms sum1: 880 ms 896 ms

Judging from

http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/ Gambit_benchmarks

and

http://www.ccs.neu.edu/home/will/Twobit/benchmarksFakeR6Linux.html

this compilation strategy of adding local versions of read-char, write-char, and peek-char, which I outline back in February 2007,

https://webmail.iro.umontreal.ca/pipermail/gambit-list/2007- February/001140.html

would allow gambit to improve in the heavy-duty character-oriented benchmarks cat, wc, tail, and slatex.

Brad<ssax-sxml.scm>_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list

--- This is a quick fix I did when I encountered this problem.

Note: you can get this patch (and some maybe other interesting small changes I did) out of the Git repository at http://scheme.mine.nu/gambit/ssax-sxml/.git

libs/gambit/parse-error.scm | 30 ++++++++++++++++++++++++++---- 1 files changed, 26 insertions(+), 4 deletions(-)