- Gambit-list - mailman.iro.umontreal.ca

gambit-c pcre loadable library crashes at ___make_vector
by Ben Weaver 22 Oct '06

22 Oct '06

Hello All, I'm trying to write a PCRE FFI for Gambit Version 4.0 beta 17. One of my C functions, c_wrap_pcre_match_ovector, is meant to convert the offset vector from a PCRE match (simply an array of integers) into a scheme vector. This function is attached at the end of this email as well as a crash report. The code works fine if everything is compiled and run like this: gsc pcre.scm gcc pcre*.c -lpcre -lgambc ./a.out But when I try to compile it as a loadable library, my programs crash as soon as ___make_vector is called from c_wrap_pcre_match_ovector. This is how I'm trying to compile and use the library: gsc -o pcre.o1.c -flat pcre > /dev/null gcc -o pcre.o1 -bundle -D___DYNAMIC -lpcre pcre-wrapper.c pcre.c pcre.o1.c gsi pcre.o1 pcre-test.scm My source files are: pcre.scm The FFI wrappers pcre-wrapper.c Some wrapper functions, including c_wrap_pcre_match_ovector pcre-wrapper.h Declarations for pcre-wrapper.c pcre-test.scm Some test calls to pcre-match, etc. I'm using gcc 4.0.1 on Mac OS X 10.4.8 (PowerPC). Any help is appreciated. On a related note, is there a benefit to constructing a vector like this in C-space versus making a c-lambda to grab an entry from the int[] and constructing the vector in scheme-space instead? I considered both options, but my rational for constructing the vector in C-space is to minimize the number of FFI calls needed to process a match. Thanks, -Ben ================================================================= ___SCMOBJ c_wrap_pcre_match_ovector(c_wrap_pcre_match *match) { ___SCMOBJ ___err = ___FIX(___NO_ERR); ___SCMOBJ result; ___SCMOBJ offset; ___SCMOBJ length; int i, limit; if ((match == NULL) || ((limit = match->rc) < 0)) { return ___FAL; } if (limit == 0) { /* * ovector wasn't big enough, the first 2/3 are used * and the rest of the matches were discarded */ limit = match->ovecsize / 3; } /* convert the ovector to a scheme vector */ limit = 2 * limit; /* CRASH happens at the line below */ result = ___EXT(___make_vector)(limit, ___FAL, ___STILL); if (___FIXNUMP(result)) { return result; } for (i = 0; (i < limit) && (___err == ___FIX(___NO_ERR)); i++) { ___BEGIN_SFUN_INT_TO_SCMOBJ(match->ovector[i], offset, 1) ___FIELD(result, i) = offset; ___END_SFUN_INT_TO_SCMOBJ(match->ovector[i], offset, 1) } ___EXT(___release_scmobj)(result); if (___err != ___FIX(___NO_ERR)) { return ___err; } return result; } ================================================================= Date/Time: 2006-10-19 22:06:37.043 -0400 OS Version: 10.4.8 (Build 8L127) Report Version: 4 Command: gsi Path: /usr/local/bin/gsi Parent: zsh [25726] Version: ??? (???) PID: 27331 Thread: 0 Exception: EXC_BAD_ACCESS (0x0001) Codes: KERN_PROTECTION_FAILURE (0x0002) at 0x0000040c Thread 0 Crashed: 0 pcre.o1 0x003e34e8 c_wrap_pcre_match_ovector + 156 (bundle1.s:283) 1 pcre.o1 0x003e5244 ___H__20_pcre_23_5 + 412 (bundle1.s:283) 2 gsi 0x001bb338 ___call + 376 (crt.c:355) 3 gsi 0x001bc2bc ___setup + 3204 (crt.c:355) 4 gsi 0x001b6390 ___main + 2464 (crt.c:355) 5 gsi 0x000048e8 ___main_char + 132 (crt.c:355) 6 gsi 0x000047f4 main + 40 (crt.c:355) 7 gsi 0x00001e30 _start + 344 (crt.c:272) 8 gsi 0x00001cd4 start + 60 Binary Images Description: 0x1000 - 0x1d7fff gsi /usr/local/bin/gsi 0x3e2000 - 0x3f6fff pcre.o1 /Users/bweaver/work/weaver(a)coptix.com/pcre/pcre.o1 0x505000 - 0x514fff libpcre.0.dylib /usr/local/lib/libpcre.0.dylib 0x8fe00000 - 0x8fe51fff dyld 45.3 /usr/lib/dyld 0x90000000 - 0x901bcfff libSystem.B.dylib /usr/lib/libSystem.B.dylib 0x90214000 - 0x90219fff libmathCommon.A.dylib /usr/lib/system/libmathCommon.A.dylib 0x94be3000 - 0x94c03fff libmx.A.dylib /usr/lib/libmx.A.dylib Model: PowerBook6,2, BootROM 4.7.5f1, 1 processors, PowerPC G4 (1.1), 1 GHz, 768 MB Graphics: GeForce FX Go5200 32MB, GeForce FX Go5200 32MB, AGP, 32 MB Memory Module: DIMM0/BUILT-IN, 256 MB, built-in, built-in Memory Module: DIMM1/J31, 512 MB, DDR SDRAM, PC2700U-25330 AirPort: AirPort Extreme, 405.1 (3.90.34.0.p18) Modem: MicroDash, UCJ, V.92, 1.0F, APPLE VERSION 2.6.6 Bluetooth: Version 1.7.8f2, 2 service, 0 devices, 1 incoming serial ports Network Service: Built-in Ethernet, Ethernet, en0 Parallel ATA Device: MATSHITACD-RW CW-8123, Parallel ATA Device: TOSHIBA MK6025GAS, 55.89 GB USB Device: Hub in Apple Pro Keyboard, Mitsumi Electric, Up to 12 Mb/sec, 500 mA USB Device: Microsoft IntelliMouse. Explorer, Microsoft, Up to 1.5 Mb/sec, 100 mA USB Device: Apple Pro Keyboard, Mitsumi Electric, Up to 12 Mb/sec, 250 mA USB Device: Bluetooth HCI, , Up to 12 Mb/sec, 500 mA

2 2

Re: [gambit-list] More benchmarks
by MichaelL＠frogware.com 20 Oct '06

20 Oct '06

Hello, Marc & Will. For what it's worth, when I last looked at the way Chez & Petite were run I noticed that it wasn't quite correct. The script I saw does this: chez_exec () { echo "(load \"$1.scm\")" | /usr/bin/time scheme } petite_chez_exec () { echo "(load \"$1.scm\")" | /usr/bin/time petite } More correct would be something like this: "(optimize-level 2) (module () (import-only scheme) (include \"$1.scm\"))" The "(optimize-level 2)" will enable immutable Scheme bindings (and primitive inlining) while still producing safe code. The top-level module (especially with the absence of any exports) will signal to Chez that it is free to inline. The "(import-only scheme)" declaration is virtually identical to the "(optimize-level 2)" declaration, at least in Chez 7, and is in fact preferred over setting the optimize level--but Will is using 6.1 so I'd recommend using both since it will do no harm and 6.1 might be different. (See http://www.scheme.com/csug7/use.html#./use:h3 for more information on all of this.) Btw, a quick look at the Scheme code shows this change will create at least one problem because expressions sometimes appear before definitions (eg, the "(setup)" in boyer.scm before the definition of "main."). Finally, make sure you aren't measuring compilation time in addition to running time. The above approach appears to be doing just that; for Chez at least compilation time can be significant relative to a small running time. Optimizations take time. :-) If you're interested in more Chez times we have Chez 7.1 for Windows, Linux, Solaris, and Mac. (Our Sun box is rather ancient though.) (Apologies if I misread something about the way the tests were set up.) > Hello. Will Clinger has done some "independent testing" of the > performance of various Scheme systems using the Gambit benchmark > suite in the "R6RS" mode (safe code, generic arithmetic, immutable > bindings). The results are published on the following page: > > http://www.ccs.neu.edu/home/will/Twobit/benchmarks2006.html > > What's particularly interesting is that he benchmarked with more > systems than I have access to (including Larceny, MIT-Scheme and Chez > Scheme) and on two platforms (Sun Sparc and Intel/Linux). > > The "geometric mean" results on the Sparc show that the code > generated by Gambit is among the fastest of all the Scheme systems > tested (it is only slightly slower than Chez Scheme and basically the > same speed as Larceny, which are both native code compilers). Gambit > is the fastest of the Scheme to C compilers (about 2 times faster > than Bigloo and 4 times faster than Chicken). MzScheme and Scheme48 > are about 10 times slower than Gambit. On Linux the situation is > similar, and Gambit clearly dominates the other systems, about 50% > faster than the next fastest, i.e. Larceny and Bigloo. Note that > Chez Scheme was not available on the Linux machine. > > So, both on Sparc and Intel, it appears that Gambit is the fastest > open-source Scheme system. > > Marc

3 3

More benchmarks
by Marc Feeley 19 Oct '06

19 Oct '06

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello. Will Clinger has done some "independent testing" of the performance of various Scheme systems using the Gambit benchmark suite in the "R6RS" mode (safe code, generic arithmetic, immutable bindings). The results are published on the following page: http://www.ccs.neu.edu/home/will/Twobit/benchmarks2006.html What's particularly interesting is that he benchmarked with more systems than I have access to (including Larceny, MIT-Scheme and Chez Scheme) and on two platforms (Sun Sparc and Intel/Linux). The "geometric mean" results on the Sparc show that the code generated by Gambit is among the fastest of all the Scheme systems tested (it is only slightly slower than Chez Scheme and basically the same speed as Larceny, which are both native code compilers). Gambit is the fastest of the Scheme to C compilers (about 2 times faster than Bigloo and 4 times faster than Chicken). MzScheme and Scheme48 are about 10 times slower than Gambit. On Linux the situation is similar, and Gambit clearly dominates the other systems, about 50% faster than the next fastest, i.e. Larceny and Bigloo. Note that Chez Scheme was not available on the Linux machine. So, both on Sparc and Intel, it appears that Gambit is the fastest open-source Scheme system. Marc -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iD8DBQFFN5F5//V9Zc2T/v4RAp1nAKCFMdMptGXMIphSrZNdGYW9xF11lgCgkliY pnXa35cP+s0UOVbrMEQbWEI= =72E+ -----END PGP SIGNATURE-----

1 0

Attach Emacs to a web-repl?
by Andrew Lentvorski 18 Oct '06

18 Oct '06

Is there any way to attach Emacs (or any other editor, for that matter), to a web-repl? Given that I can telnet directly to port 7000 and execute the repl, there ought to be some way. I took a quick look at cmuscheme.el, comint.el and ftelnet.el, but my Emacs-fu is clearly not good enough. I don't see anything obvious. Thanks, -a

4 5

input-port-byte-position
by John MacFarlane 18 Oct '06

18 Oct '06

I'm getting strange behavior with input-port-byte-position. When used to read the byte position of an input file, it always returns 0. On the other hand, when used to set the byte position, it works as expected. Here's an example: % gsi/gsi Gambit Version 4.0 beta 20 > (define f (open-input-file "README")) > (input-port-byte-position f) 0 > (read-char f) #\R > (read-char f) #\E > (input-port-byte-position f) 0 > (input-port-byte-position f 10) 10 > (input-port-byte-position f) 0 > (read-char f) #\e > (input-port-byte-position f) 0 > I've tried following the example in the manual to the letter, and it still always returns 0. I've tried this both with 4.0 beta 17 (installed from a deb) and with 4.0 beta 20 (compiled from source, passes all tests in "make check"). I get the same results on two different machines (both i686-gnu-linux). Does anyone have an explanation for this, or a fix? Aside from this problem, gambit-C is perfect for my project. I noticed that there is a message in the archives from September 2005 asking the same question, but no response was posted. Any help would be most appreciated. John

2 4

Re: [gambit-list] gcc stage timing
by Christian 18 Oct '06

18 Oct '06

At 13:13 Uhr -0400 17.10.2006, Lang Martin wrote: >That's just the one call to gensym, which vaguely seems like the most >expensive part of doing it this way You could move that outside the call path: (define make-lazy (let* ((nothing (gensym)) (exit (lambda () nothing))) (lambda (generator) (let lp () (delay (let ((value (generator exit))) (if (eq? value nothing) '() (cons value (lp)))))))) >, and should be as correct. Could >write a generator to break it, but I don't suppose it'd happen >accidently. (The generator has to call the exit function in tail position now. Maybe better just feed it the nothing value -- or move the nothing value to the toplevel and export it to users, so the generator can become a thunk.) Christian.

1 1

Re: [gambit-list] gcc stage timing
by Christian 18 Oct '06

18 Oct '06

At 16:21 Uhr -0400 16.10.2006, Lang Martin wrote: >quoth Christian: > >> (Note that the "limit" is not actually limiting any calculation >> (assuming that futures do terminate); it's just making other futures >> wait until earlier ones are done. You'll want this when starting >> external tools, otherwise you'll make your machine's responsitivity >> suffer.) > >yeah, I'm not actually calling external processes. However, my >instinct is to still have gambit catch the appropriate exceptions, and >go until it runs out of memory or filehandles or processes, or >whatever. Typically I set the limits from the outside with a ulimit >command before I start the process, giving me a unified interface, and >confidence that the limits will be respected. Also makes gambit >graceful if you need to scale it down, and run in an evironment with >few resources. > >From a unix perspective, I think external limits and graceful failure >are the right thing to do. Ok, but then you ignore portability. (I don't use Windows myself, but still.) In our case (parallel gcc), it's problematic since running right below the process limit will make gcc fail when it tries to run subprocesses itself. Will you be able to deduce gcc's reason of failure from it's exit status or will you have to parse it's output? Will you patch it to make it take an option telling it to retry forking his own children? Maybe we need a new GNU system to implement this idea right everywhere? The semaphore approach has at least the advantage of being efficient in the sense that new tasks are started as soon as older ones finish (no retry/polling needed, so no phases of inactivity). It may not be straight-forward to deal with OS limits. Will you make sure that the scheme thread leading to an out of memory error terminates and takes all it's data with it to make place for the others to finish? Or will you hope you can get through by just stopping the thread and waiting for the others to finish? Will you signal that no new calculations may be started? Will the emergency memory pool you have hopefully reserved suffice to survive the rest? Or will you flush data to disk temporarily and purge it from the memory--but is this reasonable, isn't this circumventing the limit? (New term: "ulimit piercing".) In the end, it's really just a question whether the limit should be in the OS or in the app. Lispers think "the lisp image is the system", and Gambit goes that route insofar as it implements such things as code reloading or threading. So it's reasonable to accept the limit is in this sub-OS, too. IIRC DJB's programs just quit on limit transgressions; only his "whole systems" (across processes) deal with failures gracefully (having stored intermediate results do disk). So the application is living in unix, unix is part of the application. But building an application completely "in the unix programming environment" is difficult. Even if you manage to pass complex data structures between processes, with data sharing (e.g. every cons cell is it's own file (how efficient does your OS handle this?), every cons is a tiny program (how efficient?..)), you'll have to implement some Lisp niceties like a garbage collector (for collecting your old unused files) yourself. If you don't, you can't just quit on failure and now have the new problem of dealing with partial failure inside an OS process (which is carrying multiple subsystem processes) while the whole process is being restricted by the OS. (I don't say it's impossible. But generally people don't seem to have found good solutions. Perl5 has the $^M variable as an emergency memory pool, but considering that this is not compiled in by default I guess almost noone is making use of it. Surely, for Gambit (which I consider better as a long-running subsystem) it would make more sense.) DJB says the limit should be enforced by the OS because he trusts the OS. It makes sense to do that, since the OS is shared by everyone and it pays off making that part of the whole system secure (once for all). But maybe the Unix concepts have partially come into the years? Nowadays you don't want to write little C tools with pipes and stream parsing anymore, right?, but instead benefit from "higher level concepts" offered by "higher level languages (and systems?)". If you're using a virtual machine like the JVM, Parrot, or also Gambit (I consider this mostly being a virtual machine too), you're running a subsystem, and it follows right quick that such a subsystem isn't integrated 100% into the host system. (So the usual 2nd-best approach is to set up OS limits to reduce possible damage from a buggy application (or otherwise overloaded system), but code the application in a way that it should never exceed the OS limits.) > > The name |fn!| in >> (define (make-reporting-thread fn! . timeout) ..) >> looks funny: fn is normally used for "function", and (pure) functions >> don't have side effects. |proc| might be a better name. But actually >> I'm not sure what it does. (Well I see that it's return value isn't >> used at all.) > >That's it. it's passed to for-each, meaning it's "called for it's >side-effects", as R5RS says. OK then I'd use |proc| as name. > > What does make-lazy do? It can't just be an alternative >> implementation of a make-promise, since those would take a thunk, not >> an 1-ary function. > >ah, right, it does this: > >(define (make-lazy fn) > (define (me) > (delay > (call/cc > (lambda (exit) > (cons (fn (lambda () (exit '()))) > (me)))))) > (me)) > >i.e., the fn creates the car part of a nearly infinite series, and can >conditionally call the exit thunk to escape the cons & terminate the >stream. Heh, interesting approach; turning an imperative (mutating state) generator into a stream, right? >That seemed like the right way to do it. Actually seems to make sense if you've got an imperative generator. (But even on Gambit, call/cc costs a little bit more than a cons or two, so rewriting that into a straight functional stream generator may be a little bit more efficient.) (BTW you should probably rename |fn| to something else here as well.) >So, overall, it folds over the stream, creating a thread for each >element which sends it's result to the mailbox of the reporting >thread. The fold counts the number of elements in the stream, and >assigns the reporter's specific field that total number. the reporter >loops over it's mailbox until it's counted as high as it's specific >field, and then it exits. > >the primoridal thread join!'s the reporter with a timeout, so that if >a thread or two hang for a long period of time, I'll ignore them and >quit anyway. So the order of the values in the output is not the same as in the stream, right? So it's not a "parallel map" really. Christian.

2 1

Re: [gambit-list] gcc stage timing
by Marc Feeley 17 Oct '06

17 Oct '06

On 14-Oct-06, at 9:59 PM, Bradley Lucier wrote: > > On Oct 4, 2006, at 12:07 PM, Marc Feeley wrote: > >> P.P.S. I wrote a genetic algorithm to figure out which >> combination of options gives the best results. Its pretty cool! >> Here's the code: > > Marc: > > Yes, this is cool. I'm wondering, can you write this as a > distributed app that I can run on my 12 opteron cores over two > systems? Most of the time nothing else is running (well, except > for Folding@Home), so it would be cool to run something like this. > Also, I see you used a population of 25 in the code you sent; when > I was doing genetic programming (where you evolve programs) for > image processing, I used a population of at least 2,000. So I'd > like to try it with a larger population, too. > > Brad Here's the program. You have to run it in the "bench" subdirectory. If you want to parallelize it, you will have to change the procedure "evaluate-group" so that it performs a "parallel map": (define (evaluate-group pop) (map evaluate-individual pop)) You also have to change the "evaluate" script (in the "bench" subdirectory) so that it doesn't clobber files that are generated by the concurrent invocations of the "evaluate" script. Finally, don't forget to modify the bin/gsc-cc-o script in the installation directory so that only a minimal set of GCC options are used (otherwise the options that the optimization program thinks it is using and the actual ones used by GCC will be different). Marc

4 6

res_query in the FFI
by Lang Martin 14 Oct '06

14 Oct '06

I'm trying to make a largish number of DNS MX type queries, thread-safely. I made a djbdns FFI, that works great. It's not thread safe -- the library internally uses global state variables. So. Started working on res_query, which seems like it's probably thread safe. However, res_query returns it's answer as a buffer of some size of u_char's. What do I do with those? There's the type conversion thing in the manual for ECDIC characters which is incomplete, and it seems as though the unicode support in Gambit is implemented through byte ports. Should I create a port? If so, can someone point me to a bit of the source code that'll get me started with that? Should I convert the answer string to some other format? I'd like to preserve any unicode characters that return from DNS. Should I return a byte vector from the function and run some scheme-land conversion on it to create a string? that'd be nice, I'm not really a C programmer. If someone can light my way, I'd appreciate it. Lang

3 4

Re: [gambit-list] strange mutation problem (bug?) (Re: "Number of arguments exceeds implementation)
by Christian 13 Oct '06

13 Oct '06

At 21:04 Uhr -0500 02.10.2006, Bill Richter wrote: >Excellent, Marc! I suppose I made a contribution to Gambit, through >tenacity if not Scheme sense. Next time, try to prove your tenacity by tracking down the problem yourself, please :) >I didn't even know that #( meant a >constant. All literal values are (or can be considered) constants. Christian.

4 7