Gambit is great: the manual should advertise better
I'm a very satisfied Gambit user, as my program became fast enough to refute some C calculations my advisor made 20 years ago. My advisor was clever enough to prove his answers were wrong, even though his program no longer runs. Thanks to Marc for offering to help optimize my code: that got me to look at my code myself and make an obvious optimization. Guillaume's profiler certainly helped locate a hotspot. But I think Gambit is a lot better than the manual would indicate to a non-expert schemer like me. I have 4 "advertising" tips: ************ blinding speed ************ The first node does some great advertising: With appropriate declarations in the source code the executable programs generated by the compiler run roughly as fast as equivalent C programs. I believe this now, with my program's speed. But it would be even better if you cross-ref'ed the node "Miscellaneous extensions", where `declarations' is explained, and if that node included Guillaume's advice to me: stick something like this at the top of your program: (declare (standard-bindings) (fixnum) (not safe) (inline) (inlining-limit 1000) (block)) One thing great about "fast as equivalent C programs" is it's the first thing a reader will see! If you bury great stuff deep in the manual, then maybe you'll restrict your users to expert programmers. The manual looks very accurate to me, but it's very technical. ************ great debugger ************ I've often felt that bad or nonexistent debuggers is the Achilles heel of Scheme. But it looks to me like Gambit has a great debugger! I think the first node "The Gambit-C system" shoul say something like: `Gambit has a good debugger. See the examples in the nodes "Debugging commands" and "Procedures related to debugging" for examining variables in case of an error, and breakpoints and single-stepping.' ************ Case sensitive ************ The node "Runtime options for all programs" says that Gambit is case-sensitive by default. I think that's a huge plus, and should be mentioned in the 1st node. I think all mathematically-inclined folks will agree with me, as one typically writes, "Let X be a set, and let x be an element of X", which turns naturally into Scheme (let ([x (car X)]) ...) I think you should target folks who use DrScheme, written by your R6RS co-Editor Matthew Flatt. DrScheme is really slow, but it has some real pedagogical advantages, I think, including case-sensitivity. I would've used Gambit years ago if I'd known it was case-sensitive. ************ read-all: don't compile data ************ I finally understood some advice Guillaume posted: don't load/include big data files: it slows down gsc/gcc unbearably. You don't need to load/include, because of Gambit extensions like read-all. This helps with my Curtis-algorithm program, but it was crucial for an example program, where I gave an independent refutation of an error my advisor made in t = 71, where I differentiated a polynomial with 75,000+ terms of length 10! In 16.6 minutes! Look at this short program: (declare (standard-bindings) (fixnum) (not safe) (inline) (inlining-limit 1000) (block)) (include "adem.scm") (define inport (open-input-file "tag-data-t-71")) (define X (read-all inport)) (close-input-port inport) (pretty-print (length X)) (pretty-print (Poly-first X)) (time (pretty-print (Poly-first (D X)))) 75299 (4 8 4 5 3 5 9 3 5 7 7) (3 4 4 2 3 5 3 5 9 7 7 7) (time (pretty-print (Poly-first (D X)))) 998093 ms real time, 16.6 minutes 997940 ms cpu time (997840 user, 100 system) 3204 collections accounting for 365525 ms real time (365690 user, 30 system) 112648202368 bytes allocated 13318 minor faults 3 major faults BTW can someone explain this figure, 112648202368 bytes allocated? Did I really use 112.6 gB? My machine "only" has 1 gB of memory. An important extension to Scheme like `read-all' maybe should be more prominently mentioned, and it's usefulness stressed. The description is fine, but I didn't find these examples very illuminating: - procedure: read-all [PORT [READER]] This procedure repeatedly calls the procedure READER with PORT as the sole argument and accumulates a list of each value returned up to the end-of-file object. The procedure `read-all' returns the accumulated list without the end-of-file object. If it is not specified, PORT defaults to the current input-port. If it is not specified, READER defaults to the procedure `read'. For example: > (call-with-input-string "3,2,1\ngo!" read-all) (3 ,2 ,1 go!) > (call-with-input-string "3,2,1\ngo!" (lambda (p) (read-all p read-char))) (#\3 #\, #\2 #\, #\1 #\newline #\g #\o #\!) > (call-with-input-string "3,2,1\ngo!" (lambda (p) (read-all p read-line))) ("3,2,1" "go!")
On Feb 6, 2005, at 10:07 PM, Bill Richter wrote:
(time (pretty-print (Poly-first (D X)))) 998093 ms real time, 16.6 minutes 997940 ms cpu time (997840 user, 100 system) 3204 collections accounting for 365525 ms real time (365690 user, 30 system) 112648202368 bytes allocated 13318 minor faults 3 major faults
BTW can someone explain this figure, 112648202368 bytes allocated? Did I really use 112.6 gB? My machine "only" has 1 gB of memory.
Probably, yes. Each of the 3204 garbage collections collected on average about 30+ megabytes of data, which it could reuse until the next GC, etc. My guess is that if you doubled the amount of real memory to 2GB the number of collections would drop to a few hundred. Brad
I'm a very satisfied Gambit user, as my program became fast enough to refute some C calculations my advisor made 20 years ago. My advisor was clever enough to prove his answers were wrong, even though his program no longer runs. Thanks to Marc for offering to help optimize my code: that got me to look at my code myself and make an obvious optimization. Guillaume's profiler certainly helped locate a hotspot.
But I think Gambit is a lot better than the manual would indicate to a non-expert schemer like me. I have 4 "advertising" tips:
Thanks for the advertising tips. I will see what I can do. It should be clear now that I don't like hype... What you read (in the docs) is what you get. Marc
But I think Gambit is a lot better than the manual would indicate to a non-expert schemer like me. I have 4 "advertising" tips:
Thanks for the advertising tips. I will see what I can do. It should be clear now that I don't like hype... What you read (in the docs) is what you get. Thanks, Marc. That's an admirable attitude, and I wouldn't want more hype than what you already wrote on page 1: With appropriate declarations in the source code the executable programs generated by the compiler run roughly as fast as equivalent C programs. But there's a real question, which I don't know the answer to: who do you want to use Gambit? If you only want experts like Brad (who 10 years ago told me to use Gambit, but I was intimidated (and thanks for the GC tip!)) to use Gambit, the manual is fine. Maybe you only mean Gambit to be a tool of the serious scientific computing community. But maybe you want all DrScheme users who need a real speed increase! If so, I recommend explaining how usable Gambit is to non-experts like me. Since Gambit is case-sensitive, the DrScheme users only need to port various functions, mostly explained in their book HtDP, so you can leave that as an exercise (which even I had no trouble with), or write blinding-speed versions yourself. I know you can write much faster quicksort and mergesort functions than mine. I largely harp about usability because I posted on Oct 17, 2004: [gambit-list] help with port from DrScheme? ... Does Gambit have a case sensitive mode? I didn't see anything like this in the info files, ... That was pretty dumb of me, because reading your manual in Emacs info, (Info-search "case-sensitive") immediately takes me to the node that explains it (a bit opaquely): The `s' option selects standard Scheme mode. In this mode the reader is case-insensitive and keywords are not recognized. The `S' option selects Gambit Scheme mode (the reader is case-sensitive and recognizes keywords which end with a colon). By default Gambit Scheme mode is used. Well, I get nothing from (Info-search "case sensitive"). But my point is that I got no response! Nobody posted, "RTFM! By default Gambit is case-sensitive!" -- Best, Bill
Thanks, Marc. That's an admirable attitude, and I wouldn't want more hype than what you already wrote on page 1:
With appropriate declarations in the source code the executable programs generated by the compiler run roughly as fast as equivalent C programs.
That is not hype... it is reality!
But there's a real question, which I don't know the answer to: who do you want to use Gambit? If you only want experts like Brad (who 10 years ago told me to use Gambit, but I was intimidated (and thanks for the GC tip!)) to use Gambit, the manual is fine. Maybe you only mean Gambit to be a tool of the serious scientific computing community.
I think Gambit is suitable for the development of many real-world applications. Execution speed is not a problem, there are many extensions for real-world applications (FFI, threads, ...), and the debugging environment is the best that I know. The main problem is that marketing takes time and currently Gambit is mostly a one man effort (actually Brad Lucier has helped quite a bit in the past 3-4 years). I'm hoping that the features of Gambit 4 and the new license will attract more developpers/contributors to Gambit and that this will broaden its user base. Marc
I think Gambit is suitable for the development of many real-world applications. Execution speed is not a problem, there are many extensions for real-world applications (FFI, threads, ...), and the debugging environment is the best that I know. Thanks. I thought that your debugger was light-years ahead of other Scheme other debugger, but I wasn't sure. BTW Felleisen posted once that Chez Scheme (costs $$) had a good debugger. Could you compare? The main problem is that marketing takes time and currently Gambit is mostly a one man effort (actually Brad Lucier has helped quite a bit in the past 3-4 years). I'm hoping that the features of Gambit 4 and the new license will attract more developers/contributors to Gambit and that this will broaden its user base. Great. But (I hate to keep bugging you) would you even want hordes of non-experts porting their DrScheme programs to Gambit and saying, "Wow, that's blinding speed!!!" Maybe for now you'd like to keep it to serious programmers, either scientific computers, like Brad, or developers/contributors. The non-experts (like me) might tax the resources of your 2-man team, especially if they clutter up the list, as I'm doing :^0 Because you don't have to do anything to attract their attention: just say Gambit is case-sensitive and has a real debugger, maybe offer a tip about declarations, maybe read-all.
With appropriate declarations in the source code the executable programs generated by the compiler run roughly as fast as equivalent C programs.
That is not hype... it is reality! That's amazing, Marc! Maybe we're having a cultural problem, since I'm from the US of A, where `hype' and `advertising' have much more positive connotations :) I wasn't trying to recommend anything sleazy.
But there's a real question, which I don't know the answer to: who do you want to use Gambit? If you only want experts like Brad (who 10 years ago told me to use Gambit, but I was intimidated (and thanks for the GC tip!)) to use Gambit, the manual is fine. Maybe you only mean Gambit to be a tool of the serious scientific computing community.
can someone explain this figure, 112648202368 bytes allocated? Did I really use 112.6 GB? My machine "only" has 1 gB of memory.
Probably, yes. Each of the 3204 garbage collections collected on average about 30+ megabytes of data, which it could reuse until the next GC, etc. My guess is that if you doubled the amount of real memory to 2GB the number of collections would drop to a few hundred. Thanks, Brad! I re-ran my job on machine with 1.28 GB, and got almost the same: 112648201888 bytes allocated, and still 3204 collections. If you (or someone else) would be willing to try it, I put this short program in my Curtis algorithm distrib at the top of my web page <http://www.math.northwestern.edu/~richter/Richter-Curtis-algorithm.tar.gz> which has some new mathematical improvements, which don't help here, slightly extending some shortcuts in Tangora's book on Lambda. I just did t = 0->65 in 37.9 minutes, beating my old mark of 63.0. Please answer a question, asked by the hotshot computer Math professor here John Franks (a Rice pal of Clarence): does gsc/gcc implement a garbage collector? I figure it must, looking at this 112.6 GB, which you say is legit (and I've seen 435 GB too). But John points out that writing a garbage collector is serious work, and gcc doesn't have one. Back to the manual, DrScheme is all about teaching their book HtDP, so the exercise I did is mandatory: porting the DrScheme functions to Gambit. Everything I did is described in HtDP, either worked out or exercises. So there's no need for Gambit to write the blinding-speed versions of these. And moderately serious DrScheme programmer who wants Gambit speed can & must roll their own. I wouldn't have mentioned it except that it's possible the DrScheme version of quicksort is faster than mine, which I cribbed from HtDP. It starts with `(polymorphism)', so I can't read it, and it has vectors... Of course DrScheme is no competitor of Gambit: it's slow, and as you posted on cls, there's a lack of interest in numerical methods. I learned a lot from HtDP (which has the usual over-hyped title, as in SICP, EOPL, PLAI...), and it's the best Scheme book I've seen, although I wish they'd given a better mathematical treatment of some matters (as I'm sure Felleisen could have done). DrScheme has a nice Emacs-like editor, which you can't modify (even to rebind function keys!) without hacking the source code, in order to maintain classroom uniformity. That shows their whole focus is teaching high school and college Scheme courses, not a fast/real Scheme implementation. And a debugger question: I'm not much of a C programmer, but I've done a lot of `gdb emacs', and I'm used to being able to skip over function calls. I got some real use out of the Gambit debugger (which is far better than any Scheme debugger I've ever seen), but I didn't see how do this. I tried leap, and step-level-set!. Also, the last 3 nodes here don't seem to have anything to do with debugging: Debugging * Menu: * Debugging model:: Debugging model * Debugging commands:: Debugging commands * Procedures related to debugging:: Procedures related to debugging * Console line-editing:: Console line-editing * Emacs interface:: Emacs interface * IDE:: IDE -- Best, Bill
Thanks, Brad! I re-ran my job on machine with 1.28 GB, and got almost the same: 112648201888 bytes allocated, and still 3204 collections.
Why do you think 112GB allocated is incorrect? This is the memory "allocated" by the program. It is not the maximal live data used by the program. Of course most of the allocated data is reclaimed by the garbage collector. The "allocated bytes" statistic indicates the sum of all the size of objects allocated. For example, on a 32 bit machine, each call to "cons" counts for 24 bytes (each field of the pair takes 4 bytes (header, car, cdr) and there is an extra factor of 2 for the space reserved in the to-space so that the garbage collector can copy the pair if it is live when the GC is performed). Marc
Please answer a question, asked by the hotshot computer Math professor here John Franks (a Rice pal of Clarence): does gsc/gcc implement a garbage collector? I figure it must, looking at this 112.6 GB, which you say is legit (and I've seen 435 GB too). But John points out that writing a garbage collector is serious work, and gcc doesn't have one.
Gambit's runtime system includes a garbage collector. This garbage collector is written in C and is precise (it is not a conservative GC). This is possible because the C code generated by the Gambit compiler manages its own runtime "stack" explicitly, and the GC knows how to parse the stack frames constructed by the compiler. I'm not sure what you mean by "does gsc/gcc implement a garbage collector?" because the gcc ***compiler*** actually contains a garbage collector (but not the code that gcc generates). Moreover Gambit can be compiled with any C compiler, so the fact that Gambit contains a garbage collector has nothing to do with gcc. Marc
Thanks, Marc! Real quick first: John Franks observed that the gsc-created C code doesn't have any calls to `free' or `*alloc', but I just checked that libgambc.so matches both. So does libgambc.so, called by `gcc ... -lgambc', set up the GC I get by gsc/gcc/a.out?
Please answer a question, asked by the computer hotshot Math professor here John Franks (a Rice pal of Clarence): does gsc/gcc implement a garbage collector? I figure it must, looking at this 112.6 GB, which you say is legit (and I've seen 435 GB too). But John points out that writing a garbage collector is serious work, and gcc doesn't have one.
Gambit's runtime system includes a garbage collector. This garbage collector is written in C and is precise (it is not a conservative GC). This is possible because the C code generated by the Gambit compiler manages its own runtime "stack" explicitly, and the GC knows how to parse the stack frames constructed by the compiler. I'm not sure what you mean by "does gsc/gcc implement a garbage collector?" because the gcc ***compiler*** actually contains a garbage collector (but not the code that gcc generates). Moreover Gambit can be compiled with any C compiler, so the fact that Gambit contains a garbage collector has nothing to do with gcc. May I recommend you explain this in the Gambit manual? And speaking of the manual, I haven't yet understood your position about non-experts using Gambit for a huge speed increase. Should I post what I'm thinking to cls, or should I not? Do you want to wait until version 4 is out of beta? My guess is that it's a very stable beta, as Brad has been posting about it to cls for years. Lotta interesting stuff here I should pass on: precise vs conservative GC, gcc has an internal GC. I knew you didn't have to use gcc. By "gsc/gcc", I meant that I type % gsc tag-t-71.scm % gcc -O2 -L. -I. tag-t-71.c tag-t-71_.c -lgambc % ./a.out and (as I just ran again) it took 16.8 minutes, incl 6.0 minutes of GC, using 112.6 GB on a 1GB machine, and `ps -aux' says I never used more than 8%. Looks like Gambit must be flushing data out of memory in an amazing & efficient way. Especially as mzscheme ran this job 6.2 times slower, i.e. in 104.7 minutes, with only 7.5 minutes of GC: 75299 (4 8 4 5 3 5 9 3 5 7 7) (3 4 4 2 3 5 3 5 9 7 7 7) cpu time: 6282240 real time: 6316034 gc time: 455500 Here's my call to strings on libgambc: (morse)PolyTree> strings /rhome/richter/my-gambit/lib/libgambc.so | grep alloc ___alloc_mem ___alloc_scmobj ___alloc_global_var ___alloc_rc ___bytes_allocated malloc Heap overflow while allocating stack marker (morse)PolyTree> strings /rhome/richter/my-gambit/lib/libgambc.so | grep free ___free_mem ___free_UCS2STRING ___free_NONNULLUCS2STRINGLIST free freeing h=%p
participants (3)
-
Bill Richter -
Bradley Lucier -
Marc Feeley