Gambit-list Février 2008

gambit-list@iro.umontreal.ca

29 participants
37 discussions

Edge triggered I/O with libevent
by Adam Langley 04 Avr '13

04 Avr '13

Gambit uses the select call on POSIX systems to multiplex IO operations. This is fine for small numbers of file descriptors, but falls apart quickly with large numbers[1]. libevent[2] is a wrapper library around a number of modern IO multiplexing systems (epoll on Linux, kqueue on the BSDs as well as poll and select). Attached is a patch which converts Gambit to using edge triggered I/O with epoll and libevent. This also requires a patch (attached) of libevent since the standard distribution doesn't include edge triggering support and the upstream appears to be on holiday. (epoll is required because I don't have a BSD box on which to develop edge triggering support for kqueue) This patch (for Gambit) preprocesses away to nothing unless --enable-libevent is passed to the configure script. (Note that, since this patches the configure.ac script, you'll need to run autoconf to regenerate). I hope I've not broken anything, but at this stage I've only tested that the console and tcp ports work. This patch (in the future) should also allow people, who write programs which use socket calls directly, to integrate into the main event loop at some point. Testing is more than welcome and, if you find a form of IO which is broken, please email me. [1] http://monkey.org/~provos/libevent/libevent-benchmark2.jpg [2] http://monkey.org/~provos/libevent/ -- Adam Langley agl(a)imperialviolet.org http://www.imperialviolet.org (+44) (0)7906 332512 PGP: 9113 256A CC0F 71A6 4C84 5087 CDA5 52DF 2CB6 3D60

2 2

New Gambit wiki
by Marc Feeley 18 Mai '08

18 Mai '08

I have just installed a recent distribution of MediaWiki and PHP, and the reCaptcha extension. In order to do this I had to use a different machine (dynamo.iro.umontreal.ca) and copy all the pages from the old wiki to the new wiki. When a page edit is saved a visual captcha will now appear to make sure the edit has been submitted by a human user. Please let me know if you find broken links, missing information, or other issues. Marc

3 6

Strange Networking Bug on MinGW (LONG)
by François Magnan 04 Avr '08

04 Avr '08

Hi, I worked hard to isolate one of my bugs and I found a rather disturbing minimal example that reproduces the bug. It can be observed in all the latests versions of Gambit (including 4.2.1). Basically, I start a minimal multithread web server and in another thread I start a loop that calls a serie of open-tcp-client to different ips in the local network. The default web browser is launched to get the page served by the web server. On MacOSX Leopard everything is fine. On WindowsXP (MingW compiled version of Gambit) the page takes an unnaceptable loooong time to load. The website index.html must require multiple HTTP GET to render (a page with multiple pictures for example). What is the most disturbing is that if I replace the calls to "read-u8- line" by simple "read-line", the bug disapears. The bug is therefore somehow related to "read-u8". Here is the sample code. You have to define "Server-ROOT" at the top so it points to a website on your local hard-drive. I didn't try on Linux but to try it the code will have to be adapted a little bit (how to launch the default browser on Linux?). What could lead to this huge difference between a sequence of read-u8 and read-line on WinXP??? Thank you, Francois Magnan ;;; Gambit Networking Test Example ;;; ;;; Minimal web server that gets incredibly slow on WinXP (MinGW) if it is run in parallel with another thread that does a network scan. ;;; ;;; If the "read-u8-line" calls are replaced by "read-line" the bug disapears. ;;; ;;; On MacOSX Leopard the bug is not present. ;;; ;;; Was tested with versions: Gambit-C 4.0.1, 4.1.2, 4.2.1 ;;; ;;; ;;;; Server-ROOT ;;; ;;; ******************* ;;; Put here the path to a static website with many files to load to render index.html ;;; ******************* ;(define Server-ROOT "/Users/magnan/TELOS/interface/TELOS-Desktop") (define Server-ROOT "C:/TELOS/interface/TELOS-Desktop") ;;; ;;;; main ;;; (define (main) (start-server) (start-broadcast) (start-client) ) ;;; ;;;; Minimal Web Server ;;; (define (start-server) (let ((rpc-command-queue (open-tcp-server (list port-number: 8000)))) (thread-start! (make-thread (lambda () (let loop () (let ((incoming-connection (read rpc-command-queue))) (force-output) (serve-connection incoming-connection)) (loop))))))) (define (serve-connection incoming-connection) (thread-start! (make-thread (lambda () (let* ((line (read-u8-line incoming-connection)) (splitted (split-string line #\space)) (path (cadr splitted))) (read-all-headers incoming-connection) (force-output) (with-exception-catcher (lambda (e) (##display-exception e (current-output-port))) (lambda () (serve-file path incoming-connection)))))))) (define (read-all-headers incoming-connection) (let ((line (read-u8-line incoming-connection))) (pp (list "read header line" line)) (force-output) (if (not (equal? line "")) (read-all-headers incoming-connection)))) (define (split-string str sep-char) (call-with-input-string str (lambda (p) (read-fields p sep-char #f)))) (define (read-fields port sep inc-sep?) (read-all port (lambda (p) (read-line p sep inc-sep?)))) (define (serve-file path incoming-connection) (let* ((complete-path (string-append Server-ROOT path)) (fileInfo (file-info complete-path)) (size (file-info-size fileInfo)) (mimetype (compute-mimetype complete-path)) (header (compute-get-result-header size mimetype))) (display header incoming-connection) (output-file-data complete-path incoming-connection) (close-port incoming-connection))) (define (compute-get-result-header size mimetype) (string-append "HTTP/1.0 200 OK\r\n" "Content-type: " mimetype "\r\n" "Content-length: " (number->string size) "\r\n" "\r\n")) (define (output-file-data filePath connection) (with-input-from-file filePath (lambda () (letrec ((output-file-data-rec (lambda () (let ((rc (read-char))) (if (not (equal? rc #!eof)) (begin (write-char rc connection) (output-file-data-rec))))))) (output-file-data-rec))))) (define (compute-mimetype filePath) (let* ((ext (path-extension filePath)) (rext (and (string? ext) (substring ext 1 (string-length ext))))) (case (string->symbol rext) ((jpg jpeg JPG JPEG) "image/jpeg") ((gif GIF) "image/gif") ((png PNG) "image/png") ((html HTML htm HTM) "text/html") ((js JS) "text/javascript") ((css CSS) "text/css") (else "application/octet-stream" )))) ;;; ;;;; start-broadcast ;;; (define (start-broadcast) (thread-start! (make-thread (lambda () (let* ((hostname (host-name)) (hi (host-info hostname)) (host-ips (host-info-addresses hi)) (host-ip (car host-ips))) (letrec ((network-broadcast (lambda (i) (if (= i 255) #t (begin (u8vector-set! host-ip 3 i) (call-addr host-ip) (thread-sleep! 0.2) (network-broadcast (+ i 1))))))) (network-broadcast 1))))))) (define (call-addr target-addr) (pp (list "trying to reach" target-addr)) (with-exception-catcher (lambda (e) #f) (lambda () (let ((port (open-tcp-client (list server-address: target-addr port-number: 5000)))) (with-exception-catcher (lambda (e) #f) (lambda () (input-port-timeout-set! port 0.1) (output-port-timeout-set! port 0.1) (write (list 1 2 3) port))) (close-port port))))) ;;; ;;;; Client ;;; (define (start-client) (if (equal? (get-host-os) "WinXP") (shell-command "start http://localhost:8000/index.html") (shell-command "open \"http://localhost:8000/index.html\""))) (define (get-host-os) (with-exception-handler (lambda (e) "WinXP") (lambda () (let* ((p (open-process '(path: "/usr/bin/uname" arguments: ("-s")))) (r (read-line p))) (close-port p) r)))) ;;; ;;;; read-u8-line ;;; (define (read-u8-line port) (letrec ((read-loop (lambda (acc) (let ((b (read-u8 port))) (if (or (equal? b 13) (equal? b 10) (equal? b #!eof)) (begin (read-u8 port) acc) (read-loop (string-append acc (list- >string (list (integer->char b)))))))))) (read-loop ""))) ;;; ;;;; start main ;;; (main)

3 5

Google Summer of Code
by Marc Feeley 18 Mar '08

18 Mar '08

Hello Gambit and Snow users. I will be applying to the Google Summer of Code program as a mentoring organization and propose projects for advancing the development of Gambit, Snow, and Termite. I'd like to start a collective brainstorming exercice to come up with a list of specific projects. If you are interested in participating as a student, please respond to this message (preferably on the mailing list). Please describe briefly what you would like to work on. If you cannot participate (e.g. you are not a student) but have project suggestions, then please speak up on the mailing list. Some projects ideas that come to mind: - native thread support - real-time/parallel GC - closer integration of Termite and Gambit - Gambit packages - web framework - documentation - François Pinard's projects (see MSLUG mailing-list archive) Marc

11 22

Profiler still up to date?
by David St-Hilaire 29 Fév '08

29 Fév '08

Hello! I'm currently trying to use Guillaume Germain's profiler (http://www.iro.umontreal.ca/~germaing/statprof.html) to well, em, profile my program ;). But I am encoutering severall problems... The exemple that comes with Guillaume's profile works just fine on my machine, but when I try to use it with my own code, it doesn't work... I have something like: include "statprof.scm") (profile-start!) (load "opengl") (load "glu") (load "glut") (load "Question2") (load "Question2-ui") (profile-stop!) (write-profile-report "profile-report") where the main execution occurs in Question2-ui. When I compile all this, i.e. compiles opengl, glu, glut, Question2 and Question2-ui to .o1 files and then compile my profile.scm file to c -> native code, well the program runs fine, but when I stop it, I get: *** ERROR IN write-profile-report -- Wrong number of arguments passed to procedure (max) With a little investigation, I found out that statprof's *bucket* variable is still empty when the profiling has ended... :( Guillaume's code that should fill the *buckets* looks like: (define (profile-heartbeat!) (##continuation-capture (lambda (cont) (##thread-heartbeat!) (let ((id (identify-continuation cont))) (pp id) (if (not (eq? id 'unknown)) (let ((bucket (assoc (car id) *buckets*))) (set! *total* (+ *total* 1)) (if (not bucket) (begin (set! *buckets* (cons (cons (car id) ;; fixme: arbitrary hard limit ;; on the length of source ;; files (make-vector 5000 0)) *buckets*)) (set! bucket (car *buckets*)))) (vector-set! (cdr bucket) (cadr id) (+ (vector-ref (cdr bucket) (cadr id)) 1)))))))) As you can see, I've added a (pp id) to see what happens, and I only get some unknown printed in the console... So how come code doesn't get recognized in the profiler?? Thank you! ^_^ David

2 1

--enable-single-host and --enable-gcc-opts
by Joel Borggrén-Franck 29 Fév '08

29 Fév '08

Do --enable-single-host and --enable-gcc-opts affect the compilation of the compiler, or the compiler in itself? IE if I compile using both options and package a binary, will hosts with low amount of ram that install my binary have difficulty using them? /Joel

4 4

Qwaq releases Hydra multi-core Squeak VM � The
by Bradley Lucier 28 Fév '08

28 Fév '08

Marc: There's currently a lot of interest in education and in funding agencies for multi-core programming strategies; here's another implementation. Gambit may be good for this. Brad http://news.squeak.org/2008/02/21/qwaq-releases-hydra-multi-core- squeak-vm/

10 18

Re: [gambit-list] Qwaq releases Hydra multi-core Squeak VM « The
by James Long 27 Fév '08

27 Fév '08

Thanks for the detailed reply Christian - I'll merge this conversation back into the Gambit list since there's some good stuff here. On Tue, Feb 26, 2008 at 1:51 PM, Christian Jaeger <christian(a)pflanze.mine.nu> wrote: > James Long wrote: > > Yes, all I need is a *little* bit of sharing so that I don't have to > > copy potentially large amounts of data every frame (frame being a > > scene rendering pass, which happens ~60 times a second). Having a few > > structures (which would receive a lot of attention) shared is the > > efficiency boost I'm looking for. How would you do this, however? > > Using shared memory objects and mmap? > > Yes. Or pthreads and multiple Gambit runtimes in the same process, but > that might involve making the Gambit runtime not use any C globals, first. I thought about that at one point, but I just don't know enough about the Gambit runtime to know if that's feasible or not. > > > > I'm not sure how you could > > share Gambit objects across processes, since the Gambit system is so > > isolated within itself. > > Not sure what you mean with "so isolated"; of course if you want the > Gambit gc alone to collect unreachable memory it needs access to all memory. > > There are several ways approaching it. > > (A) use the refcount in still objects. One Gambit engine is the master, > the others are slaves and increase/decrease the refcount as they need. > This requires either the ability to run multiple Gambit runtimes in the > same process, or an extension to make Gambit allocate still objects from > another heap. You would have to lock the the whole memory allocator per allocation, and every object per refcount increase/decrease. I'm not sure if it would be safe for the allocator to be garbage collecting across the buffer in parallel either - because it's just scanning for memory which has a refcount of 0 it might be safe. Still though, seems like you would get a bad case of lock contention. > (B) use wrapper objects around shared storage outside of the Gambit > heap. E.g. you would use plain C arrays/datastructures in shared memory, > each carrying a refcount, and the finalizers in the normal Gambit FFI > objects around them can decrement the refcount and free the C > datastructure wenn it drops to zero. This could work, but the refcount would only be for each runtime. Also, I'm assuming you'd have to use a separate memory allocator to manage this external heap. There's probably thread-safe memory allocators out there. As for garbage collection, I think you'd have to manually free these objects. > For B you want a module system with parametrization so that you can run > your code calling e.g. f64vector-ref transparently for a f64vector-ref > implementation which really accesses the C data. > > For performance, those ops would be (transparently, by the module > system) inlined into the client module body as ##c-code forms. > > The ops would normally have to do a pointer indirection through the FFI > object; there are several ways around that (for inside tight loops): > > (- you would just write those parts in C) > > - you first get the address of the C object (as a fixnum), then act upon > that address with a second set of ops; e.g.: > > (define (f64vector-map! outvec invec fn) > ;; outvec and invec are (possibly) FFI objects > (let ((inadr (f64vector-address invec)) > (outadr (f64vector-address outvec)) > (len (min (f64vector-length invec) (f64vector-length outvec)))) > (let lp ((i 0)) > (if (< i len) > (begin > (direct-f64vector-set! outadr i > (fn (direct-f64vector-ref inadr i))) > (lp (+ i 1))))))) > > That's easy to implement, but a little less pretty since you have then > an additional "fast-access" operator set. Also using fixnums looses type > safety, of course; but you want to avoid runtime type checking in > performance critical places anyway, and I've got the idea that the > module system would let the library know whether it is being used in > safe or unsafe mode, and the library implementing the shared f64vector > ops could then decide whether it would use a fixnum (in unsafe mode) or > a typed structure (in safe mode), so in safe mode you'd still get type > checking of the above; for perfect safety, this would require analyzing > the code for "leaks" where such addresses / address objects are passed > on to places with different safety settings, and either throw an error > or warning if found, or wrap/unwrap the object at those boundaries--this > is quite a bit involved, though, but could be realistic, after all I'm > already into lexical / (type / dataflow) analysis anyway. Perfect safety > would also require static checking that the ffi object is not released > as long as the adresses taken from it are still being used. > > - The compiler (/ module system) would recognize places where such > optimizations can be done automatically. This will require similar > involvement as the above safety analysis mentioned above. > Yeah - I'd say disregard type checking. We're already pretty far down the rabbit hole, and essentially coding like C at this point. As as far as being released, like I said before, I would just say these special objects should be manually freed. I'm not going to be doing this a whole lot, but just for a few special buffers that will already be receiving a lot of attention. > > A fundamental thing to keep in mind is that shared objects can only ever > contain references to other shared objects, not to objects residing in > any private Gambit heap. The easy way is simply not using any references > at all, of course; working on homogenous vectors first may offer the > most benefit for the least work. Shared objects containing references > will be difficult for a few reasons; one is how local objects should be > transformed into shared ones (if at all (should we just get an error?)), > another one is garbage collection: either you resort to reference > counting there, too (which would at least be consistent, but has it's > usual problems with cyclic data structures, and won't be fast on > modern/future machines), or implement a garbage collector that stops the > world (not nice, right?), or implement an incremental asynchronous > garbage collector (which I think will not work with mutation, here is > the point where I'm not sure whether transparently transforming mutating > programs to functional ones would be possible, and if so, worthwhile). I > think a pool of shared objects containing references to other shared > objects is best thought of as a database; now this is the area which > I've been talking about last month in the thread about Mnesia. > That's certainly where it starts getting tricky. And it's where I feel like I want the system that deals with shared memory to be as explicit and restrictive as possible. It should almost be discouraged because of the complexities of it. Only the people who want to get dirty can allocate these special objects, manipulate their data, and deallocate them. > > Regarding freeing memory: the easy way is to use e.g. linux tmpfs and a > separate file for each object. This should work well enough when only > big objects are used. If it should work for small objects, probably one > of the free memory allocation implementations could be used on a single > file. (For many very small objects and purely functional data > structures, a mostly-lockfree copying GC, as mentioned in the Mnesia > thread, would be better.) > I'll have to look at the Mnesia thread and mostly-lockfree copying GC. I'm still unclear how you could have a thread-safe GC, even with refcounts. > > > > > > I would be interested in this kind of work. > > > > Jeff Read's comment about supporting Unix domain sockets in Termite is > > interesting also. I didn't know about Unix domain sockets - I was > > looking all over the web for that exact thing. I still think there > > would be too much overhead serializing/deserializing structures, but > > that would certainly help avoid so much overhead. > > > > > > Unix domain sockets are only marginally faster than local TCP/IP; you > probably won't usually notice the difference with Termite, since the > serialization/deserialization overhead will overshadow it (but I'd love > to be proven otherwise; I think there is room for speed improvements in > the ser/deser routines, at least if it gives up on cyclic data > structures, which may not be a problem for a purely functional language). > > (Feel free to reply to the list for further answers.) > > Christian. > Good to know. At some point I'll play around with them and see how well they work. -- James Long Coptix, Inc. longster(a)gmail.com

2 1

[PATCH] Patch to speed string comparisons and hashing
by Bradley Lucier 26 Fév '08

26 Fév '08

It cuts about 5% off some benchmarks.

3 6

Looking for an industrial partner for a research grant
by Marc Feeley 25 Fév '08

25 Fév '08

Dear Gambit users and MSLUG members, I am planning to submit a substantial research grant proposal which requires participation from industry. The grant will have a direct impact on the development of Gambit and related tools. I am currently looking for an industrial partner which can benefit from the project and which can participate in the research. The details of the research proposal will be developed with the partner. A cash contribution is **not** required, but a real involvement is necessary (for example: engineer's time to interact with the project members, equipment donation (multicore, FPGAs, ...), salary/space for an intern, etc). It is necessary to demonstrate impact on the canadian economy, so it is best if the partner is based in Canada or has a canadian branch. I am particularly interested in partners in the game industry (Ubisoft, Electronic Arts, ...), in web technology (Google, Amazon, ...), in embedded systems, telecommunication and hardware (Intel, AMD, Xilinx, ...). If you are interested please email me. The proposal must be submitted in a month an a half, so it is important to act soon. Marc

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Gambit-list Février 2008