Andrew Lentvorski wrote:
Christian Jaeger wrote:
From what I've seen, the experiences of the GHC implementors working on a parallel system (and at first, iirc, a concurrent GC?) may be worthwhile readings.
How does Erlang handle this? The latest versions handle multicore/multiprocessor if I remember correctly.
Just to make clear about what we are talking: Erlang "processes" do not share state (in the form of mutations that could be directly visible in other processes), from the point of view of a user they are *not* shared memory threads. And, not only does the language not offer direct cross-process mutations, but their implementation traditionally did use a separate heaps for every process, even in their user-space scheduled implementation, so not even the *implementation* is/has been using shared memory (except probably for (immutable) program code). As long as they keep it that way, it's dead easy running multiple OS threads, you simply use the OS communication instruments for passing messages between the threads. Whether using OS threads is significantly faster than OS processes in this case is what I have been wondering about several times, numbers would be interesting here (I guess it would eliminate (or reduce) the costs of setting up MMU tables during context switches between OS threads of the same process, other than that, I see no reason for a speedup).
Now Scheme is obviously different in that it offers mutation and if you want to make the mutations been passed over to other processes (like the heavy math vector calculation people would like), the shared-nothing approach doesn't work. (I've already suggested using separate shared memory for such tasks.)
But on top of that, even if you're using Termite (which does not support mutation): even if you strip shared-memory mutation from Scheme completely (e.g. apply the Termite semantics to the underlying Scheme system), there is one thing left: Termite is so fast in passing big data structures between Termite "processes" (Scheme threads) because Termite "processes" are employing underlying shared memory storage. So no serialization/deserialization has to be done for local data transfers. Now you can't simply run multiple Termite "threads" (i.e. underlying Scheme threads) on different OS threads anymore, because you've got a heap that needs synchronization. You now need a concurrent GC etc., something like what non-green threaded Java virtual machines do (or something less scalable like the global interpreter lock of Python).
Now that being said, Erlang people have been discussing moving to using shared heaps, just like Termite. Now the interesting question is: have they managed to efficiently implements heaps being shared *across OS threads*? Note that if they did not (e.g. one heap is only ever green-threaded), they are simply using the Gambit approach, but saving the above mentioned MMU overhead. This would be the approach of running multiple Gambit interpreters in separate pthreads in the same OS process. This should be quite easy (but, as mentioned, I wonder how much of a difference that will make?). Do you have pointers to corresponding Erlang posts/docs?
BTW the chicken people have also had some interesting discussion about GSoC participation in the last few days (don't be sad about the difference in volume), also including a few points about threading. Gambit does have a better chance of doing fancy stuff with garbage collection (because of it handling the continuation stack explicitely), of course.
Is there some documentation about this somewhere? I'd love to read it.
(I've got that knowledge gradually over time from experiments / mailing list posts / reading the sources. I suggest to start by reading the paper Marc has posted on March 5th.)
- web framework
Should I mentor on this? I've actually got some actual professional (well, that doesn't mean particularly successful) experience in this area (with an XML background). But I'd rather prefer having a module system done before continuing writing suboptimally modularized code as I did up to now.
I'm not sure a "web framework" is that great an idea. "Frameworks" work best when they are scratching a particular itch. In addition, any good "web framework" for Scheme is likely to be not very Gambit specific.
Well, of course I've got some ideas on how that framework should look like.
It could be open to plug Termite in.
Of course you've got a point in that you could most probably port it to other Scheme systems, and the same of course applies to frameworks developed on other systems, they could be ported to Gambit. But if there is no Scheme system agnostic organization out there willing to take on the task of implementing such libraries, then this is being left to the individual system development organizations. I see no reason for Gambit playing purely a library consumer, as opposed to contributor as everyone else. Of course creating such a framework could also leverage (by porting) *parts* of other projects. Two things to keep in mind are: (a) system specifics to make something efficient; (b) module system specifics to make something particularly elegant. Part (a) could help show off Gambit's performance, of course (but yeah, is there a user out there right now who wants this?), part (b) is why I'd prefer seeing module stuff done first.
Christian.