Qwaq releases Hydra multi-core Squeak VM � The
Marc: There's currently a lot of interest in education and in funding agencies for multi-core programming strategies; here's another implementation. Gambit may be good for this. Brad http://news.squeak.org/2008/02/21/qwaq-releases-hydra-multi-core- squeak-vm/
Brad, On Mon, Feb 25, 2008 at 12:24 AM, Bradley Lucier <lucier@math.purdue.edu> wrote:
There's currently a lot of interest in education and in funding agencies for multi-core programming strategies; here's another implementation. Gambit may be good for this.
The Hydra VM seems to work, if I read the description right, by running a separate Squeak image on each core. This is not the multicore solution we are looking for. You are right, though; programming for multicore CPUs is becoming increasingly important and support for it should be a design goal for Gambit. --Jeff
On Mon, Feb 25, 2008 at 3:04 PM, Jeff Read <bitwize@gmail.com> wrote:
The Hydra VM seems to work, if I read the description right, by running a separate Squeak image on each core.
This is not the multicore solution we are looking for.
What are we looking for? The capability for gambit to mux gambit continuations over multiple OS threads seems like a bad conceptual fit in my mind. But IIRC Erlang got a multithreaded VM's some releases ago, so some have chosen that path. Running a separate gambit on each core and hiding it all under termite glue to make it appear as one image, which i believe is the gambit counterpart to what you are not looking for, is IMHO way neater.
Native threads have been on Gambit's wish-list for a while. http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/Wish_list#Native_.2F_p... As the page states, "Support for multiple processes is good but pales in comparison to light weight, native, pre-emptive thread support." I agree with this. Haskell is able to schedule its threads on OS threads, but it's probably significantly more difficult to do that in Scheme because of mutation. I am unqualified to offer a solution for native threading, but I would love to see them in Gambit if it's technically possible. On Mon, Feb 25, 2008 at 10:35 AM, Joel Borggrén-Franck <joel.borggren.franck@gmail.com> wrote:
On Mon, Feb 25, 2008 at 3:04 PM, Jeff Read <bitwize@gmail.com> wrote:
The Hydra VM seems to work, if I read the description right, by running a separate Squeak image on each core.
This is not the multicore solution we are looking for.
What are we looking for? The capability for gambit to mux gambit continuations over multiple OS threads seems like a bad conceptual fit in my mind. But IIRC Erlang got a multithreaded VM's some releases ago, so some have chosen that path.
Running a separate gambit on each core and hiding it all under termite glue to make it appear as one image, which i believe is the gambit counterpart to what you are not looking for, is IMHO way neater.
_______________________________________________ Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
-- James Long Coptix, Inc. longster@gmail.com
James Long wrote:
Native threads have been on Gambit's wish-list for a while.
http://dynamo.iro.umontreal.ca/~gambit/wiki/index.php/Wish_list#Native_.2F_p...
As the page states, "Support for multiple processes is good but pales in comparison to light weight, native, pre-emptive thread support." I agree with this. Haskell is able to schedule its threads on OS threads, but it's probably significantly more difficult to do that in Scheme because of mutation. I am unqualified to offer a solution for native threading, but I would love to see them in Gambit if it's technically possible.
Why would you love to see them exactly? What do you want to do? BTW both the Perl5.x and Python implementors have tried to make use of pthreads, and I think both have failed at it. Python may have failed more since (if I'm not wrong) their model will scale less, but Perl's model is so slow and memory hungry that I'm never using it. Christian.
On Mon, Feb 25, 2008 at 5:01 PM, Christian Jaeger <christian@pflanze.mine.nu> wrote:
Why would you love to see them exactly? What do you want to do?
Heh, well, one of my side projects is a high performance graphics engine. True parallel processing is becoming extremely important; though I'm still not sure of the best way to achieve it. Using STM can incur high lock contention, but going with the Termite model incurs quite a bit of overhead with serializing/deserializing, copying, and TCP packet construction/destruction. It's possible you could avoid the TCP overhead for local processes using pipes of some sort. I don't know how native threads in Gambit would look, but it's bound to be more efficient than separate processes. I wouldn't have thought Gambit could support native threads, but I was happy to read that on the wish list, and someone must have some idea of how to achieve it. -- James Long Coptix, Inc. longster@gmail.com
On Mon, Feb 25, 2008 at 11:01 PM, Christian Jaeger <christian@pflanze.mine.nu> wrote:
James Long wrote:
BTW both the Perl5.x and Python implementors have tried to make use of pthreads, and I think both have failed at it. Python may have failed more since (if I'm not wrong) their model will scale less, but Perl's model is so slow and memory hungry that I'm never using it.
I wouldn't go as far as claiming that Python has failed, rather that the solutions is far from perfect. The Python interpreter suffers from its Global Interpreter Lock that hinders the smooth operation of large amounts of threads. Gambits model can be different though. In gambit, in theory, you only need a fixed amount of pthreds, maybe 1-2 per core, then you multiplex your gambit processes over the pthreads. I think this is the model the Erlang VM uses.
Joel Borggrén-Franck wrote:
I wouldn't go as far as claiming that Python has failed, rather that the solutions is far from perfect. The Python interpreter suffers from its Global Interpreter Lock that hinders the smooth operation of large amounts of threads.
How are you going to improve it? What will happen with an ever-increasing number of cores?
Gambits model can be different though. In gambit, in theory, you only need a fixed amount of pthreds, maybe 1-2 per core, then you multiplex your gambit processes over the pthreads.
This sounds like the classical N:M threading idea. It's not clear this is an advantage over just only using native threads, as the NPTL people have shown. Letting *one* scheduler (the one in the OS kernel) handle everything was more efficient. ("Machine"/"C style" threads require separate continguous stacks, though, and this always pays the overhead of in average 1/2 block of non-used data per thread. Gambit's kernel does better than that. You'd have to give up that advantage when using only those native threads. But could you keep the advantage even with a N:M model?) OS threads have the disadvantage of not knowing about your language. And I'm not sure how OS threads (aka pthreads) will solve the problem of the ever-increasing number of cpus. To prevent communication/synchronization overhead between cpus becoming a bottleneck quickly, some hierarchical interconnection mechanism is necessary. It's not that just magically in the future every pthread can communicate with any other running on another cpu as efficiently as it can do in your dual core machine now, I think. I suppose, the more cores you have, the more will communication between cpu's start looking like socket communication between processes. After ignoring all the terminology, a multicore machine is basically nothing else than a more closely connected cluster. Thus afaict, the multicore architectures, once reaching a big number of cores, always ever only have a constant advantage over clusters. Only creating a hierarchy is making more than a constant advantage (multicore for fast local communication, clustering for the next layer), so I suppose some time in the future one machine will contain *several* layers of calculation engines, some more closely tied to each other than others, to continue this pyramide. I don't know very much about clustering, and even less about using clusters of smp machines, but I guess if you want to make best use of both layers (cluster and smp, in the mentioned pyramide-like way), this is manual work.
I think this is the model the Erlang VM uses.
Ignorant question: does Erlang have a shared heap? Any shared data at all? Except in Mnesia maybe? Christian.
James Long wrote:
On Mon, Feb 25, 2008 at 5:01 PM, Christian Jaeger <christian@pflanze.mine.nu> wrote:
Why would you love to see them exactly? What do you want to do?
Heh, well, one of my side projects is a high performance graphics engine. True parallel processing is becoming extremely important; though I'm still not sure of the best way to achieve it. Using STM can incur high lock contention, but going with the Termite model incurs quite a bit of overhead with serializing/deserializing, copying, and TCP packet construction/destruction. It's possible you could avoid the TCP overhead for local processes using pipes of some sort.
I don't know how native threads in Gambit would look, but it's bound to be more efficient than separate processes.
Could you do your work with homogenous vectors as the only shared data structure (including locks)? (And apart of that, Termite style communication.) If so, I could help implement something when I get the time (in a month or two). Christian.
On Mon, Feb 25, 2008 at 10:39 PM, James Long <longster@gmail.com> wrote:
Heh, well, one of my side projects is a high performance graphics engine. True parallel processing is becoming extremely important; though I'm still not sure of the best way to achieve it. Using STM can incur high lock contention, but going with the Termite model incurs quite a bit of overhead with serializing/deserializing, copying, and TCP packet construction/destruction. It's possible you could avoid the TCP overhead for local processes using pipes of some sort.
My library Gamsock was supposed to provide more extensive socket support for Gambit, including Unix domain sockets, UDP, and IPv6. Gamsock development is for the time being discontinued because without proper support from and integration with Gambit's I/O system, things are getting really ugly really fast. --Jeff
On 26-Feb-08, at 6:37 AM, Jeff Read wrote:
On Mon, Feb 25, 2008 at 10:39 PM, James Long <longster@gmail.com> wrote:
Heh, well, one of my side projects is a high performance graphics engine. True parallel processing is becoming extremely important; though I'm still not sure of the best way to achieve it. Using STM can incur high lock contention, but going with the Termite model incurs quite a bit of overhead with serializing/deserializing, copying, and TCP packet construction/destruction. It's possible you could avoid the TCP overhead for local processes using pipes of some sort.
My library Gamsock was supposed to provide more extensive socket support for Gambit, including Unix domain sockets, UDP, and IPv6.
Gamsock development is for the time being discontinued because without proper support from and integration with Gambit's I/O system, things are getting really ugly really fast.
It is possible to interface to Gambit's low-level I/O system, but it is not documented and non-obvious. I will put together a brief documentation so that you can experiment with this. Marc
Christian Jaeger wrote:
Joel Borggrén-Franck wrote:
I wouldn't go as far as claiming that Python has failed, rather that the solutions is far from perfect. The Python interpreter suffers from its Global Interpreter Lock that hinders the smooth operation of large amounts of threads. How are you going to improve it? What will happen with an ever-increasing number of cores?
My understanding (as a long time Python user) is that Python isn't interested in the billion-threads "solution" to multi-processing, (Guido seems to think, and I agree with him, that multi-threaded programs are too hard/impossible to get correct) and would rather see a different "fix" for the problem, whatever that may be. Shared-nothing/Multiple-processes seems like a contender. Later, Blake.
On Tue, Feb 26, 2008 at 12:30 PM, Christian Jaeger <christian@pflanze.mine.nu> wrote:
Joel Borggrén-Franck wrote:
I wouldn't go as far as claiming that Python has failed, rather that the solutions is far from perfect. The Python interpreter suffers from its Global Interpreter Lock that hinders the smooth operation of large amounts of threads.
How are you going to improve it? What will happen with an ever-increasing number of cores?
You could for example use a more fine-grained locks. This will allow you to use more threads, but won't change the time it takes to spawn them, so it solves part of the problem. Now I'm not going to argue that using one os-threads per language-thread is a good thing, but it can be made to work for large number of cores (presumably with a lot of work), there are instances of java running on 700+ cores using thousands of threads in a single image. This is not how I envision the future of concurrent programming, but it is working, right now, hence I won't name it 'failure'.
Gambits model can be different though. In gambit, in theory, you only need a fixed amount of pthreds, maybe 1-2 per core, then you multiplex your gambit processes over the pthreads.
This sounds like the classical N:M threading idea. It's not clear this is an advantage over just only using native threads, as the NPTL people have shown. Letting *one* scheduler (the one in the OS kernel) handle everything was more efficient.
From what I can see, the gambit model promises fast thread spawning, and this is something we want to keep. Now this can not be done with os-threads, so you simply can't use a 1:1 model, because no OS today delivers the spawn-times we want. Also NPTL is supposed to be a general purpose thread-library, I imagine you can cut some corners when you know more of your target system.
Also there is the question of relative efficiency and programmer/machine efficiency. I do believe there can be a place for a N:M model, at least in the short run.
OS threads have the disadvantage of not knowing about your language. And I'm not sure how OS threads (aka pthreads) will solve the problem of the ever-increasing number of cpus.
And green threads usually have the disadvantage of not beeing schedulable over different processors. This leaves us with the only option of using multiple gambit processes scheduled over the cores (presumably glued together by termite), OR use an N:M scheme OR using both of them. Also in the foreseeable future we will only have a bounded number of cores.
I think this is the model the Erlang VM uses.
Ignorant question: does Erlang have a shared heap? Any shared data at all? Except in Mnesia maybe?
Though far from an Erlang expert, i believe it's supposed to not share anything. The native thread implementation uses one erlang scheduler per thread, and in the initial release only one os-thread could have outstanding I/O at anytime (if I got my facts straight).
Blake Winton wrote:
My understanding (as a long time Python user) is that Python isn't interested in the billion-threads "solution" to multi-processing, (Guido seems to think, and I agree with him, that multi-threaded programs are too hard/impossible to get correct) and would rather see a different "fix" for the problem, whatever that may be. Shared-nothing/Multiple-processes seems like a contender.
Starting up multiple Python processes is *very* heavyweight. Communicating between those processes is also heavyweight. The Python "solution" of starting multiple processes and talking through sockets is, quite simply, garbage. The issue isn't "threads" or processes per se. Threads or processes could be fine, it's just that they're too heavy on an OS. Look at all the optimization and work people put in to make Apache scale on an OS. The real issue is that no current language except Erlang supports message passing as an easy primitive between "threads". A solution to concurrency will be the issue that dethrones the current languages. I know that I'm looking. -a
Andrew Lentvorski wrote:
Blake Winton wrote:
My understanding (as a long time Python user) is that Python isn't interested in the billion-threads "solution" to multi-processing, (Guido seems to think, and I agree with him, that multi-threaded programs are too hard/impossible to get correct) and would rather see a different "fix" for the problem, whatever that may be. Shared-nothing/Multiple-processes seems like a contender.
Starting up multiple Python processes is *very* heavyweight. Communicating between those processes is also heavyweight.
The Python "solution" of starting multiple processes and talking through sockets is, quite simply, garbage.
Let's take a step back here. I'm not Python. Saying that something may be a contender is a far cry from calling it a "solution". I never said that the processes would communicate through sockets. (Unix pipes or shared memory would be far better for things on the same box!) Neither of us has any idea what the startup time of Python will be in the future, nor how heavyweight inter-process communication will be. You seem to have an axe to grind, and while I appreciate that none of the current languages do concurrency particularly well, I think you've missed the mark in this particular criticism.
The issue isn't "threads" or processes per se. Threads or processes could be fine, it's just that they're too heavy on an OS. Look at all the optimization and work people put in to make Apache scale on an OS.
Which OS is that? ;) You have to believe that, as processors gain more and more cores, OSes will react by making process creation and communication far lighter-weight, just like JVM implementors made synchronizing on objects lighter and lighter-weight as people used it more. And if you're going to look at web servers, why not look at some which use Shared-nothing/Multiple-processes, like Yaws (found at http://yaws.hyber.org/ )? They seem to have beaten the pants off of Apache, according to http://www.sics.se/~joe/apachevsyaws.html
The real issue is that no current language except Erlang supports message passing as an easy primitive between "threads".
So you do like Erlang's shared-nothing/multiple-processes approach?
A solution to concurrency will be the issue that dethrones the current languages. I know that I'm looking.
I'm sure the entire programming language community will be interested in anything you find. Seriously. It seems to be considered one of the major problems of our age on every language-related mailing list I'm on, and everyone is looking for a halfway-decent answer. (Not even _the_ answer, just something that's better than what we have now.) Later, Blake.
Blake Winton wrote:
Andrew Lentvorski wrote:
Blake Winton wrote:
My understanding (as a long time Python user) is that Python isn't interested in the billion-threads "solution" to multi-processing, (Guido seems to think, and I agree with him, that multi-threaded programs are too hard/impossible to get correct) and would rather see a different "fix" for the problem, whatever that may be. Shared-nothing/Multiple-processes seems like a contender.
Starting up multiple Python processes is *very* heavyweight. Communicating between those processes is also heavyweight.
The Python "solution" of starting multiple processes and talking through sockets is, quite simply, garbage.
Let's take a step back here. I'm not Python. Saying that something may be a contender is a far cry from calling it a "solution". I never said that the processes would communicate through sockets. (Unix pipes or shared memory would be far better for things on the same box!)
Sorry, I was under the impression that you had witnessed one of these arguments firsthand from the Python community. I have seen the Python community propose that as a solution multiple times. Unix pipes or shared memory are not obviously better on the same box anymore. Both of those solutions require significant intervention from the OS kernel and are usually fairly unoptimized(read: painfully slow and limited). By contrast, network stacks are *heavily* optimized an often have zero-copy modes in which pointers to memory are passed around with very little overhead.
Neither of us has any idea what the startup time of Python will be in the future, nor how heavyweight inter-process communication will be.
That's disingenuous. Of course we do. It's some factor within what it is now barring miracles. Aka: painfully slow.
You seem to have an axe to grind, and while I appreciate that none of the current languages do concurrency particularly well, I think you've missed the mark in this particular criticism.
I absolutely have an axe to grind. I want a good language that handles concurrency well. Nothing particularly fits right now.
The issue isn't "threads" or processes per se. Threads or processes could be fine, it's just that they're too heavy on an OS. Look at all the optimization and work people put in to make Apache scale on an OS.
Which OS is that? ;) You have to believe that, as processors gain more and more cores, OSes will react by making process creation and communication far lighter-weight, just like JVM implementors made synchronizing on objects lighter and lighter-weight as people used it more.
Why do I have to believe that? None of the modern OS's seem to be heading that direction. User responsiveness is a far more compelling application (think uninterrupted audio and video) than concurrency communication. OS's have plenty of background tasks to soak up those cores so they don't really care.
And if you're going to look at web servers, why not look at some which use Shared-nothing/Multiple-processes, like Yaws (found at http://yaws.hyber.org/ )? They seem to have beaten the pants off of Apache, according to http://www.sics.se/~joe/apachevsyaws.html
I have, but these systems have their own quirks. In particular, everything is in the Erlang world. The moment you have to leave it (think mod_XXXX), your performance drops to the same problems as Apache.
The real issue is that no current language except Erlang supports message passing as an easy primitive between "threads".
So you do like Erlang's shared-nothing/multiple-processes approach?
Yes. But I'd like to see other languages (thing like Python, Ruby, etc.) at least *try*--and maybe even try other solutions. Python has simply dismissed the problem.
A solution to concurrency will be the issue that dethrones the current languages. I know that I'm looking.
I'm sure the entire programming language community will be interested in anything you find. Seriously. It seems to be considered one of the major problems of our age on every language-related mailing list I'm on, and everyone is looking for a halfway-decent answer. (Not even _the_ answer, just something that's better than what we have now.)
Right now, Erlang is it. It's okay, but I didn't really like it for general programming. It's kind of a strange combination of static and dynamic (records are static). It works *great* for network stack-like programs, though. It reduced about 10,000 lines of Java newtork stack code to about 1000 lines of Erlang. Big, *BIG* win. I just wish there was another Erlang implementation. I've looked at the VM code for Erlang, and it's painful. Reimplementing from scratch is probably going to be easier. -a
Andrew Lentvorski wrote:
Blake Winton wrote:
The Python "solution" of starting multiple processes and talking through sockets is, quite simply, garbage. Saying that something may be a contender is a far cry from calling it a "solution". Sorry, I was under the impression that you had witnessed one of these arguments firsthand from the Python community. I have seen the Python community propose that as a solution multiple times.
Urls? I can believe that they suggest it as preferable to multi-threading (working and slow is always better than fast and broken ;), or even the best solution for now, but I'm sure we both agree that it's hardly the best solution possible.
Unix pipes or shared memory would be far better for things on the same box!) Unix pipes or shared memory are not obviously better on the same box anymore. [snip...]
Cool... It's good to hear that network-y things are getting faster.
Neither of us has any idea what the startup time of Python will be in the future, nor how heavyweight inter-process communication will be. That's disingenuous. Of course we do. It's some factor within what it is now barring miracles. Aka: painfully slow.
Perhaps, but if the future ends up being multi-process and sockets, it would hardly surprise me to see heavy work get done on optimizing Python's startup, or native-code compilation, or whatever else will relieve that particular pain.
You seem to have an axe to grind, and while I appreciate that none of the current languages do concurrency particularly well, I think you've missed the mark in this particular criticism. I absolutely have an axe to grind. I want a good language that handles concurrency well. Nothing particularly fits right now.
We completely agree on that. I'ld love to see something that works better than what we've got, and if it could manage to be a library on top of an existing language, so much the better.
You have to believe that, as processors gain more and more cores, OSes will react by making process creation and communication far lighter-weight Why do I have to believe that? None of the modern OS's seem to be heading that direction.
Yet. But the multi-core era is just starting. Also, didn't BeOS go that direction? I thought they concentrated on allowing you to create billions of threads, and had some impressive demos as a result.
User responsiveness is a far more compelling application (think uninterrupted audio and video) than concurrency communication. OS's have plenty of background tasks to soak up those cores so they don't really care.
How many background tasks do they have? My WinXP box seems to have 32-ish. My OSX box has 72. They're both fewer than the 64 and 128 core processors I expect to see in my lifetime. At some point they're going to run out, and then what?
And if you're going to look at web servers, why not look at some which use Shared-nothing/Multiple-processes, like Yaws (found at http://yaws.hyber.org/ )? They seem to have beaten the pants off of Apache, according to http://www.sics.se/~joe/apachevsyaws.html I have, but these systems have their own quirks. In particular, everything is in the Erlang world. The moment you have to leave it (think mod_XXXX), your performance drops to the same problems as Apache.
I'm sure I've heard of a library that talks to Erlang nodes, so that you could write your programs in whatever you want, and hook them in to Erlang... My point was more that the work the Apache people are doing to make Apache scale is beside the point, since a better concurrency paradigm is beating the pants off of them in terms of scalability.
The real issue is that no current language except Erlang supports message passing as an easy primitive between "threads". So you do like Erlang's shared-nothing/multiple-processes approach? Yes. But I'd like to see other languages (thing like Python, Ruby, etc.) at least *try*--and maybe even try other solutions.
What (other than STM) are the other solutions in this space?
Python has simply dismissed the problem.
I don't see that. My reading of it is that the Python community has dismissed the "solution" of shared-state multi-threading. (And if you're not sharing state, are they really threads anymore?) But if someone comes up with another solution, I'm sure the Python community would be more than happy to implement it.
A solution to concurrency will be the issue that dethrones the current languages. I know that I'm looking. I'm sure the entire programming language community will be interested in anything you find. Right now, Erlang is it. It's okay, but I didn't really like it for general programming.
Hmm. I haven't used it for general programming, but that doesn't surprise me terribly. It looked neat, but not necessarily practical.
It's kind of a strange combination of static and dynamic (records are static).
It's my understanding that that's where it gets its main wins. The data is static, so you don't need to copy it all over the place, or worry about thread contention, but the code is dynamic so you can do something... Uh, that would be the point where I would start waving my hands. ;)
I just wish there was another Erlang implementation. I've looked at the VM code for Erlang, and it's painful. Reimplementing from scratch is probably going to be easier.
If only there was an implementation of the Erlang ideas in, say, Scheme. ;) Later, Blake.
Blake Winton wrote:
Andrew Lentvorski wrote:
Blake Winton wrote:
The Python "solution" of starting multiple processes and talking through sockets is, quite simply, garbage. Saying that something may be a contender is a far cry from calling it a "solution". Sorry, I was under the impression that you had witnessed one of these arguments firsthand from the Python community. I have seen the Python community propose that as a solution multiple times.
Urls? I can believe that they suggest it as preferable to multi-threading (working and slow is always better than fast and broken ;), or even the best solution for now, but I'm sure we both agree that it's hardly the best solution possible.
Well that's Unix, it has always been about flat files and unstructured streams and leaving it open to the users making something out of those. Fork is remarkedly efficient for starting a new process, btw.
My point was more that the work the Apache people are doing to make Apache scale is beside the point, since a better concurrency paradigm is beating the pants off of them in terms of scalability.
I've thought a bit about the M:N thing. Namely, that you could implement e.g. a webserver (relatively) easily by forking N (where N is the number of cores or so) processes from a Gambit parent process, and having each of them handle M green threads. And I've always thought of that being good and efficient. So why would I point out that the NPTL people found out that purely-native threads were better for performance than an M:N model? Because we are talking different things here: The M:N model for web servers does not need to do any (or almost no) communication between the threads (neither between the OS threads/processes nor the green threads) for the http serving part. So M:N scheduling will be trivial, there is no difference between that and just M processes doing *anything* on the same machine. I think the point where NPTL will be faster is where a thread X needs to talk to thread Y, and without using NPTL (or similar), X and Y could happen to be green threads in different OS threads/processes. So the information has first to be passed on (or locks resolved) to the right OS thread/process, and then inside that process the user-space scheduler has to pass it on (schedule again) to the right green thread. If you're not doing much communication between your threads, that is not an issue. OTOH, saving on memory with green threads will really be an advantage (and using multiple processes at the same time will give each core something to churn on), which is why I agree that the green threads are a good thing to have. Now before anyone protests hinting that the new generation of web applications are highly interactive and users are all interchanging data with each other live upon every ajaxy request and thus claiming that http server threads don't need no intercommunication is utter nonsense, I'd like to point out that for such data sharing you'd likely also want transactional integrity (with the nice benefit that using a transactional model can fit well with multiprocessing for efficiency), (mostly-)transparent persistent storage (most data is not to be forgotten, and be it only so that when a user hits reload he can still see it), distribution (since one machine won't cut your next generation web app needs, even with 2**n cores), replication (since having multiple machines will involve some of them failing), and possibly data merging after a split brain situation (since while your two data centers lost their interconnect you didn't want to stop both of them serving your clients). Yeah that sounds like a database. Of course since you're a functional programmer already, creating new data from old data without mutation is pretty deep in your brain, you're creating new sets of data by referencing the new things together with the unchanged parts, and then you pass on the new changeset over to the other threads. Oh that's the whole database, that's impossibly costly to serialize, well, we need threads! Pthreads. The memory must be shared. Ah, but there are multiple machines involved, so pthreads won't cut it, and we want that stuff stored on disk, a heap of data on disk after all. Hm you didn't mutate old data (which, since you want it to be shared with other threads, may still be in use by other transactions anyway, so actually nobody is even *allowed* to mutate it), well one doesn't need locking of any object that isn't going to be mutated anyway, and so it comes you don't happen to need any communication of such mutations. (It's only the changing tip of the whole set which needs to be communicated.) That's the miracle database. (Oh, that name's already taken by a company servicing certain database needs, maybe it's better to call it something else.) Christian.
participants (10)
-
Andrew Lentvorski -
Blake Winton -
Bob McIsaac -
Bradley Lucier -
Christian Jaeger -
Christian Jaeger -
James Long -
Jeff Read -
Joel Borggrén-Franck -
Marc Feeley