Blake Winton wrote:
Andrew Lentvorski wrote:
Blake Winton wrote:
The Python "solution" of starting multiple processes and talking through sockets is, quite simply, garbage.
Saying that something may be a contender is a far cry from calling it a "solution".
Sorry, I was under the impression that you had witnessed one of these arguments firsthand from the Python community. I have seen the Python community propose that as a solution multiple times.
Urls? I can believe that they suggest it as preferable to multi-threading (working and slow is always better than fast and broken ;), or even the best solution for now, but I'm sure we both agree that it's hardly the best solution possible.
Well that's Unix, it has always been about flat files and unstructured streams and leaving it open to the users making something out of those. Fork is remarkedly efficient for starting a new process, btw.
My point was more that the work the Apache people are doing to make Apache scale is beside the point, since a better concurrency paradigm is beating the pants off of them in terms of scalability.
I've thought a bit about the M:N thing. Namely, that you could implement e.g. a webserver (relatively) easily by forking N (where N is the number of cores or so) processes from a Gambit parent process, and having each of them handle M green threads. And I've always thought of that being good and efficient. So why would I point out that the NPTL people found out that purely-native threads were better for performance than an M:N model? Because we are talking different things here:
The M:N model for web servers does not need to do any (or almost no) communication between the threads (neither between the OS threads/processes nor the green threads) for the http serving part. So M:N scheduling will be trivial, there is no difference between that and just M processes doing *anything* on the same machine. I think the point where NPTL will be faster is where a thread X needs to talk to thread Y, and without using NPTL (or similar), X and Y could happen to be green threads in different OS threads/processes. So the information has first to be passed on (or locks resolved) to the right OS thread/process, and then inside that process the user-space scheduler has to pass it on (schedule again) to the right green thread. If you're not doing much communication between your threads, that is not an issue. OTOH, saving on memory with green threads will really be an advantage (and using multiple processes at the same time will give each core something to churn on), which is why I agree that the green threads are a good thing to have.
Now before anyone protests hinting that the new generation of web applications are highly interactive and users are all interchanging data with each other live upon every ajaxy request and thus claiming that http server threads don't need no intercommunication is utter nonsense, I'd like to point out that for such data sharing you'd likely also want transactional integrity (with the nice benefit that using a transactional model can fit well with multiprocessing for efficiency), (mostly-)transparent persistent storage (most data is not to be forgotten, and be it only so that when a user hits reload he can still see it), distribution (since one machine won't cut your next generation web app needs, even with 2**n cores), replication (since having multiple machines will involve some of them failing), and possibly data merging after a split brain situation (since while your two data centers lost their interconnect you didn't want to stop both of them serving your clients).
Yeah that sounds like a database.
Of course since you're a functional programmer already, creating new data from old data without mutation is pretty deep in your brain, you're creating new sets of data by referencing the new things together with the unchanged parts, and then you pass on the new changeset over to the other threads. Oh that's the whole database, that's impossibly costly to serialize, well, we need threads! Pthreads. The memory must be shared. Ah, but there are multiple machines involved, so pthreads won't cut it, and we want that stuff stored on disk, a heap of data on disk after all. Hm you didn't mutate old data (which, since you want it to be shared with other threads, may still be in use by other transactions anyway, so actually nobody is even *allowed* to mutate it), well one doesn't need locking of any object that isn't going to be mutated anyway, and so it comes you don't happen to need any communication of such mutations. (It's only the changing tip of the whole set which needs to be communicated.)
That's the miracle database. (Oh, that name's already taken by a company servicing certain database needs, maybe it's better to call it something else.)
Christian.