[gambit-list] Qwaq releases Hydra multi-core Squeak VM « The

Andrew Lentvorski bsder at allcaps.org
Thu Feb 28 14:47:46 EST 2008


Blake Winton wrote:
> Andrew Lentvorski wrote:
>> Blake Winton wrote:
>>> My understanding (as a long time Python user) is that Python isn't 
>>> interested in the billion-threads "solution" to multi-processing, 
>>> (Guido seems to think, and I agree with him, that multi-threaded 
>>> programs are too hard/impossible to get correct) and would rather see 
>>> a different "fix" for the problem, whatever that may be.  
>>> Shared-nothing/Multiple-processes seems like a contender.
>>
>> Starting up multiple Python processes is *very* heavyweight.
>> Communicating between those processes is also heavyweight.
>>
>> The Python "solution" of starting multiple processes and talking 
>> through sockets is, quite simply, garbage.
> 
> Let's take a step back here.  I'm not Python.  Saying that something may 
> be a contender is a far cry from calling it a "solution".  I never said 
> that the processes would communicate through sockets.  (Unix pipes or 
> shared memory would be far better for things on the same box!)

Sorry, I was under the impression that you had witnessed one of these 
arguments firsthand from the Python community.  I have seen the Python 
community propose that as a solution multiple times.

Unix pipes or shared memory are not obviously better on the same box 
anymore.  Both of those solutions require significant intervention from 
the OS kernel and are usually fairly unoptimized(read: painfully slow 
and limited).  By contrast, network stacks are *heavily* optimized an 
often have zero-copy modes in which pointers to memory are passed around 
with very little overhead.

> Neither of us has any idea what the startup time of Python will be in the 
> future, nor how heavyweight inter-process communication will be.

That's disingenuous.  Of course we do.  It's some factor within what it 
is now barring miracles.  Aka: painfully slow.

> You seem to have an axe to grind, and while I appreciate that none of 
> the current languages do concurrency particularly well, I think you've 
> missed the mark in this particular criticism.

I absolutely have an axe to grind.  I want a good language that handles 
concurrency well.  Nothing particularly fits right now.

>> The issue isn't "threads" or processes per se.  Threads or processes 
>> could be fine, it's just that they're too heavy on an OS.  Look at all 
>> the optimization and work people put in to make Apache scale on an OS.
> 
> Which OS is that?  ;)  You have to believe that, as processors gain more 
> and more cores, OSes will react by making process creation and 
> communication far lighter-weight, just like JVM implementors made 
> synchronizing on objects lighter and lighter-weight as people used it more.

Why do I have to believe that?  None of the modern OS's seem to be 
heading that direction.  User responsiveness is a far more compelling 
application (think uninterrupted audio and video) than concurrency 
communication.  OS's have plenty of background tasks to soak up those 
cores so they don't really care.

> And if you're going to look at web servers, why not look at some which 
> use Shared-nothing/Multiple-processes, like Yaws (found at 
> http://yaws.hyber.org/ )?  They seem to have beaten the pants off of 
> Apache, according to http://www.sics.se/~joe/apachevsyaws.html

I have, but these systems have their own quirks.  In particular, 
everything is in the Erlang world.  The moment you have to leave it 
(think mod_XXXX), your performance drops to the same problems as Apache.

>> The real issue is that no current language except Erlang supports 
>> message passing as an easy primitive between "threads".
> 
> So you do like Erlang's shared-nothing/multiple-processes approach?

Yes.  But I'd like to see other languages (thing like Python, Ruby, 
etc.) at least *try*--and maybe even try other solutions.  Python has 
simply dismissed the problem.

>> A solution to concurrency will be the issue that dethrones the current 
>> languages.  I know that I'm looking.
> 
> I'm sure the entire programming language community will be interested in 
> anything you find.  Seriously.  It seems to be considered one of the 
> major problems of our age on every language-related mailing list I'm on, 
> and everyone is looking for a halfway-decent answer.  (Not even _the_ 
> answer, just something that's better than what we have now.)

Right now, Erlang is it.  It's okay, but I didn't really like it for 
general programming.  It's kind of a strange combination of static and 
dynamic (records are static).  It works *great* for network stack-like 
programs, though.  It reduced about 10,000 lines of Java newtork stack 
code to about 1000 lines of Erlang.  Big, *BIG* win.

I just wish there was another Erlang implementation.  I've looked at the 
VM code for Erlang, and it's painful.  Reimplementing from scratch is 
probably going to be easier.

-a



More information about the Gambit-list mailing list