[gambit-list] 0mq with Gambit

Jason E. Aten j.e.aten at gmail.com
Thu Apr 7 18:39:15 EDT 2011


On Thu, Apr 7, 2011 at 4:50 PM, Feng Hou <houfen at gmail.com> wrote:

>
>
> On Wed, Apr 6, 2011 at 12:17 PM, Jason E. Aten <j.e.aten at gmail.com> wrote:
>
>> On Wed, Apr 6, 2011 at 8:13 AM, Marc Feeley <feeley at iro.umontreal.ca>wrote:
>>
>>>
>>> On 2011-04-03, at 11:04 PM, Feng Hou wrote:
>>>
>>> > Hello everyone,
>>> >
>>> > Gambit has great concurrency support by green-thread and no-blocking
>>> I/O scheduling. However, they are only able to use one native thread on
>>> multicore hardware. I have noticed some wish list on wiki for
>>> multicore/multiprocessing concurrency support. I'm wondering whether 0MQ (
>>> http://www.zeromq.org/) can be used as substance for Gambit to build
>>> such capabilities.
>>> >
>>> > My thoughts are,
>>> >
>>> > -- Not just a FFI binding.
>>> > -- Use it as message passing broker for multicore and distributed
>>> concurrency.
>>> > -- Share nothing between the GVM thread and other in-process native
>>> threads (doing long CPU bound computation or blocking I/O).
>>> > -- How to integrate 0MQ inter-thread transport to gambit green-thread
>>> scheduler without blocking it?
>>> > -- Would it be possible to bind 0MQ inter-thread socket to gambit
>>> mailbox?
>>> > -- How to integrate 0MQ I/O event poller with gambit I/O loop?
>>> > -- Would it be better to expose 0MQ IPC/TCP/PGM socket types as Gambit
>>> Port objects?
>>> >
>>> > I realized they were way beyond my knowledge and skill level to
>>> implement (some may not even make sense, please correct me). Nevertheless,
>>> just want to see if others have similar thoughts or needs.
>>> >
>>> > Cheers,
>>> > Feng Hou
>>>
>>> I quickly read the 0MQ docs and it seems interesting.  I can give
>>> pointers to whoever wants to implement them into Gambit.
>>>
>>> Marc
>>
>>
>> I'd be up for contributing on this.  ZMQ and Gambit together would be
>> quite sweet on multicore and as the basis for very scalable distributed
>> systems.
>>
>> As an aside, the last missing piece would be using Google's Protocol
>> Buffers (PB) for very efficient serialization.  I say this in reference to
>> the comments in the Termite paper that the new serialization in Gambit was
>> the only bottleneck that kept Termite from beating Erlang outright in all
>> categories.  By the way, has the serialization situation improved? If so
>> then perhaps PB is not necessary.  But PB would also give one really nice
>> inter-language interoperability, since PB bindings are available from just
>> about any language (sadly except Scheme; but perhaps the Common Lisp
>> bindings could be ported without much trouble) at this point.
>>
>> But back to ZMQ.   It would seem to be more general, as Feng suggests, to
>> be able to use ZMQ from Gambit rather than just Termite.  I say this because
>> if one could use ZMQ from Gambit, then there would be no need play elaborate
>> games to get mutable state, (or variables that actually vary =) ; i.e.
>> without needing to do the gen_server trick/(section 4.6 of the Termite paper
>> workaround) when you actually need mutable variables. Unfortunately, I work
>> with big enough data, that I really do need mutable variables.
>>
>> So assuming that one wants to be able to use ZMQ in Gambit (which I
>> would!), I'd be very glad to hear any advice on how to do that.
>>
>> Jason
>>
>
> I'm glad to hear so much interest and help offered. I've been playing
> around a toy binding, and reading 0mq source. I'd like to share a few more
> concrete ideas.
>
> 0mq socket is a concurrent device, rather than an I/O device. It's very
> lightweight, essentially a concurrent in-memory queue. Hundreds of thousands
> to the same tcp end-point would only consume just one tcp connection. This
> seems a perfect fit to gambit green-thread model. Pulling socket send/recv
> in noblock mode loop is equivalent to spinning on memory CAS+fance of a
> pointer (this appears to be how zmq implements it). 0mq also provides a
> timer facility, but I believe it'd block gambit thread scheduler. Instead,
> gambit thread-yield!/thread-sleep! can be used for scheduling noblock
> send/recv pulling.  We can have thousands of green-threads pulling on
> thousands of 0mq sockets concurrently. This can be done fairly
> straightforward in user land. Ideally, it'd be nice to just write,
>
> (write u8vector a-zmq-port)
>
> (read u8vector a-zmq-port timeout: 10)
>
> All looping/retry/yield/sleep can happen in gambit scheduler (need
> continuation magic?).  Even nicer would be able to bind sockets to thread
> mailboxes so messages are pumped into mailboxes on receiving automatically.
> So we can use Erlang/Termite style receive loop, which gambit already
> adopted for local thread message passing (Though I'm not too keen on sending
> messages to remote threads, i.e. erlang Pid model. IMO, It's too tightly
> coupled in distributed  environment, where 0mq subject-oriented message
> passing is better architecturally).
>
Hi Feng,

I'm glad to hear about your experiments and ideas.  Could elaborate on the
last comment that I've quoted above, re "not too keen on sending messages to
remote threads."   I think I must be misunderstanding something, because the
whole point of ZMQ is to send messages to remote threads.  I'm also not
clear what you mean by the terms subject-oriented message vs. too tightly
coupled. Feel free to elaborate on these, if you can.

Thank you.

Best regards,

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.iro.umontreal.ca/pipermail/gambit-list/attachments/20110407/06b2fc13/attachment.htm>


More information about the Gambit-list mailing list