[gambit-list] 0mq with Gambit

Marc Feeley feeley at iro.umontreal.ca
Fri Apr 8 09:04:01 EDT 2011


On 2011-04-06, at 12:17 PM, Jason E. Aten wrote:

> As an aside, the last missing piece would be using Google's Protocol Buffers (PB) for very efficient serialization.  I say this in reference to the comments in the Termite paper that the new serialization in Gambit was the only bottleneck that kept Termite from beating Erlang outright in all categories.

There are many reasons why serialization in Gambit/Termite might be slower than in Erlang:

1) The serialization code is written in Scheme and I haven't really tried to optimize it (in Erlang it is carefully hand-written C code).

2) There are multiple levels of buffering.  Gambit's serialization is done by the function object->u8vector which builds a u8vector which encodes the object (this requires some form of buffering and memory allocation because the length of the buffer required is not known in advance).  Then the u8vector is written to a port which buffers the bytes in the I/O byte buffer before they are sent off to the network.

3) object->u8vector has to deal with things Erlang doesn't deal with such as cycles, and continuations.

4) object->u8vector has a hook function to customize the encoding algorithm.  The function is called on each subobject that is serialized. This feature is used by Termite to serialize threads (by giving them a thread ID and using the ID in the serialized representation).  This approach could be used for serializing things like ports, foreign pointers, etc which would not be otherwise serializable.

5) The length of the serialized representation must be known so that it can be sent first, before the serialized representation (to simplify deserialization by the receiving end).

I'm sure it is possible to improve the performance of the serialization by writing directly to a fixed size C allocated buffer sitting just before the socket (and drained to the socket when it is full).  I wonder if Protocol Buffers improve on this simple idea.

Marc




More information about the Gambit-list mailing list