[gambit-list] A couple of random questions

Marc Feeley feeley at iro.umontreal.ca
Mon Jul 25 07:23:10 EDT 2011


On 2011-07-22, at 4:41 AM, mikel evins wrote:

> Question 2:
> (I asked this one before, but didn't see an answer.)
> 
> Suppose I have a file containing data serialized by calling object->u8vector and then written to a file. Suppose, further, that I happen to know that an object of interest to me starts at a specific byte offset within the file.
> 
> What's the right way to read that object from the file?
> 
> An additional constraint is that I don't want to have to read the entire contents of the file in order to read the object; I want to be able to read just enough bytes to reconstitute the object whose serialized representation starts at the given offset.

The object encoding used by object->u8vector (and u8vector->object) does not contain a length.  The stream of bytes has to be parsed to know how long the object encoding is.  To facilitate I/O, I usually prefix the encoding with a 32 bit integer which indicates the length of the encoding.

A second problem is that the object encoding supports shared structures.  This means that the object created with

    (let ((shared (cons 1 2))) (cons shared (cons 3 shared)))

will be encoded with something like this:

    OBJ0=[pair
          OBJ1=[pair 1 2]
          OBJ2=[pair 3 OBJ1]
         ]

or if you want all the details:

  > (object->u8vector
     (let ((shared (cons 1 2))) (cons shared (cons 3 shared))))
  #u8(100 100 81 82 100 83 129 0)
      ^   ^   ^  ^  ^   ^  ^
      |   |   |  |  |   |  |
      |   |   1  2  |   3  reference to shared object 0
      |   |         |
      |   |         "pair" (index 2)
      |   |
      |   "pair" (index 1)
      |
      "pair" (index 0)

So you can't just start parsing an object from an arbitrary point inside the encoding.  Some context will be missing.

The approach I would take is to dump each object independently in the file and keep a separate index of where each object starts, and its length (then use input-port-byte-position to seek to the right place).  Even better, store the serialized objects in a database and let it do the indexing for you (it also simplifies deleting objects).

Marc




More information about the Gambit-list mailing list