On 2011-07-22, at 4:41 AM, mikel evins wrote:
Question 2: (I asked this one before, but didn't see an answer.)
Suppose I have a file containing data serialized by calling object->u8vector and then written to a file. Suppose, further, that I happen to know that an object of interest to me starts at a specific byte offset within the file.
What's the right way to read that object from the file?
An additional constraint is that I don't want to have to read the entire contents of the file in order to read the object; I want to be able to read just enough bytes to reconstitute the object whose serialized representation starts at the given offset.
The object encoding used by object->u8vector (and u8vector->object) does not contain a length. The stream of bytes has to be parsed to know how long the object encoding is. To facilitate I/O, I usually prefix the encoding with a 32 bit integer which indicates the length of the encoding.
A second problem is that the object encoding supports shared structures. This means that the object created with
(let ((shared (cons 1 2))) (cons shared (cons 3 shared)))
will be encoded with something like this:
OBJ0=[pair OBJ1=[pair 1 2] OBJ2=[pair 3 OBJ1] ]
or if you want all the details:
(object->u8vector
(let ((shared (cons 1 2))) (cons shared (cons 3 shared)))) #u8(100 100 81 82 100 83 129 0) ^ ^ ^ ^ ^ ^ ^ | | | | | | | | | 1 2 | 3 reference to shared object 0 | | | | | "pair" (index 2) | | | "pair" (index 1) | "pair" (index 0)
So you can't just start parsing an object from an arbitrary point inside the encoding. Some context will be missing.
The approach I would take is to dump each object independently in the file and keep a separate index of where each object starts, and its length (then use input-port-byte-position to seek to the right place). Even better, store the serialized objects in a database and let it do the indexing for you (it also simplifies deleting objects).
Marc