Nouvel objet : {Spam?} Re: {Spam?} (Marc + ML:) Extension/patch feedback & inclusion request: Practical custom types in the sexp reader (finally)

17 Fév 2016


      Dear Marc & list,
First, the fun stuff suggested is in the *"Result, usage"* secton below -
check it out first :D
This is to discuss the pull request
https://github.com/feeley/gambit/pull/180 that I just made.
I hope this provides a clean and practical solution to a problem that has
been discussed here on the list last years, and to which not really clean
solutions have been suggested until now.
Marc, in particular can you please check out my question secton
"Implementation stuff I don't remember why - Marc clarify?" right at the
bottom here, and let us know so everyone will know why it needs to be
exactly like this? Thanks!
*Purpose*
There are times when as a user you want to enhance your s-expression format
with a custom type.
E.g., sometimes you have a custom value type such as an FFI type, that you
want to make serializable.
*Problem*
The writer is fairly easy to extend, you just overlap Gambit's |##wr|
procedure with your own (lambda (write-env obj) ...) procedure, where |obj|
is the value that Gambit is serializing right now, and that procedure
either does its own serialization by outputting the customs structure using
(##wr-str write-env string-serialized-form), or passes on execution to the
underlying (original) |##wr|.
So so far so good.
The reader is more complex to extend though.
One syntax that theoretically could have worked would have been took into
gambit's #. syntax, to inline actual code to be evaluated by the reader, in
the serialized form of the sexp. That is not only unsafe though, but also,
Gambit namespace management is a task that is difficult to impossible -
Gambit is simply not really made for that, so having your custom forms of
data serialized to #.(my-data-form ...) does not scale.
*The solution I suggest*
Luckily, internally Gambit has a procedure
|##readtable-char-sharp-handler-set!|, which installs user-defined
hash-sequence deserializer in a readtable!
E.g. install an #\X-character handler into it, and any read element
#Xyour-data will have your-data readable by that custom handler.
*Implementation problem 1*
The problem then is that Gambit not exports any ways to make your own
custom handlers - it just exports its own handlers, such as
|##read-sharp-vector| (for "#(..."), |##read-sharp-char| (for "#..."),
|##read-sharp-dot| (for "#...."), |##read-sharp-bang| (for "#!..."), etc. .
(All those are implemented in lib/_io.scm , and are installed in every
readtable using - you guessed it - |##readtable-char-sharp-handler-set!|,
in |##make-standard-readtable|.)
To implement your own handler, you need access to the lowlevel macros
|macro-read-next-char-or-eof|, |macro-readenv-filepos-set!| and
|macro-readenv-wrap|.
Also, perhaps, even if you had access to those macros in userland, because
Gambit's IO could change at some point perhaps actually using them in
userland would be a hack - and so, it would be better if Gambit exported a
generalized mechanism for the user to implement his own readtable handlers,
that itself takes care of the lowlevel stuff.
*Implementation problem 1 solution*
To enable users to implement their own readtable handlers, I suggest the
inclusion into Gambit of a procedure
|##make-readtable-char-sharp-sexp-reader-transformer|, which takes the
arguments (transform #!optional (read ##read-datum-or-label)) and returns a
readtable hash-sequence deserializer for use with
|##readtable-char-sharp-handler-set!|.
|transform| is the user-provided function that receives as only argument
the data that was read out of the sexp in an ordinary format (discussed
below), and performs any conversion to that to the custom format (e.g. C
FFI type etc.) - the transform procedure evaluates to the conversion result
to be used in the sexp.
|##make-readtable-char-sharp-sexp-reader-transformer| abstracts away the
need of taking care of/takes care of all the Gambit-IO-internal macros, as
discussed above.
*Implementation problem 2*
The next problem is how the content of that custom hash-sequnce type is
read -
The different ##read-sharp-* procedures all implement their own specific
deserializer logics, to read the different hash-sequence formats e.g. "#(e1
e2)", "#\newline", "#!value", "#|| comment ||", etc. - rightly so as the
formats are completely custom to the respective hash-sequence.
The ##read-sharp-* procedures frequently, but not always, implement the
whole reading logic locally in themselves.
There are some instances of reading logics defined globally though -
- |##read-datum-or-label| reads a single datum (e.g. "abc", "123", "#f"
   or "(1 2 3)"), that one is used by |##read-sharp-ampersand|, for
   deserialization of the box form e.g. "#&#t", or for the datum-comment form
   e.g. "#;(im not read)".
- |##read-expr-from-port| reads any expression from a port, for |##eval|
   to be fed with it. That one is used by |##read-sharp-dot|, e.g. for "#.(+ 1
   2)".
- There's a bunch of ##build- forms for reading vectors, lists, integers
- |##read-datum-or-label-or-none-or-dot| seems to read pretty much
   anything, and is used for instance as continuation for the whole reading
   process.
*Implementation problem 2 solution*
Of these, it seems to me that |##read-datum-or-label| is a perfect default
- in particular, the list type makes perfect sense for storing complex
custom types.
*Implementation stuff I don't remember why - Marc clarify?*
As of the moment of writing,
- What's the label-marker check and error handling for?
- Why is the "unwrap":ping during the call to the |read| procedure
   needed?
*Result, usage*
The result of this addition is that you can plug in your custom
serialization code as follows:
(define-type coord x y)
(##readtable-char-sharp-handler-set! (input-port-readtable
(repl-input-port))
                                     #$
(##make-readtable-char-sharp-sexp-reader-transformer
                                        (lambda (v)  (apply make-coord v))))
(set! ##wr (let* ((old-wr ##wr)) (lambda (we obj)
                                   (if (coord? obj)
                                       (begin (##wr-str we "#$") (##wr-str
we (object->string (list (coord-x obj) (coord-y obj)))))
                                       (old-wr we obj)))))
And it's used as follows:
...
*(make-coord 15 16)*
#$(15 16)
...
*(define c #)*
*(coord-x c)*
15
...
*c*
#$(15 16)
...
*(object->string c)*
"#$(15 16)"
...
*(call-with-input-string # read)*
#$(15 16)
...
The only important catch to be aware of, is that actually typing in #$(15
16) in the REPL will evaluate to the object, which will then cause an
exception:
...
*#$(15 16)*
*** ERROR IN (console)@19.3 -- Ill-formed expression
That's analogous to:
...
*(eval print)*
*** ERROR -- Ill-formed expression
#<procedure #16 print>
1>
As of the time of this post, the extention consists only of the following
addition to lib/_io.scm :
(define (##make-readtable-char-sharp-sexp-reader-transformer transform
#!optional (read ##read-datum-or-label))
(define (read-without-wrap re)
    (let* ((old-wrapper (macro-readenv-wrapper re))
           (old-unwrapper (macro-readenv-unwrapper re)))
      (macro-readenv-wrapper-set! re (lambda (re x) x))
      (macro-readenv-unwrapper-set! re (lambda (re x) x))
      (let ((result (read re)))
        (macro-readenv-wrapper-set! re old-wrapper)
        (macro-readenv-unwrapper-set! re old-unwrapper)
        result)))
(lambda (re next start-pos)
    (macro-read-next-char-or-eof re) ;; skip char after ##
    (macro-readenv-filepos-set! re start-pos) ;; set pos to start of datum
    (let ((args (read-without-wrap re)))
      (if (##label-marker? args)
          (begin
            (##raise-datum-parsing-exception 'datum-or-eof-expected re)
            (##void))
          (let ((obj (transform args)))
            (macro-readenv-wrap re obj))))))
Perhaps this can be simplified further.
Looking forward to your thoughts and feedback, and I hope to finally enable
Gambit with this :)
In particular if you feel any name would be more relevant than
"##make-readtable-char-sharp-sexp-reader-transformer" then let me know.
Thanks!