[gambit-list] {Spam?} Re: {Spam?} (Marc + ML:) Extension/patch feedback & inclusion request: Practical custom types in the sexp reader (finally)

Marc Feeley feeley at iro.umontreal.ca
Mon Feb 22 14:57:17 EST 2016


There are two things I don’t like about this extension to the reader.  The implementation unconditionnaly throws away the location information (the “unwrap”) that could be useful if the form being read contains code.  I understand that this is probably what you want when the form is treated as a literal constant, but this is not always the case.

Also, it opens the door to add custom syntax that may clash with future extensions to Gambit's lexical syntax.  While it is good to have hooks to extend Gambit, it would be good if such extensions have some support from the community (in other words, do other people think this is the best way to extend the lexical syntax).

I remember SRFI 10, which is an extension to the lexical syntax with similar goals, and also the JazzScheme syntax for literal constants, i.e. {typename …}.  Perhaps something like that would be better to avoid a proliferation of different syntaxes for the same thing.  Also there are interesting “bootstrapping” issues such as the need to define the type before using the extended lexical syntax.  It would be nice if literal constants could be used in the same file that defines the types (i.e. no phasing problem).

Marc

> On Feb 17, 2016, at 11:48 AM, Adam <adam.mlmb at gmail.com> wrote:
> 
> Dear Marc & list,
> 
> First, the fun stuff suggested is in the "Result, usage" secton below - check it out first :D
> 
> This is to discuss the pull request https://github.com/feeley/gambit/pull/180 that I just made.
> 
> I hope this provides a clean and practical solution to a problem that has been discussed here on the list last years, and to which not really clean solutions have been suggested until now.
> 
> Marc, in particular can you please check out my question secton "Implementation stuff I don't remember why - Marc clarify?" right at the bottom here, and let us know so everyone will know why it needs to be exactly like this? Thanks!
> 
> Purpose
> There are times when as a user you want to enhance your s-expression format with a custom type.
> 
> E.g., sometimes you have a custom value type such as an FFI type, that you want to make serializable.
> 
> Problem
> The writer is fairly easy to extend, you just overlap Gambit's |##wr| procedure with your own (lambda (write-env obj) ...) procedure, where |obj| is the value that Gambit is serializing right now, and that procedure either does its own serialization by outputting the customs structure using (##wr-str write-env string-serialized-form), or passes on execution to the underlying (original) |##wr|.
> 
> So so far so good.
> 
> The reader is more complex to extend though.
> 
> One syntax that theoretically could have worked would have been took into gambit's #. syntax, to inline actual code to be evaluated by the reader, in the serialized form of the sexp. That is not only unsafe though, but also, Gambit namespace management is a task that is difficult to impossible - Gambit is simply not really made for that, so having your custom forms of data serialized to #.(my-data-form ...) does not scale.
> 
> The solution I suggest
> Luckily, internally Gambit has a procedure |##readtable-char-sharp-handler-set!|, which installs user-defined hash-sequence deserializer in a readtable!
> 
> E.g. install an #\X-character handler into it, and any read element #Xyour-data will have your-data readable by that custom handler.
> 
> Implementation problem 1
> The problem then is that Gambit not exports any ways to make your own custom handlers - it just exports its own handlers, such as |##read-sharp-vector| (for "#(..."), |##read-sharp-char| (for "#\..."), |##read-sharp-dot| (for "#...."), |##read-sharp-bang| (for "#!..."), etc. . (All those are implemented in lib/_io.scm , and are installed in every readtable using - you guessed it - |##readtable-char-sharp-handler-set!|, in |##make-standard-readtable|.)
> 
> To implement your own handler, you need access to the lowlevel macros |macro-read-next-char-or-eof|, |macro-readenv-filepos-set!| and |macro-readenv-wrap|.
> 
> Also, perhaps, even if you had access to those macros in userland, because Gambit's IO could change at some point perhaps actually using them in userland would be a hack - and so, it would be better if Gambit exported a generalized mechanism for the user to implement his own readtable handlers, that itself takes care of the lowlevel stuff.
> 
> Implementation problem 1 solution
> To enable users to implement their own readtable handlers, I suggest the inclusion into Gambit of a procedure |##make-readtable-char-sharp-sexp-reader-transformer|, which takes the arguments (transform #!optional (read ##read-datum-or-label)) and returns a readtable hash-sequence deserializer for use with |##readtable-char-sharp-handler-set!|.
> 
> |transform| is the user-provided function that receives as only argument the data that was read out of the sexp in an ordinary format (discussed below), and performs any conversion to that to the custom format (e.g. C FFI type etc.) - the transform procedure evaluates to the conversion result to be used in the sexp.
> 
> |##make-readtable-char-sharp-sexp-reader-transformer| abstracts away the need of taking care of/takes care of all the Gambit-IO-internal macros, as discussed above.
> 
> Implementation problem 2
> The next problem is how the content of that custom hash-sequnce type is read -
> 
> The different ##read-sharp-* procedures all implement their own specific deserializer logics, to read the different hash-sequence formats e.g. "#(e1 e2)", "#\newline", "#!value", "#|| comment ||", etc. - rightly so as the formats are completely custom to the respective hash-sequence.
> 
> The ##read-sharp-* procedures frequently, but not always, implement the whole reading logic locally in themselves.
> 
> There are some instances of reading logics defined globally though -
> 
> 	• |##read-datum-or-label| reads a single datum (e.g. "abc", "123", "#f" or "(1 2 3)"), that one is used by |##read-sharp-ampersand|, for deserialization of the box form e.g. "#&#t", or for the datum-comment form e.g. "#;(im not read)".
> 
> 	• |##read-expr-from-port| reads any expression from a port, for |##eval| to be fed with it. That one is used by |##read-sharp-dot|, e.g. for "#.(+ 1 2)".
> 
> 	• There's a bunch of ##build- forms for reading vectors, lists, integers
> 
> 	• |##read-datum-or-label-or-none-or-dot| seems to read pretty much anything, and is used for instance as continuation for the whole reading process.
> Implementation problem 2 solution
> Of these, it seems to me that |##read-datum-or-label| is a perfect default - in particular, the list type makes perfect sense for storing complex custom types.
> 
> Implementation stuff I don't remember why - Marc clarify?
> As of the moment of writing,
> 	• What's the label-marker check and error handling for?
> 
> 	• Why is the "unwrap":ping during the call to the |read| procedure needed?
> Result, usage
> The result of this addition is that you can plug in your custom serialization code as follows:
> 
> (define-type coord x y)
> 
> (##readtable-char-sharp-handler-set! (input-port-readtable (repl-input-port))
>                                      #\$
>                                      (##make-readtable-char-sharp-sexp-reader-transformer
>                                         (lambda (v)  (apply make-coord v))))
> 
> (set! ##wr (let* ((old-wr ##wr)) (lambda (we obj)
>                                    (if (coord? obj)
>                                        (begin (##wr-str we "#$") (##wr-str we (object->string (list (coord-x obj) (coord-y obj)))))
>                                        (old-wr we obj)))))
> 
> 
> And it's used as follows:
> 
> 
> > (make-coord 15 16)
> #$(15 16)
> > (define c #)
> > (coord-x c)
> 15
> > c
> #$(15 16)
> > (object->string c)
> "#$(15 16)"
> > (call-with-input-string # read)
> #$(15 16)
> >
> 
> The only important catch to be aware of, is that actually typing in #$(15 16) in the REPL will evaluate to the object, which will then cause an exception:
> 
> > #$(15 16)
> *** ERROR IN (console)@19.3 -- Ill-formed expression
> 
> That's analogous to:
> 
> > (eval print)
> *** ERROR -- Ill-formed expression
> #<procedure #16 print>
> 1>
> 
> 
> As of the time of this post, the extention consists only of the following addition to lib/_io.scm :
> 
> (define (##make-readtable-char-sharp-sexp-reader-transformer transform #!optional (read ##read-datum-or-label))
> 
>   (define (read-without-wrap re)
>     (let* ((old-wrapper (macro-readenv-wrapper re))
>            (old-unwrapper (macro-readenv-unwrapper re)))
>       (macro-readenv-wrapper-set! re (lambda (re x) x))
>       (macro-readenv-unwrapper-set! re (lambda (re x) x))
>       (let ((result (read re)))
>         (macro-readenv-wrapper-set! re old-wrapper)
>         (macro-readenv-unwrapper-set! re old-unwrapper)
>         result)))
> 
>   (lambda (re next start-pos)
>     (macro-read-next-char-or-eof re) ;; skip char after #\#
>     (macro-readenv-filepos-set! re start-pos) ;; set pos to start of datum
>     (let ((args (read-without-wrap re)))
>       (if (##label-marker? args)
>           (begin
>             (##raise-datum-parsing-exception 'datum-or-eof-expected re)
>             (##void))
>           (let ((obj (transform args)))
>             (macro-readenv-wrap re obj))))))
> 
> 
> Perhaps this can be simplified further.
> 
> Looking forward to your thoughts and feedback, and I hope to finally enable Gambit with this :)
> 
> In particular if you feel any name would be more relevant than "##make-readtable-char-sharp-sexp-reader-transformer" then let me know.
> 
> Thanks!
> 




More information about the Gambit-list mailing list