On 2011-03-18, at 3:39 PM, mikel evins wrote:
>
> On Mar 18, 2011, at 2:04 PM, Marc Feeley wrote:
>
>> There is much more extensibility in the Gambit reader. Much of it is not documented because the API is not simple. If you tell me what you want to do I'll give you instructions.
>
> Here's a brief summary of Bard's lexical rules.
You'll find the code to extend Gambit's reader at the end of this message. Gambit uses "readtables", which are attached to ports, to specify the lexical syntax and the actions the reader must perform when it encounters given characters. Actions are Scheme procedures with 2 parameters: a "read environment" and a character (the character that triggered the action). A read environment (readenv for short) is a structure containing: the input port, the current "file position", the readtable, and wrap/unwrap functions to attach source code location information to the data being read (this is used by the source code reader when using "load" on source code, i.e. (load "foo.scm")).
A standard readtable can be created by calling the procedure ##make-standard-readtable . A readtable can be copied with the procedure ##readtable-copy . Finally, the procedure ##readtable-char-class-set! modifies the action procedure attached to a given character, and the nature of the character (is it a delimiter or not). For example, a readtable with treats the quote character as an independent token could be implemented like this:
(define (bard-make-readtable)
(let ((rt (##make-standard-readtable)))
(##readtable-char-class-set! rt #\' #t bard-read-quote)
rt))
(define (bard-read-quote re c)
(let ((start-pos (##readenv-current-filepos re)))
(macro-read-next-char-or-eof re) ;; skip '
(macro-readenv-filepos-set! re start-pos) ;; set pos to start of datum
(macro-readenv-wrap re bard-quote)))
(define bard-quote (list 'bard-quote))
Note that some operations are done with macros, which are defined in "~~lib/_gambit#.scm" which must be "##included". The action procedure "bard-read-quote" will receive the readenv in "re" and the character #\' in "c". A call to ##readenv-current-filepos gets the current file position of the quote character (before skipping over it), so that macro-readenv-wrap will attach the correct position to the wrapped object (only when reading source code). Note that the quote token is represented simply by a list containing the symbol bard-quote. You could use a unique type if that is more convenient.
>
> There are four special lexical tokens that correspond to unique values. I'd like to read these tokens directly:
>
> undefined
> nothing
> true
> false
This can be done with an action procedure for reading numbers/keywords/symbols which checks if the symbol read is one of these 4 special tokens.
>
> I'd like to read single-quote (') and comma (,) as my own tokens; Bard handling of these is slightly different from older Lisps.
As explained above these are simple action procedures attached to #\' and #\, .
>
> I need to read characters like this:
>
> \c is the character 'c'
> \space is the character #\space
> \u+0000 is the Unicode null character
An action procedure attached to #\\ .
>
> Gambit's handling of numeric and string literals is already exactly right (except for type annotations, about which more below).
>
> The rules for names (symbols) recognized three cases:
>
> Unqualified names: foo
> Qualified names: bar:foo
> Keywords :foo
Gambit's readtables can already be configured to accept the syntax :foo for keywords (prefix keywords). The implementation of qualified names can be done by checking if the symbol that is read contains a colon. Note that the handling of qualified names is probably best done by your compiler, so if it was me, I would simply let bar:foo pass through as a single token.
>
> Bard uses a module system similar to that of Dylan. Names (symbols) inhabit modules. The textual form of a name indicates the module against which the name is resolved.
>
> A keyword is in the keyword module.
>
> A qualified name is in the module whose name appears before the colon. There is a single global namespace for module names, and a separate (more restrictive) set of rules for the lexical syntax of module names.
>
> An unqualified name is in the lexically-apparent ("current") module.
>
> There are three kinds of structured literals: applications, value sequences, and maps:
>
> Applications look like traditional Lisp lists, e.g. (+ 2 3)
Standard.
>
> Value sequences look like this: [+ 2 3]
> This example is equivalent in older Lisps to this expression: `(,+ ,2 ,3)
>
> Map literals look like this:
> {:name "Fred" :age 101}
> {1 1.0 2 2.0}
Gambit's reader already has support for [ ... ] and { ... } . A keyword is attached to each type of parentheses, and if it is non false, it will be added at the head of the list. In other words, if the keyword attached to [ ... ] is value-sequence, them [1 2 3] will be read as (value-sequence 1 2 3).
>
> Values constructed by the Bard reader are guaranteed to conform to the protocol of the category of value being constructed, but Bard doesn't make any promises about the specific type of the resulting value. For example, historically, a Lisp promises that reading "(foo bar baz)" results in a value consisting of a chain of cons cells. Bard makes no such promise; it promises only that the value obeys the Sequence protocol. The representation is up to the runtime; it might be a finger tree or a vlist or something else.
>
> Similarly, "foo bar" does not specify the representation that will be used for the text string; Bard's only promise is that the resulting object will conform to the Text protocol.
>
> (Bard does provide ways to obtain value of specific types, e.g. type-constraint expressions:
> (as <finger-tree> [0 1 2 3]))
>
> I am considering whether to provide literal syntax for boxes, and would be interested in any opinion you might have, since Gambit has supported them for some time. Bard boxes are a bit different from Gambit's, in that they are transactionally-protected shared mutable containers.
>
> Thanks for any guidance.
>
> --me
Let me know if the code works for you.
Marc
;;---------------------------------------------------------------------
(##include "~~lib/gambit#.scm")
(##include "~~lib/_gambit#.scm")
(##define-macro (macro-peek-next-char-or-eof re) ;; possibly returns EOF
`(macro-peek-char (macro-readenv-port ,re)))
(##define-macro (macro-read-next-char-or-eof re) ;; possibly returns EOF
`(macro-read-char (macro-readenv-port ,re)))
;;---------------------------------------------------------------------
(define (bard-make-readtable)
(let ((rt (##make-standard-readtable)))
(define (set-number/keyword/symbol-starter first last)
(let loop ((i (char->integer first)))
(if (<= i (char->integer last))
(begin
(##readtable-char-class-set!
rt
(integer->char i)
#f
bard-read-number/keyword/symbol)
(loop (+ i 1))))))
(set-number/keyword/symbol-starter #\a #\z)
(set-number/keyword/symbol-starter #\A #\Z)
(set-number/keyword/symbol-starter #\0 #\9)
(set-number/keyword/symbol-starter #\+ #\+)
(set-number/keyword/symbol-starter #\- #\-)
(macro-readtable-keywords-allowed?-set! rt 'prefix)
(##readtable-char-class-set! rt #\' #t bard-read-quote)
(##readtable-char-class-set! rt #\, #t bard-read-comma)
(##readtable-char-class-set! rt #\\ #t bard-read-character)
(macro-readtable-bracket-keyword-set! rt 'value-sequence)
(macro-readtable-brace-keyword-set! rt 'map-literal)
rt))
(define (bard-read-number/keyword/symbol re c)
(let ((start-pos (##readenv-current-filepos re)))
(macro-read-next-char-or-eof re) ;; skip "c"
(macro-readenv-filepos-set! re start-pos) ;; set pos to start of datum
(let ((obj (##build-delimited-number/keyword/symbol re c #t)))
(let ((x (assq obj bard-special-symbols)))
(cond (x
(macro-readenv-wrap re (cdr x)))
((symbol? obj)
(let ((lst
(map string->symbol
(call-with-input-string
(symbol->string obj)
(lambda (port)
(read-all port
(lambda (port) (read-line port #\:))))))))
(if (= 1 (length lst))
(macro-readenv-wrap re (car lst))
(macro-readenv-wrap
re
(map (lambda (x) (macro-readenv-wrap re x))
(cons 'qualified-name lst))))))
(else
(macro-readenv-wrap re obj)))))))
(define bard-special-symbols
(list (cons 'undefined (if #f 123))
(cons 'nothing 'nil)
(cons 'true #t)
(cons 'false #f)))
(define (bard-read-quote re c)
(let ((start-pos (##readenv-current-filepos re)))
(macro-read-next-char-or-eof re) ;; skip "c"
(macro-readenv-filepos-set! re start-pos) ;; set pos to start of datum
(macro-readenv-wrap re bard-quote)))
(define bard-quote (list 'bard-quote))
(define (bard-read-comma re c)
(let ((start-pos (##readenv-current-filepos re)))
(macro-read-next-char-or-eof re) ;; skip "c"
(macro-readenv-filepos-set! re start-pos) ;; set pos to start of datum
(macro-readenv-wrap re bard-comma)))
(define bard-comma (list 'bard-comma))
(define (bard-read-character re c)
(let ((start-pos (##readenv-current-filepos re)))
(macro-read-next-char-or-eof re) ;; skip backslash
(macro-readenv-filepos-set! re start-pos) ;; set pos to start of datum
(let ((c (macro-read-next-char-or-eof re))) ;; read char after backslash
(cond ((not (char? c))
(error "incomplete form, EOF reached"))
((char-alphabetic? c)
(let ((name (##build-delimited-string re c 1)))
(define (not-hex)
(let ((x (assoc name
(macro-readtable-named-char-table
(macro-readenv-readtable re)))))
(if x
(macro-readenv-wrap re (cdr x))
(if (= 1 (string-length name))
(macro-readenv-wrap re (string-ref name 0))
(error "unknown character name" name)))))
(if (and (= 6 (string-length name))
(char=? (string-ref name 0) #\u)
(char=? (string-ref name 1) #\+))
(let ((n (string->number (substring name 2 6) 16)))
(if n
(macro-readenv-wrap re (integer->char n))
(not-hex)))
(not-hex))))
(else
(macro-readenv-wrap re c))))))
;;---------------------------------------------------------------------
;; test it
(define (bard-read-all-from-string str)
(call-with-input-string
(list init: str
readtable: (bard-make-readtable))
read-all))
(pretty-print (bard-read-all-from-string #<<EOF
12345
undefined
nothing
true
false
'
,
\c
\space
\u+1234
\\
\(
\)
foo
bar:foo
:foo
(1 2 3)
[1 2 3]
{1 2 3}
EOF
))
;;---------------------------------------------------------------------
