[gambit-list] Character encoding and the repl

Marc Feeley feeley at iro.umontreal.ca
Fri May 29 15:08:11 EDT 2015


By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters).  This allows Scheme strings that are printed to be processed by external tools which only support ASCII.  For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).

Note that this behavior was chosen a long time ago when UTF-8 was not widespread.  I’m open to changing this behavior, but I’d like to know what other people think.

Should the external representation of strings depend on the character encoding of the output port?  In other words, if the character encoding of the byte output port is

- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded

By the way this would also affect the external representation of symbols, such as 'über .

Marc


> On May 29, 2015, at 2:12 PM, Atticus <atticus0 at posteo.org> wrote:
> 
> Hello,
> 
> First and formost thanks for developing and maintaining gambit.
> 
> As a beginner I'm a litte confused about the handling of the character
> encoding in the repl. 
> 
> $ gsi -:d,t8,f8
> $> "ü"
> "\374"
> 
> $> (string-ref "ü" 0)
> #\xfc
> 
> $> (string->list "über")
> (#\xfc #\b #\e #\r)
> 
> $> (with-output-to-file "test2" (lambda () (##write-string "ü")
> $> ,q
> 
> $ cat test2
> ü
> 
> $ gsi -:d,t8,f8
> $> (with-input-from-file "test2" (lambda () (read-char)))
> #\xfc
> 
> 
> With racket:
> 
> $ racket
> $>"ü"
> "ü"
> 
> $> (with-input-from-file "test2" (lambda () (read-char)))
> #\ü
> 
> $> (string->list "über")
> '(#\ü #\b #\e #\r)
> 
> 
> Why does gambit return "\374" instead of "ü" in the repl when evaluating
> the string "ü"? I'm curious because in racket the repl returns the
> evaluated string as "ü"", also read-char in racket returns #\ü instead
> of #\xfc.
> 
> Looking at the documentation I could change the behaviour with:
> ;;;;
> (output-port-readtable-set!
>   (repl-output-port)
>   (readtable-max-unescaped-char-set
>     (output-port-readtable (repl-output-port))
>     #\U0010ffff))
> ;;;;
> 
> Are there any advantages for having this default readtable setting in
> gambit? As a beginner I think it's confusing.
> 
> _______________________________________________
> Gambit-list mailing list
> Gambit-list at iro.umontreal.ca
> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4839 bytes
Desc: not available
URL: <http://mailman.iro.umontreal.ca/pipermail/gambit-list/attachments/20150529/501eb390/attachment.bin>


More information about the Gambit-list mailing list