[gambit-list] Character encoding and the repl

Atticus atticus0 at posteo.org
Sat May 30 02:43:06 EDT 2015


> By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters).  This allows Scheme strings that are printed to be processed by external tools which only support ASCII.  For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).

Very interesting, thank you.

> Should the external representation of strings depend on the character encoding of the output port?  In other words, if the character encoding of the byte output port is
>
> - ASCII then characters whose code are >= 128 would use escapes like \374
> - ISO-8859-1 then characters whose code are >= 256 would use escapes
> - UCS-2 then characters whose code are >= 65536 would use escapes
> - in all other cases escapes would not be used because all Unicode characters can be encoded

Imho yes. But to be clear, let's say the external representation of
scheme strings depends now on the character encoding of the output port
and I have the above explained situation, I have an utf-8 port and need
to print to ASCII only, then I can still do that by changing the output
port to ASCII manually before printing?

Marc Feeley <feeley at iro.umontreal.ca> writes:

> By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters).  This allows Scheme strings that are printed to be processed by external tools which only support ASCII.  For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
>
> Note that this behavior was chosen a long time ago when UTF-8 was not widespread.  I’m open to changing this behavior, but I’d like to know what other people think.
>
> Should the external representation of strings depend on the character encoding of the output port?  In other words, if the character encoding of the byte output port is
>
> - ASCII then characters whose code are >= 128 would use escapes like \374
> - ISO-8859-1 then characters whose code are >= 256 would use escapes
> - UCS-2 then characters whose code are >= 65536 would use escapes
> - in all other cases escapes would not be used because all Unicode characters can be encoded
>
> By the way this would also affect the external representation of symbols, such as 'über .
>
> Marc
>
>
>> On May 29, 2015, at 2:12 PM, Atticus <atticus0 at posteo.org> wrote:
>> 
>> Hello,
>> 
>> First and formost thanks for developing and maintaining gambit.
>> 
>> As a beginner I'm a litte confused about the handling of the character
>> encoding in the repl. 
>> 
>> $ gsi -:d,t8,f8
>> $> "ü"
>> "\374"
>> 
>> $> (string-ref "ü" 0)
>> #\xfc
>> 
>> $> (string->list "über")
>> (#\xfc #\b #\e #\r)
>> 
>> $> (with-output-to-file "test2" (lambda () (##write-string "ü")
>> $> ,q
>> 
>> $ cat test2
>> ü
>> 
>> $ gsi -:d,t8,f8
>> $> (with-input-from-file "test2" (lambda () (read-char)))
>> #\xfc
>> 
>> 
>> With racket:
>> 
>> $ racket
>> $>"ü"
>> "ü"
>> 
>> $> (with-input-from-file "test2" (lambda () (read-char)))
>> #\ü
>> 
>> $> (string->list "über")
>> '(#\ü #\b #\e #\r)
>> 
>> 
>> Why does gambit return "\374" instead of "ü" in the repl when evaluating
>> the string "ü"? I'm curious because in racket the repl returns the
>> evaluated string as "ü"", also read-char in racket returns #\ü instead
>> of #\xfc.
>> 
>> Looking at the documentation I could change the behaviour with:
>> ;;;;
>> (output-port-readtable-set!
>>   (repl-output-port)
>>   (readtable-max-unescaped-char-set
>>     (output-port-readtable (repl-output-port))
>>     #\U0010ffff))
>> ;;;;
>> 
>> Are there any advantages for having this default readtable setting in
>> gambit? As a beginner I think it's confusing.
>> 
>> _______________________________________________
>> Gambit-list mailing list
>> Gambit-list at iro.umontreal.ca
>> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list



More information about the Gambit-list mailing list