[gambit-list] Character encoding and the repl

Atticus atticus0 at posteo.org
Fri Jun 5 09:57:01 EDT 2015


Some observations about the escaping of characters with the 8d2c2e6
commit:

$ gsi -:tU

;;;;;;;;

(string->list "über")
;; -> (#\xfc #\b #\e #\r)
;; expected (#\ü #\b #\e #\r)

(list->string (string->list "über"))
;; -> "über"
;; ok

(map list->string (list (string->list "über")))
;; -> ("\374ber")
;; expected ("über")

;;;;;;;;

I expected (string->list "über") to return (#\ü #\b #\e #\r) and that
list->string does not behave different when used with map.

When setting readtable-max-unescaped-char-set in the gambcini file
string->list and list->string work as expected.

Marc Feeley <feeley at iro.umontreal.ca> writes:

> I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127).  When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded.  I have also changed the default character encoding of the terminal, files, etc to use ASCII.  This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
>
> This means that when no runtime options are passed to gsi the system will escape characters exactly as before.  However, with
>
>    gsi -:tU    (or gsi -:t8)
>
> the system will not escape characters whose code is > 127.
>
> For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
>
>   GAMBCOPT=tU
>
> This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
>
> Marc
>
>
>> On May 30, 2015, at 2:43 AM, Atticus <atticus0 at posteo.org> wrote:
>> 
>> 
>>> By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters).  This allows Scheme strings that are printed to be processed by external tools which only support ASCII.  For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
>> 
>> Very interesting, thank you.
>> 
>>> Should the external representation of strings depend on the character encoding of the output port?  In other words, if the character encoding of the byte output port is
>>> 
>>> - ASCII then characters whose code are >= 128 would use escapes like \374
>>> - ISO-8859-1 then characters whose code are >= 256 would use escapes
>>> - UCS-2 then characters whose code are >= 65536 would use escapes
>>> - in all other cases escapes would not be used because all Unicode characters can be encoded
>> 
>> Imho yes. But to be clear, let's say the external representation of
>> scheme strings depends now on the character encoding of the output port
>> and I have the above explained situation, I have an utf-8 port and need
>> to print to ASCII only, then I can still do that by changing the output
>> port to ASCII manually before printing?
>> 
>> Marc Feeley <feeley at iro.umontreal.ca> writes:
>> 
>>> By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters).  This allows Scheme strings that are printed to be processed by external tools which only support ASCII.  For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
>>> 
>>> Note that this behavior was chosen a long time ago when UTF-8 was not widespread.  I’m open to changing this behavior, but I’d like to know what other people think.
>>> 
>>> Should the external representation of strings depend on the character encoding of the output port?  In other words, if the character encoding of the byte output port is
>>> 
>>> - ASCII then characters whose code are >= 128 would use escapes like \374
>>> - ISO-8859-1 then characters whose code are >= 256 would use escapes
>>> - UCS-2 then characters whose code are >= 65536 would use escapes
>>> - in all other cases escapes would not be used because all Unicode characters can be encoded
>>> 
>>> By the way this would also affect the external representation of symbols, such as 'über .
>>> 
>>> Marc
>>> 
>>> 
>>>> On May 29, 2015, at 2:12 PM, Atticus <atticus0 at posteo.org> wrote:
>>>> 
>>>> Hello,
>>>> 
>>>> First and formost thanks for developing and maintaining gambit.
>>>> 
>>>> As a beginner I'm a litte confused about the handling of the character
>>>> encoding in the repl. 
>>>> 
>>>> $ gsi -:d,t8,f8
>>>> $> "ü"
>>>> "\374"
>>>> 
>>>> $> (string-ref "ü" 0)
>>>> #\xfc
>>>> 
>>>> $> (string->list "über")
>>>> (#\xfc #\b #\e #\r)
>>>> 
>>>> $> (with-output-to-file "test2" (lambda () (##write-string "ü")
>>>> $> ,q
>>>> 
>>>> $ cat test2
>>>> ü
>>>> 
>>>> $ gsi -:d,t8,f8
>>>> $> (with-input-from-file "test2" (lambda () (read-char)))
>>>> #\xfc
>>>> 
>>>> 
>>>> With racket:
>>>> 
>>>> $ racket
>>>> $>"ü"
>>>> "ü"
>>>> 
>>>> $> (with-input-from-file "test2" (lambda () (read-char)))
>>>> #\ü
>>>> 
>>>> $> (string->list "über")
>>>> '(#\ü #\b #\e #\r)
>>>> 
>>>> 
>>>> Why does gambit return "\374" instead of "ü" in the repl when evaluating
>>>> the string "ü"? I'm curious because in racket the repl returns the
>>>> evaluated string as "ü"", also read-char in racket returns #\ü instead
>>>> of #\xfc.
>>>> 
>>>> Looking at the documentation I could change the behaviour with:
>>>> ;;;;
>>>> (output-port-readtable-set!
>>>>  (repl-output-port)
>>>>  (readtable-max-unescaped-char-set
>>>>    (output-port-readtable (repl-output-port))
>>>>    #\U0010ffff))
>>>> ;;;;
>>>> 
>>>> Are there any advantages for having this default readtable setting in
>>>> gambit? As a beginner I think it's confusing.
>>>> 
>>>> _______________________________________________
>>>> Gambit-list mailing list
>>>> Gambit-list at iro.umontreal.ca
>>>> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list



More information about the Gambit-list mailing list