[gambit-list] Character encoding and the repl

Adam adam.mlmb at gmail.com
Thu Jun 18 12:19:40 EDT 2015


Atticus,

I see the printout and corrections you offer here but just for
clarity, can you provide a definition of the problem you're seeing?


2015-06-16 15:23 GMT+05:30, Atticus <atticus0 at posteo.org>:
> I encountered another problem.
>
> 'shell-command' in gambit release 4.7.6 version and 6b353f does not
> recognize the character encoding when used with the optional argument:
>
> $ file testfile
> testfile: UTF-8 Unicode text
>
> $ cat testfile
> Verhüllung
>
> $ grep hüll testfile
> Verhüllung
>
> Gambit 6b353f:
>
> $ gsi -:tU
> (shell-command "grep hüll testfile" #t)
> -> (256 . "")
> ;; expected (0 . "Verhüllung\n")
>
> (shell-command "grep Ver testfile" #t)
> -> (0 . "Verhüllung\n")
> ;; expected (0 . "Verhüllung\n")
>
> (shell-command "grep Ver testfile" #f)
> -> Verhüllung
> -> 0
> ;; ok
>
> $ gsi -:tA
> (shell-command "grep hüll testfile" #t)
> -> (256 . "")
>
> Now with readtable-max-unescaped-char-set set to #\U0010ffff:
>
> $ gsi -:tA
> (shell-command "grep hüll testfile" #t)
> (0 . "Verhüllung\n")
>
> (shell-command "grep Ver testfile" #t)
> (0 . "Verhüllung\n")
>
> $ gsi -:tU
> (shell-command "grep hüll testfile" #t)
> -> (256 . "")
> (shell-command "grep Ver testfile" #t)
> -> (0 . "Verhüllung\n")
>
>
> Marc Feeley <feeley at iro.umontreal.ca> writes:
>
>> Ha ha!  That’s a funny issue caused by an optimization of the pretty
>> printer.  Now fixed.
>>
>> Marc
>>
>>> On Jun 5, 2015, at 3:57 PM, Atticus <atticus0 at posteo.org> wrote:
>>>
>>> Some observations about the escaping of characters with the 8d2c2e6
>>> commit:
>>>
>>> $ gsi -:tU
>>>
>>> ;;;;;;;;
>>>
>>> (string->list "über")
>>> ;; -> (#\xfc #\b #\e #\r)
>>> ;; expected (#\ü #\b #\e #\r)
>>>
>>> (list->string (string->list "über"))
>>> ;; -> "über"
>>> ;; ok
>>>
>>> (map list->string (list (string->list "über")))
>>> ;; -> ("\374ber")
>>> ;; expected ("über")
>>>
>>> ;;;;;;;;
>>>
>>> I expected (string->list "über") to return (#\ü #\b #\e #\r) and that
>>> list->string does not behave different when used with map.
>>>
>>> When setting readtable-max-unescaped-char-set in the gambcini file
>>> string->list and list->string work as expected.
>>>
>>> Marc Feeley <feeley at iro.umontreal.ca> writes:
>>>
>>>> I have pushed a change which allows #f as the setting of the
>>>> max-unescaped-char of a readtable, and it is now the default instead of
>>>> (integer->char 127).  When max-unescaped-char is #f the printer will
>>>> take into account the character encoding of the output port and use
>>>> escapes when the character can’t be encoded.  I have also changed the
>>>> default character encoding of the terminal, files, etc to use ASCII.
>>>> This setting is actually equivalent to ISO-8859-1 (latin-1) at the
>>>> lowest level of the runtime system, i.e. chars are encoded using 8 bits
>>>> and the top bit is significant, but the printer will use escapes when
>>>> the character code is above 127 (because ASCII is a 7 bit code).
>>>>
>>>> This means that when no runtime options are passed to gsi the system
>>>> will escape characters exactly as before.  However, with
>>>>
>>>>   gsi -:tU    (or gsi -:t8)
>>>>
>>>> the system will not escape characters whose code is > 127.
>>>>
>>>> For the users prefering this behavior, including you I assume, it can be
>>>> made the default by setting the GAMBCOPT environment variable like this
>>>> in your .profile:
>>>>
>>>>  GAMBCOPT=tU
>>>>
>>>> This is better than changing the readtable in the gambcini file because
>>>> it will apply to all Scheme programs compiled with Gambit, not just the
>>>> interpreter, and it can be overriden with an explicit -:tA when starting
>>>> gsi if you want to escape non-ASCII characters in a specific situation.
>>>>
>>>> Marc
>>>>
>>>>
>>>>> On May 30, 2015, at 2:43 AM, Atticus <atticus0 at posteo.org> wrote:
>>>>>
>>>>>
>>>>>> By default the “write”, “pretty-print”, etc procedures (including the
>>>>>> REPL’s printer) assume that the output port can only display ASCII, so
>>>>>> it encodes non-ASCII characters as sequences such as \374 or #xfc
>>>>>> (which are composed of ASCII characters).  This allows Scheme strings
>>>>>> that are printed to be processed by external tools which only support
>>>>>> ASCII.  For example, if you pretty-print to a file a function that
>>>>>> contains the string "ü" you will end up with a file containing the
>>>>>> sequence of 6 characters "\374" which any editor can edit. In other
>>>>>> words, it makes minimal requirements on the features of external tools
>>>>>> (not just editors, but email clients if you want to email some Scheme
>>>>>> code, shell utilities, C compilers, other Scheme systems, etc).
>>>>>
>>>>> Very interesting, thank you.
>>>>>
>>>>>> Should the external representation of strings depend on the character
>>>>>> encoding of the output port?  In other words, if the character
>>>>>> encoding of the byte output port is
>>>>>>
>>>>>> - ASCII then characters whose code are >= 128 would use escapes like
>>>>>> \374
>>>>>> - ISO-8859-1 then characters whose code are >= 256 would use escapes
>>>>>> - UCS-2 then characters whose code are >= 65536 would use escapes
>>>>>> - in all other cases escapes would not be used because all Unicode
>>>>>> characters can be encoded
>>>>>
>>>>> Imho yes. But to be clear, let's say the external representation of
>>>>> scheme strings depends now on the character encoding of the output port
>>>>> and I have the above explained situation, I have an utf-8 port and need
>>>>> to print to ASCII only, then I can still do that by changing the output
>>>>> port to ASCII manually before printing?
>>>>>
>>>>> Marc Feeley <feeley at iro.umontreal.ca> writes:
>>>>>
>>>>>> By default the “write”, “pretty-print”, etc procedures (including the
>>>>>> REPL’s printer) assume that the output port can only display ASCII, so
>>>>>> it encodes non-ASCII characters as sequences such as \374 or #xfc
>>>>>> (which are composed of ASCII characters).  This allows Scheme strings
>>>>>> that are printed to be processed by external tools which only support
>>>>>> ASCII.  For example, if you pretty-print to a file a function that
>>>>>> contains the string "ü" you will end up with a file containing the
>>>>>> sequence of 6 characters "\374" which any editor can edit. In other
>>>>>> words, it makes minimal requirements on the features of external tools
>>>>>> (not just editors, but email clients if you want to email some Scheme
>>>>>> code, shell utilities, C compilers, other Scheme systems, etc).
>>>>>>
>>>>>> Note that this behavior was chosen a long time ago when UTF-8 was not
>>>>>> widespread.  I’m open to changing this behavior, but I’d like to know
>>>>>> what other people think.
>>>>>>
>>>>>> Should the external representation of strings depend on the character
>>>>>> encoding of the output port?  In other words, if the character
>>>>>> encoding of the byte output port is
>>>>>>
>>>>>> - ASCII then characters whose code are >= 128 would use escapes like
>>>>>> \374
>>>>>> - ISO-8859-1 then characters whose code are >= 256 would use escapes
>>>>>> - UCS-2 then characters whose code are >= 65536 would use escapes
>>>>>> - in all other cases escapes would not be used because all Unicode
>>>>>> characters can be encoded
>>>>>>
>>>>>> By the way this would also affect the external representation of
>>>>>> symbols, such as 'über .
>>>>>>
>>>>>> Marc
>>>>>>
>>>>>>
>>>>>>> On May 29, 2015, at 2:12 PM, Atticus <atticus0 at posteo.org> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> First and formost thanks for developing and maintaining gambit.
>>>>>>>
>>>>>>> As a beginner I'm a litte confused about the handling of the
>>>>>>> character
>>>>>>> encoding in the repl.
>>>>>>>
>>>>>>> $ gsi -:d,t8,f8
>>>>>>> $> "ü"
>>>>>>> "\374"
>>>>>>>
>>>>>>> $> (string-ref "ü" 0)
>>>>>>> #\xfc
>>>>>>>
>>>>>>> $> (string->list "über")
>>>>>>> (#\xfc #\b #\e #\r)
>>>>>>>
>>>>>>> $> (with-output-to-file "test2" (lambda () (##write-string "ü")
>>>>>>> $> ,q
>>>>>>>
>>>>>>> $ cat test2
>>>>>>> ü
>>>>>>>
>>>>>>> $ gsi -:d,t8,f8
>>>>>>> $> (with-input-from-file "test2" (lambda () (read-char)))
>>>>>>> #\xfc
>>>>>>>
>>>>>>>
>>>>>>> With racket:
>>>>>>>
>>>>>>> $ racket
>>>>>>> $>"ü"
>>>>>>> "ü"
>>>>>>>
>>>>>>> $> (with-input-from-file "test2" (lambda () (read-char)))
>>>>>>> #\ü
>>>>>>>
>>>>>>> $> (string->list "über")
>>>>>>> '(#\ü #\b #\e #\r)
>>>>>>>
>>>>>>>
>>>>>>> Why does gambit return "\374" instead of "ü" in the repl when
>>>>>>> evaluating
>>>>>>> the string "ü"? I'm curious because in racket the repl returns the
>>>>>>> evaluated string as "ü"", also read-char in racket returns #\ü
>>>>>>> instead
>>>>>>> of #\xfc.
>>>>>>>
>>>>>>> Looking at the documentation I could change the behaviour with:
>>>>>>> ;;;;
>>>>>>> (output-port-readtable-set!
>>>>>>> (repl-output-port)
>>>>>>> (readtable-max-unescaped-char-set
>>>>>>>   (output-port-readtable (repl-output-port))
>>>>>>>   #\U0010ffff))
>>>>>>> ;;;;
>>>>>>>
>>>>>>> Are there any advantages for having this default readtable setting in
>>>>>>> gambit? As a beginner I think it's confusing.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gambit-list mailing list
>>>>>>> Gambit-list at iro.umontreal.ca
>>>>>>> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
> _______________________________________________
> Gambit-list mailing list
> Gambit-list at iro.umontreal.ca
> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
>



More information about the Gambit-list mailing list