[gambit-list] Character encoding and the repl

Atticus atticus0 at posteo.org
Thu Jun 18 13:24:56 EDT 2015


The problem is when you use 'shell-command' with grep on a file that
contains non-ASCII characters (the testfile contains just one line with
the word "Verhüllung"), for example searching for "hüll", grep can not
find the line or word. I think the examples make this more clear:

With gsi -:tU

(shell-command "grep hüll testfile" #t)

returns (256 . "") but with the runtime option -:tU I expected (0 . "Verhüllung") and

(shell-command "grep Ver testfile" #t)

returns (0 . "Verhüllung\n") but I expected the return value (0 . "Verhüllung").

As I said in the previous post that with
readtable-max-unescaped-char-set set to #\U0010ffff, 'shell-command'
shows the expected behaviour but not with the -:tU runtime
option. Shouldn't be both options equivalent?


Adam <adam.mlmb at gmail.com> writes:

> Atticus,
>
> I see the printout and corrections you offer here but just for
> clarity, can you provide a definition of the problem you're seeing?
>
>
> 2015-06-16 15:23 GMT+05:30, Atticus <atticus0 at posteo.org>:
>> I encountered another problem.
>>
>> 'shell-command' in gambit release 4.7.6 version and 6b353f does not
>> recognize the character encoding when used with the optional argument:
>>
>> $ file testfile
>> testfile: UTF-8 Unicode text
>>
>> $ cat testfile
>> Verhüllung
>>
>> $ grep hüll testfile
>> Verhüllung
>>
>> Gambit 6b353f:
>>
>> $ gsi -:tU
>> (shell-command "grep hüll testfile" #t)
>> -> (256 . "")
>> ;; expected (0 . "Verhüllung\n")
>>
>> (shell-command "grep Ver testfile" #t)
>> -> (0 . "Verhüllung\n")
>> ;; expected (0 . "Verhüllung\n")
>>
>> (shell-command "grep Ver testfile" #f)
>> -> Verhüllung
>> -> 0
>> ;; ok
>>
>> $ gsi -:tA
>> (shell-command "grep hüll testfile" #t)
>> -> (256 . "")
>>
>> Now with readtable-max-unescaped-char-set set to #\U0010ffff:
>>
>> $ gsi -:tA
>> (shell-command "grep hüll testfile" #t)
>> (0 . "Verhüllung\n")
>>
>> (shell-command "grep Ver testfile" #t)
>> (0 . "Verhüllung\n")
>>
>> $ gsi -:tU
>> (shell-command "grep hüll testfile" #t)
>> -> (256 . "")
>> (shell-command "grep Ver testfile" #t)
>> -> (0 . "Verhüllung\n")
>>
>>
>> Marc Feeley <feeley at iro.umontreal.ca> writes:
>>
>>> Ha ha!  That’s a funny issue caused by an optimization of the pretty
>>> printer.  Now fixed.
>>>
>>> Marc
>>>
>>>> On Jun 5, 2015, at 3:57 PM, Atticus <atticus0 at posteo.org> wrote:
>>>>
>>>> Some observations about the escaping of characters with the 8d2c2e6
>>>> commit:
>>>>
>>>> $ gsi -:tU
>>>>
>>>> ;;;;;;;;
>>>>
>>>> (string->list "über")
>>>> ;; -> (#\xfc #\b #\e #\r)
>>>> ;; expected (#\ü #\b #\e #\r)
>>>>
>>>> (list->string (string->list "über"))
>>>> ;; -> "über"
>>>> ;; ok
>>>>
>>>> (map list->string (list (string->list "über")))
>>>> ;; -> ("\374ber")
>>>> ;; expected ("über")
>>>>
>>>> ;;;;;;;;
>>>>
>>>> I expected (string->list "über") to return (#\ü #\b #\e #\r) and that
>>>> list->string does not behave different when used with map.
>>>>
>>>> When setting readtable-max-unescaped-char-set in the gambcini file
>>>> string->list and list->string work as expected.
>>>>
>>>> Marc Feeley <feeley at iro.umontreal.ca> writes:
>>>>
>>>>> I have pushed a change which allows #f as the setting of the
>>>>> max-unescaped-char of a readtable, and it is now the default instead of
>>>>> (integer->char 127).  When max-unescaped-char is #f the printer will
>>>>> take into account the character encoding of the output port and use
>>>>> escapes when the character can’t be encoded.  I have also changed the
>>>>> default character encoding of the terminal, files, etc to use ASCII.
>>>>> This setting is actually equivalent to ISO-8859-1 (latin-1) at the
>>>>> lowest level of the runtime system, i.e. chars are encoded using 8 bits
>>>>> and the top bit is significant, but the printer will use escapes when
>>>>> the character code is above 127 (because ASCII is a 7 bit code).
>>>>>
>>>>> This means that when no runtime options are passed to gsi the system
>>>>> will escape characters exactly as before.  However, with
>>>>>
>>>>>   gsi -:tU    (or gsi -:t8)
>>>>>
>>>>> the system will not escape characters whose code is > 127.
>>>>>
>>>>> For the users prefering this behavior, including you I assume, it can be
>>>>> made the default by setting the GAMBCOPT environment variable like this
>>>>> in your .profile:
>>>>>
>>>>>  GAMBCOPT=tU
>>>>>
>>>>> This is better than changing the readtable in the gambcini file because
>>>>> it will apply to all Scheme programs compiled with Gambit, not just the
>>>>> interpreter, and it can be overriden with an explicit -:tA when starting
>>>>> gsi if you want to escape non-ASCII characters in a specific situation.
>>>>>
>>>>> Marc
>>>>>
>>>>>
>>>>>> On May 30, 2015, at 2:43 AM, Atticus <atticus0 at posteo.org> wrote:
>>>>>>
>>>>>>
>>>>>>> By default the “write”, “pretty-print”, etc procedures (including the
>>>>>>> REPL’s printer) assume that the output port can only display ASCII, so
>>>>>>> it encodes non-ASCII characters as sequences such as \374 or #xfc
>>>>>>> (which are composed of ASCII characters).  This allows Scheme strings
>>>>>>> that are printed to be processed by external tools which only support
>>>>>>> ASCII.  For example, if you pretty-print to a file a function that
>>>>>>> contains the string "ü" you will end up with a file containing the
>>>>>>> sequence of 6 characters "\374" which any editor can edit. In other
>>>>>>> words, it makes minimal requirements on the features of external tools
>>>>>>> (not just editors, but email clients if you want to email some Scheme
>>>>>>> code, shell utilities, C compilers, other Scheme systems, etc).
>>>>>>
>>>>>> Very interesting, thank you.
>>>>>>
>>>>>>> Should the external representation of strings depend on the character
>>>>>>> encoding of the output port?  In other words, if the character
>>>>>>> encoding of the byte output port is
>>>>>>>
>>>>>>> - ASCII then characters whose code are >= 128 would use escapes like
>>>>>>> \374
>>>>>>> - ISO-8859-1 then characters whose code are >= 256 would use escapes
>>>>>>> - UCS-2 then characters whose code are >= 65536 would use escapes
>>>>>>> - in all other cases escapes would not be used because all Unicode
>>>>>>> characters can be encoded
>>>>>>
>>>>>> Imho yes. But to be clear, let's say the external representation of
>>>>>> scheme strings depends now on the character encoding of the output port
>>>>>> and I have the above explained situation, I have an utf-8 port and need
>>>>>> to print to ASCII only, then I can still do that by changing the output
>>>>>> port to ASCII manually before printing?
>>>>>>
>>>>>> Marc Feeley <feeley at iro.umontreal.ca> writes:
>>>>>>
>>>>>>> By default the “write”, “pretty-print”, etc procedures (including the
>>>>>>> REPL’s printer) assume that the output port can only display ASCII, so
>>>>>>> it encodes non-ASCII characters as sequences such as \374 or #xfc
>>>>>>> (which are composed of ASCII characters).  This allows Scheme strings
>>>>>>> that are printed to be processed by external tools which only support
>>>>>>> ASCII.  For example, if you pretty-print to a file a function that
>>>>>>> contains the string "ü" you will end up with a file containing the
>>>>>>> sequence of 6 characters "\374" which any editor can edit. In other
>>>>>>> words, it makes minimal requirements on the features of external tools
>>>>>>> (not just editors, but email clients if you want to email some Scheme
>>>>>>> code, shell utilities, C compilers, other Scheme systems, etc).
>>>>>>>
>>>>>>> Note that this behavior was chosen a long time ago when UTF-8 was not
>>>>>>> widespread.  I’m open to changing this behavior, but I’d like to know
>>>>>>> what other people think.
>>>>>>>
>>>>>>> Should the external representation of strings depend on the character
>>>>>>> encoding of the output port?  In other words, if the character
>>>>>>> encoding of the byte output port is
>>>>>>>
>>>>>>> - ASCII then characters whose code are >= 128 would use escapes like
>>>>>>> \374
>>>>>>> - ISO-8859-1 then characters whose code are >= 256 would use escapes
>>>>>>> - UCS-2 then characters whose code are >= 65536 would use escapes
>>>>>>> - in all other cases escapes would not be used because all Unicode
>>>>>>> characters can be encoded
>>>>>>>
>>>>>>> By the way this would also affect the external representation of
>>>>>>> symbols, such as 'über .
>>>>>>>
>>>>>>> Marc
>>>>>>>
>>>>>>>
>>>>>>>> On May 29, 2015, at 2:12 PM, Atticus <atticus0 at posteo.org> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> First and formost thanks for developing and maintaining gambit.
>>>>>>>>
>>>>>>>> As a beginner I'm a litte confused about the handling of the
>>>>>>>> character
>>>>>>>> encoding in the repl.
>>>>>>>>
>>>>>>>> $ gsi -:d,t8,f8
>>>>>>>> $> "ü"
>>>>>>>> "\374"
>>>>>>>>
>>>>>>>> $> (string-ref "ü" 0)
>>>>>>>> #\xfc
>>>>>>>>
>>>>>>>> $> (string->list "über")
>>>>>>>> (#\xfc #\b #\e #\r)
>>>>>>>>
>>>>>>>> $> (with-output-to-file "test2" (lambda () (##write-string "ü")
>>>>>>>> $> ,q
>>>>>>>>
>>>>>>>> $ cat test2
>>>>>>>> ü
>>>>>>>>
>>>>>>>> $ gsi -:d,t8,f8
>>>>>>>> $> (with-input-from-file "test2" (lambda () (read-char)))
>>>>>>>> #\xfc
>>>>>>>>
>>>>>>>>
>>>>>>>> With racket:
>>>>>>>>
>>>>>>>> $ racket
>>>>>>>> $>"ü"
>>>>>>>> "ü"
>>>>>>>>
>>>>>>>> $> (with-input-from-file "test2" (lambda () (read-char)))
>>>>>>>> #\ü
>>>>>>>>
>>>>>>>> $> (string->list "über")
>>>>>>>> '(#\ü #\b #\e #\r)
>>>>>>>>
>>>>>>>>
>>>>>>>> Why does gambit return "\374" instead of "ü" in the repl when
>>>>>>>> evaluating
>>>>>>>> the string "ü"? I'm curious because in racket the repl returns the
>>>>>>>> evaluated string as "ü"", also read-char in racket returns #\ü
>>>>>>>> instead
>>>>>>>> of #\xfc.
>>>>>>>>
>>>>>>>> Looking at the documentation I could change the behaviour with:
>>>>>>>> ;;;;
>>>>>>>> (output-port-readtable-set!
>>>>>>>> (repl-output-port)
>>>>>>>> (readtable-max-unescaped-char-set
>>>>>>>>   (output-port-readtable (repl-output-port))
>>>>>>>>   #\U0010ffff))
>>>>>>>> ;;;;
>>>>>>>>
>>>>>>>> Are there any advantages for having this default readtable setting in
>>>>>>>> gambit? As a beginner I think it's confusing.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gambit-list mailing list
>>>>>>>> Gambit-list at iro.umontreal.ca
>>>>>>>> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
>> _______________________________________________
>> Gambit-list mailing list
>> Gambit-list at iro.umontreal.ca
>> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
>>



More information about the Gambit-list mailing list