[gambit-list] Some UTF-8 support issues.

Marc Feeley feeley at iro.umontreal.ca
Mon Sep 7 11:13:07 EDT 2009


On 7-Sep-09, at 9:13 AM, Ramil Farkhshatov wrote:

> Hello.
>
> 1. WRITE and PP procedures output unicode (UTF-8 with nonlatin
> characters) strings in such form:
> 	"\uXXXX\uXXXX" ; when run with t8 option
> 	"\XXX\XXX" ; when run without t8 option
> Also procedures returning strings output in REPL acts in same manner.

I don't get this behavior with v4.5.1.  Which version are you using?

> Is there a way to change that behaviour?

You can change the readtable's "max-unescaped-char" attribute:

(define (set-no-escapes-for-unicode! port)
   (output-port-readtable-set!
    port
    (readtable-max-unescaped-char-set
     (output-port-readtable port)
     (integer->char #x10ffff))))

(define (go)
   (pretty-print
    (list->string (map integer->char
                       (list #x80 #x100 #x10000)))))

(go)

(set-no-escapes-for-unicode! (current-output-port))

(go)

> 2. Have troubles operating files with unicode names when gsi runned
> with t8 option.
>
> With t8 option:
>> (file-exists? "~/develop/каталог")
> #f
>
> Without t8 option:
>> (file-exists? "~/develop/каталог")
> #f
>
> OPEN-DIRECTORY cannot open directories with unicode names:
>> (define fd (open-directory "~/develop/каталог/"))
> *** ERROR IN (console)@20.12 -- (Argument 1) Can't convert to C char- 
> string
> (open-directory "~/develop/\u043a\u0430\u0442\u0430\u043b\u043e 
> \u0433/")
>
> Additional info:
> % locale
> LANG=ru_RU.UTF-8
> LC_CTYPE="ru_RU.UTF-8"
> LC_NUMERIC="ru_RU.UTF-8"
> LC_TIME="ru_RU.UTF-8"
> LC_COLLATE=C
> LC_MONETARY="ru_RU.UTF-8"
> LC_MESSAGES="ru_RU.UTF-8"
> LC_PAPER="ru_RU.UTF-8"
> LC_NAME="ru_RU.UTF-8"
> LC_ADDRESS="ru_RU.UTF-8"
> LC_TELEPHONE="ru_RU.UTF-8"
> LC_MEASUREMENT="ru_RU.UTF-8"
> LC_IDENTIFICATION="ru_RU.UTF-8"
> LC_ALL=
>
> % echo $GAMBCOPT
> dar,t8E,f8,-8

I'm not sure what should be the right behavior.  Currently, a latin-1  
encoding will be used for the filenames.  There is very little code to  
change in the runtime system to use UTF-8 encoding for the filenames,  
but this means that filesystems using plain character (latin-1)  
encoding for filenames will stop working.  I could add a configure  
option, such as --enable-path-encoding=UTF-8 .  But that begs the  
question: what's the right default?  Is the default OS dependent?

Marc




More information about the Gambit-list mailing list