Hello,
First and formost thanks for developing and maintaining gambit.
As a beginner I'm a litte confused about the handling of the character encoding in the repl.
$ gsi -:d,t8,f8 $> "ü" "\374"
$> (string-ref "ü" 0) #\xfc
$> (string->list "über") (#\xfc #\b #\e #\r)
$> (with-output-to-file "test2" (lambda () (##write-string "ü") $> ,q
$ cat test2 ü
$ gsi -:d,t8,f8 $> (with-input-from-file "test2" (lambda () (read-char))) #\xfc
With racket:
$ racket $>"ü" "ü"
$> (with-input-from-file "test2" (lambda () (read-char))) #\ü
$> (string->list "über") '(#\ü #\b #\e #\r)
Why does gambit return "\374" instead of "ü" in the repl when evaluating the string "ü"? I'm curious because in racket the repl returns the evaluated string as "ü"", also read-char in racket returns #\ü instead of #\xfc.
Looking at the documentation I could change the behaviour with: ;;;; (output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) #\U0010ffff)) ;;;;
Are there any advantages for having this default readtable setting in gambit? As a beginner I think it's confusing.
Afficher les réponses par date
You forgot doing (print "\374\n") in gsi -:t8 .
Return in the REPL is not correct terminology, but why does the write used in the print step in the REPL escape.
Guaranteed gorrectness by default, disable by gsi -e (your script) - .
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374 - ISO-8859-1 then characters whose code are >= 256 would use escapes - UCS-2 then characters whose code are >= 65536 would use escapes - in all other cases escapes would not be used because all Unicode characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Marc
On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote:
Hello,
First and formost thanks for developing and maintaining gambit.
As a beginner I'm a litte confused about the handling of the character encoding in the repl.
$ gsi -:d,t8,f8 $> "ü" "\374"
$> (string-ref "ü" 0) #\xfc
$> (string->list "über") (#\xfc #\b #\e #\r)
$> (with-output-to-file "test2" (lambda () (##write-string "ü") $> ,q
$ cat test2 ü
$ gsi -:d,t8,f8 $> (with-input-from-file "test2" (lambda () (read-char))) #\xfc
With racket:
$ racket $>"ü" "ü"
$> (with-input-from-file "test2" (lambda () (read-char))) #\ü
$> (string->list "über") '(#\ü #\b #\e #\r)
Why does gambit return "\374" instead of "ü" in the repl when evaluating the string "ü"? I'm curious because in racket the repl returns the evaluated string as "ü"", also read-char in racket returns #\ü instead of #\xfc.
Looking at the documentation I could change the behaviour with: ;;;; (output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) #\U0010ffff)) ;;;;
Are there any advantages for having this default readtable setting in gambit? As a beginner I think it's confusing.
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
2015-05-30 0:38 GMT+05:30 Marc Feeley feeley@iro.umontreal.ca:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode
characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Symbols should under ordinary conditions not contain chars >= 128 as standard software practice is to implement all identifiers in English, or englishify them at the level of alphabet use e.g. 'uber ;
If anybody with a non-English-alphabet wants to use Gambti for instance for conveniency in educational purposes, then they can reconfigure themselves.
The amount of systems that are not UTF8-compliant is unlimited, and you never know when data copied/pasted etc. would pass through that, so keeping the default setting as sparse as possible makes enormous sense. For instance; * Protocols: the SMTP and HTTP protcol is not UTF-8 by default, at least with not-too-unrecent browsers. I guess the same applies with, * Filesystems (FAT), * Various text editors I guess - at least VIM last I checked - and * Various other tools. * Fonts: You don't even know what characters beyond 0..127 that fonts implement, and absence of chars in the 128..256 range happens.
After having pondered extending the default even just from 127 to 255 by looking at http://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout , I think an extension would be a bad idea because of the above reasons.
The one who wants more than that for text output can just use the non-escaping output routines e.g. print and display, that is really part of minimum knowledge.
2015-05-30 1:03 GMT+05:30 Adam adam.mlmb@gmail.com:
Symbols should under ordinary conditions not contain chars >= 128 as standard software practice is to implement all identifiers in English, or englishify them at the level of alphabet use e.g. 'uber ;
If anybody with a non-English-alphabet wants to use Gambti for instance for conveniency in educational purposes, then they can reconfigure themselves.
The amount of systems that are not UTF8-compliant is unlimited, and you never know when data copied/pasted etc. would pass through that,
(Wait, it's not UTF8 compliancy that's the question here, but whether the protocol/tool/etc. does that and also is UTF8 by default.)
so keeping the default setting as sparse as possible makes enormous sense. For instance;
- Protocols: the SMTP and HTTP protcol is not UTF-8 by default, at least
with not-too-unrecent browsers. I guess the same applies with,
- Filesystems (FAT),
- Various text editors I guess - at least VIM last I checked - and
- Various other tools.
- Fonts: You don't even know what characters beyond 0..127 that fonts
implement, and absence of chars in the 128..256 range happens.
After having pondered extending the default even just from 127 to 255 by looking at http://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout , I think an extension would be a bad idea because of the above reasons.
The one who wants more than that for text output can just use the non-escaping output routines e.g. print and display, that is really part of minimum knowledge.
Wait, if the default -:t setting could be changed from "1"("A"?) to "8" with the motivation that that would make it friendlier out of the box,
what would the implied worst-case be, for instance on a 0..255 only terminal?
So the -:t setting affects both console IO (print read-line etc.) and REPL IO (the Read step).
( I guess the implied worst-case for the current "1"("A"?) default is
* If the terminal is UTF-8 only: Trashing of input of >=128 characters, e.g.
gsi
(string->list "ü")
(#\xc3 #\xbc)
(REPL left-right input behavior when attempting to input them is messed up but that could maybe be fixed.
Non-escaped output of the trashed input has an "untrashing" effect which can conceal the trashing problem, but that is intended, as in that the ISO-8859-1(&ASCII?) coder has a 1:1 byte to char mapping, and also showing the same with escaped output immediately shows the problem e.g. "\303\274" for what should have been "\374".)
* Scary error messages when trying to output unicode chars
(print (make-string 1 (integer->char 1234)) "\n")
*** ERROR IN ##write-substring -- Can't convert to C ISO-8859-1-string (write-char #\u04d2 '#<input-output-port #2 (console)>) 1>
)
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Marc
On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote:
Hello,
First and formost thanks for developing and maintaining gambit.
As a beginner I'm a litte confused about the handling of the character encoding in the repl.
$ gsi -:d,t8,f8 $> "ü" "\374"
$> (string-ref "ü" 0) #\xfc
$> (string->list "über") (#\xfc #\b #\e #\r)
$> (with-output-to-file "test2" (lambda () (##write-string "ü") $> ,q
$ cat test2 ü
$ gsi -:d,t8,f8 $> (with-input-from-file "test2" (lambda () (read-char))) #\xfc
With racket:
$ racket $>"ü" "ü"
$> (with-input-from-file "test2" (lambda () (read-char))) #\ü
$> (string->list "über") '(#\ü #\b #\e #\r)
Why does gambit return "\374" instead of "ü" in the repl when evaluating the string "ü"? I'm curious because in racket the repl returns the evaluated string as "ü"", also read-char in racket returns #\ü instead of #\xfc.
Looking at the documentation I could change the behaviour with: ;;;; (output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) #\U0010ffff)) ;;;;
Are there any advantages for having this default readtable setting in gambit? As a beginner I think it's confusing.
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
What could maybe be something, would be to add another -: option to Gambit, that instructs Gambit to use a non-default(non-127) readtable-max-unescaped-char-set setting.
Like, -:eN (e as in "escape" and "N" as in "nothing", meaning #\U0010ffff).
This would make the escaping behavior you seem to be hinting at, more easily accessible (i.e. more easy to get going than to need to include the whole script you printed out before in an -e ... - argument).
2015-05-30 12:13 GMT+05:30 Atticus atticus0@posteo.org:
Should the external representation of strings depend on the character
encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode
characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output
Atticus, you're forgetting that the "external representation" discussed here regards only the escaping applied in the serialization of objects.
This is a really sensitive thing where exactitude and correctness in any real use must have priority over extra nice features.
The character encoding of the output and input port here is the same, and setting Gambit in -:f8 to be able to read UTF-8 encoded sourcecode files, and -:t8 to be able to input unicode chars directly in the REPL, is generally a good thing -
And there's no reason that that would somehow be related to that escaping behavior would change.
Again, the direct output is always done unescaped:
gsi -:t8 (print "\374\n")
If you want unescaped object output occasionally only, you could implement your own write/pp routine with readtable-max-unescaped-char-set set as you want.
Like, -:eN (e as in "escape" and "N" as in "nothing", meaning #\U0010ffff).
Would be really convenient but I can also put the configuration into my gambcini file.
Atticus, you're forgetting that the "external representation" discussed here regards only the escaping applied in the serialization of objects.
I dont know if I understand that correctly. I assumed we are discussing this change:
with t8 or f8, before: (write "ü") -> "\374" (read port) -> "ü"
with t8 or f8 after change: (write "ü") -> "ü" (read port) -> "ü"
This is a really sensitive thing where exactitude and correctness in any real use must have priority over extra nice features.
Exactness and correctness > nice features, agreed. But I don't understand whats not exact or not correct about the proposed change. It seems to be just different. I'm quite inexperienced so it's possible that I underestimate the implications of the change.
Also it's not that important or a priority for me, and as you said, I can easily change the readtable-max-unescaped-char-set so I have nice looking utf-8 character when reading non-english data into the repl. So no problem for me :)
It would be interesting to hear more opinions on that matter.
Adam adam.mlmb@gmail.com writes:
What could maybe be something, would be to add another -: option to Gambit, that instructs Gambit to use a non-default(non-127) readtable-max-unescaped-char-set setting.
Like, -:eN (e as in "escape" and "N" as in "nothing", meaning #\U0010ffff).
This would make the escaping behavior you seem to be hinting at, more easily accessible (i.e. more easy to get going than to need to include the whole script you printed out before in an -e ... - argument).
2015-05-30 12:13 GMT+05:30 Atticus atticus0@posteo.org:
Should the external representation of strings depend on the character
encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode
characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output
Atticus, you're forgetting that the "external representation" discussed here regards only the escaping applied in the serialization of objects.
This is a really sensitive thing where exactitude and correctness in any real use must have priority over extra nice features.
The character encoding of the output and input port here is the same, and setting Gambit in -:f8 to be able to read UTF-8 encoded sourcecode files, and -:t8 to be able to input unicode chars directly in the REPL, is generally a good thing -
And there's no reason that that would somehow be related to that escaping behavior would change.
Again, the direct output is always done unescaped:
gsi -:t8 (print "\374\n")
If you want unescaped object output occasionally only, you could implement your own write/pp routine with readtable-max-unescaped-char-set set as you want.
2015-05-30 14:15 GMT+05:30 Atticus atticus0@posteo.org:
Like, -:eN (e as in "escape" and "N" as in "nothing", meaning
#\U0010ffff).
Would be really convenient but I can also put the configuration into my gambcini file.
Atticus, you're forgetting that the "external representation" discussed here regards only the escaping applied in the serialization of objects.
I dont know if I understand that correctly. I assumed we are discussing this change:
with t8 or f8, before: (write "ü") -> "\374" (read port) -> "ü"
with t8 or f8 after change: (write "ü") -> "ü" (read port) -> "ü"
Right - what I meant was, remember that, if you desire to do output to the terminal in your application, you should *never* use |write|.
And, if you would write an application where a user inputs content from the terminal, then you should *never* use |read|.
|read| and |write| are not IO routines specifically, but object /de/serialization + IO combo routines.
I.e., |write| is not used for presentation, but to give you an absolute serialized representation in text form of an object's contents, and there human readability is secondary to technical correctness.
For instance, (write "ü") printing "\374" shows you that the string's character coding is correct (vs. for instance "\303\274" which would show you it's broken). If it prints out "ü", you have no idea.
Also, a good reason for escaping is that there would even be need of security considerations for not escaping special characters like https://en.wikipedia.org/wiki/Unicode_control_characters and https://en.wikipedia.org/wiki/Bi-directional_text#Unicode_bidi_support . If there exists any sensible across-the-board way to handle those except for escaping them, and if there's anything more of the sort than these in particular in Unicode, is beyond my present knowledge.
The pure IO output routines are |display|,|print|, |read-substring|, |read-char|, and the pure IO intput routines are |read-char|, |read-line|, |read-subu8vector|.
I.e., with t8 or f8, (display "ü") -> "ü" already and it always did.
And in the light of this, probably if you want (write "ü") to specifically ouptut "ü" rather than "\374", then maybe/probably you are doing something wrong, as reflected by Marc through what he said in the previous email
2015-05-30 0:38 GMT+05:30 Marc Feeley feeley@iro.umontreal.ca:
This [escaping of all >=128 chars] allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
The only strict usecase I can see of having >=128 characters not escaped by default, is if you have a class full of elementary school kids with another language than English as mother tongue, who find it so confusing during basic programming classes that them typing in 'über is object-serialized as |\374ber|, that you need to do something about it.
This question of escaping we're discussing, is very similar to the discussion of if it's good practice to use unicode characters in variable names.
Until now I never saw any project do that.
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 2:43 AM, Atticus atticus0@posteo.org wrote:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Marc
On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote:
Hello,
First and formost thanks for developing and maintaining gambit.
As a beginner I'm a litte confused about the handling of the character encoding in the repl.
$ gsi -:d,t8,f8 $> "ü" "\374"
$> (string-ref "ü" 0) #\xfc
$> (string->list "über") (#\xfc #\b #\e #\r)
$> (with-output-to-file "test2" (lambda () (##write-string "ü") $> ,q
$ cat test2 ü
$ gsi -:d,t8,f8 $> (with-input-from-file "test2" (lambda () (read-char))) #\xfc
With racket:
$ racket $>"ü" "ü"
$> (with-input-from-file "test2" (lambda () (read-char))) #\ü
$> (string->list "über") '(#\ü #\b #\e #\r)
Why does gambit return "\374" instead of "ü" in the repl when evaluating the string "ü"? I'm curious because in racket the repl returns the evaluated string as "ü"", also read-char in racket returns #\ü instead of #\xfc.
Looking at the documentation I could change the behaviour with: ;;;; (output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) #\U0010ffff)) ;;;;
Are there any advantages for having this default readtable setting in gambit? As a beginner I think it's confusing.
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
Wow that was quick.
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
Awesome.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Thanks a lot Marc!
Marc Feeley feeley@iro.umontreal.ca writes:
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 2:43 AM, Atticus atticus0@posteo.org wrote:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Marc
On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote:
Hello,
First and formost thanks for developing and maintaining gambit.
As a beginner I'm a litte confused about the handling of the character encoding in the repl.
$ gsi -:d,t8,f8 $> "ü" "\374"
$> (string-ref "ü" 0) #\xfc
$> (string->list "über") (#\xfc #\b #\e #\r)
$> (with-output-to-file "test2" (lambda () (##write-string "ü") $> ,q
$ cat test2 ü
$ gsi -:d,t8,f8 $> (with-input-from-file "test2" (lambda () (read-char))) #\xfc
With racket:
$ racket $>"ü" "ü"
$> (with-input-from-file "test2" (lambda () (read-char))) #\ü
$> (string->list "über") '(#\ü #\b #\e #\r)
Why does gambit return "\374" instead of "ü" in the repl when evaluating the string "ü"? I'm curious because in racket the repl returns the evaluated string as "ü"", also read-char in racket returns #\ü instead of #\xfc.
Looking at the documentation I could change the behaviour with: ;;;; (output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) #\U0010ffff)) ;;;;
Are there any advantages for having this default readtable setting in gambit? As a beginner I think it's confusing.
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Marc, what do those who want preserve the escaping of >=128 do (independent of which terminal encoding is used but for instance with -:t8)?
2015-05-30 21:57 GMT+05:30 Marc Feeley feeley@iro.umontreal.ca:
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 3:26 PM, Adam adam.mlmb@gmail.com wrote:
Marc, what do those who want preserve the escaping of >=128 do (independent of which terminal encoding is used but for instance with -:t8)?
(output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) (integer->char 127)))
In your Scheme program or gambcini file.
Marc
2015-05-31 1:11 GMT+05:30 Marc Feeley feeley@iro.umontreal.ca:
On May 30, 2015, at 3:26 PM, Adam adam.mlmb@gmail.com wrote:
Marc, what do those who want preserve the escaping of >=128 do
(independent of which terminal encoding is used but for instance with -:t8)?
(output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) (integer->char 127)))
In your Scheme program or gambcini file.
Ouch;
What's your take now on what you said yesterday as quoted in https://mercure.iro.umontreal.ca/pipermail/gambit-list/2015-May/007914.html , and the other reflections on this topic there?
Marc,
The new autodetection routine at lib/_io#.scm:747+ in https://github.com/feeley/gambit/commit/8d2c2e60c007a473f1aa1d16110c4a547d29... is super nice.
I agree it's nice that all readable european, asian, etc. characters come out unescaped, but it's weird to know that Gambit would come not-copy-paste-proof out of the box, and that you need a specialized script at startup to fix that.
I use the best terminal software and it does not show all unicode characters;
To understand the implications of unescaped Unicode characters, I read up on http://unicode.org/charts/ and other places, and it seems that the worst abuse of Unicode characters is illustrated here http://www.marlborotech.com/Zalgo.html , and maybe also the right-to-left/left-to-right characters can cause disorder -
for reference, attached a screenshot of how messed up unicode shows in Chrome and its DOM inspector, so this is how a Gambit console window could look. In itself not too bad.
I would primarily suggest that a command line option would be added that configures the escaping e.g. -:E , -:E127 (or maybe -:T127 for terminal and -:F127 for files).
Secondarily I would suggest reverting lib/_io.scm:13119 back to (##integer->char 127) i.e. escape all >127 again, as at least I find this to be really more a "safe by default" behavior.
Anyone else has any thoughts on this topic?
2015-05-31 1:24 GMT+05:30 Adam adam.mlmb@gmail.com:
2015-05-31 1:11 GMT+05:30 Marc Feeley feeley@iro.umontreal.ca:
On May 30, 2015, at 3:26 PM, Adam adam.mlmb@gmail.com wrote:
Marc, what do those who want preserve the escaping of >=128 do
(independent of which terminal encoding is used but for instance with -:t8)?
(output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) (integer->char 127)))
In your Scheme program or gambcini file.
Ouch;
What's your take now on what you said yesterday as quoted in https://mercure.iro.umontreal.ca/pipermail/gambit-list/2015-May/007914.html , and the other reflections on this topic there?
The functionnality is in the Gambit runtime system to configure the printer to escape characters based on a user preference (max-unescaped-char in the readtable) and also based on the output port character encoding. Your proposed -:E127 command-line runtime options would add another way to customize the printer based on user preference, which shouldn’t be too hard to add but would still require user intervention.
So the real issue is what should the default be. You favour “safe by default” (i.e. external representation of Scheme chars, strings, symbols is always in pure ASCII regardless of the output port), and I can appreciate its value. But I can also appreciate the value of seeing literal non-ASCII characters in code without the escapes, for example:
(define (f x) (string-append "déjà " x)) (println (f "vu"))
déjà vu
(pp f)
(lambda (x) (string-append "déjà " x))
instead of
(lambda (x) (string-append "d\351j\340 " x))
It makes editing and debugging code more pleasant for programs manipulating text other than english.
So I’d like to hear what other people think the default should be.
Marc
On May 31, 2015, at 7:14 AM, Adam adam.mlmb@gmail.com wrote:
I would primarily suggest that a command line option would be added that configures the escaping e.g. -:E , -:E127 (or maybe -:T127 for terminal and -:F127 for files).
Secondarily I would suggest reverting lib/_io.scm:13119 back to (##integer->char 127) i.e. escape all >127 again, as at least I find this to be really more a "safe by default" behavior.
Anyone else has any thoughts on this topic?
It is not clear to me why it isn’t sufficient for your use case to just call gsi without the -:t8 runtime option. This will display strings/symbols/chars pretty-printed by the REPL using escapes on Unicode characters > 127.
Put another way, what would be the motivation to start gsi with the -:t8 runtime option if you want to see pretty-printed strings with escapes?
Marc
On May 30, 2015, at 9:54 PM, Adam adam.mlmb@gmail.com wrote:
2015-05-31 1:11 GMT+05:30 Marc Feeley feeley@iro.umontreal.ca:
On May 30, 2015, at 3:26 PM, Adam adam.mlmb@gmail.com wrote:
Marc, what do those who want preserve the escaping of >=128 do (independent of which terminal encoding is used but for instance with -:t8)?
(output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) (integer->char 127)))
In your Scheme program or gambcini file.
Ouch;
What's your take now on what you said yesterday as quoted in https://mercure.iro.umontreal.ca/pipermail/gambit-list/2015-May/007914.html , and the other reflections on this topic there?
Marc,
The new autodetection routine at lib/_io#.scm:747+ in https://github.com/feeley/gambit/commit/8d2c2e60c007a473f1aa1d16110c4a547d29... is super nice.
I agree it's nice that all readable european, asian, etc. characters come out unescaped, but it's weird to know that Gambit would come not-copy-paste-proof out of the box, and that you need a specialized script at startup to fix that.
I use the best terminal software and it does not show all unicode characters;
To understand the implications of unescaped Unicode characters, I read up on http://unicode.org/charts/ and other places, and it seems that the worst abuse of Unicode characters is illustrated here http://www.marlborotech.com/Zalgo.html , and maybe also the right-to-left/left-to-right characters can cause disorder - it's quite bad but not fatal.
I would primarily suggest that a command line option would be added that configures the escaping e.g. -:E , -:E127 (or maybe -:T127 for terminal and -:F127 for files).
Secondarily I would suggest reverting lib/_io.scm:13119 back to (##integer->char 127) i.e. escape all >127 again, as at least I find this to be really more a "safe by default" behavior.
Anyone else has any thoughts on this topic?
2015-05-31 1:11 GMT+05:30 Marc Feeley feeley@iro.umontreal.ca:
On May 30, 2015, at 3:26 PM, Adam adam.mlmb@gmail.com wrote:
Marc, what do those who want preserve the escaping of >=128 do
(independent of which terminal encoding is used but for instance with -:t8)?
(output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) (integer->char 127)))
In your Scheme program or gambcini file.
Marc
Some observations about the escaping of characters with the 8d2c2e6 commit:
$ gsi -:tU
;;;;;;;;
(string->list "über") ;; -> (#\xfc #\b #\e #\r) ;; expected (#\ü #\b #\e #\r)
(list->string (string->list "über")) ;; -> "über" ;; ok
(map list->string (list (string->list "über"))) ;; -> ("\374ber") ;; expected ("über")
;;;;;;;;
I expected (string->list "über") to return (#\ü #\b #\e #\r) and that list->string does not behave different when used with map.
When setting readtable-max-unescaped-char-set in the gambcini file string->list and list->string work as expected.
Marc Feeley feeley@iro.umontreal.ca writes:
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 2:43 AM, Atticus atticus0@posteo.org wrote:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Marc
On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote:
Hello,
First and formost thanks for developing and maintaining gambit.
As a beginner I'm a litte confused about the handling of the character encoding in the repl.
$ gsi -:d,t8,f8 $> "ü" "\374"
$> (string-ref "ü" 0) #\xfc
$> (string->list "über") (#\xfc #\b #\e #\r)
$> (with-output-to-file "test2" (lambda () (##write-string "ü") $> ,q
$ cat test2 ü
$ gsi -:d,t8,f8 $> (with-input-from-file "test2" (lambda () (read-char))) #\xfc
With racket:
$ racket $>"ü" "ü"
$> (with-input-from-file "test2" (lambda () (read-char))) #\ü
$> (string->list "über") '(#\ü #\b #\e #\r)
Why does gambit return "\374" instead of "ü" in the repl when evaluating the string "ü"? I'm curious because in racket the repl returns the evaluated string as "ü"", also read-char in racket returns #\ü instead of #\xfc.
Looking at the documentation I could change the behaviour with: ;;;; (output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) #\U0010ffff)) ;;;;
Are there any advantages for having this default readtable setting in gambit? As a beginner I think it's confusing.
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Ha ha! That’s a funny issue caused by an optimization of the pretty printer. Now fixed.
Marc
On Jun 5, 2015, at 3:57 PM, Atticus atticus0@posteo.org wrote:
Some observations about the escaping of characters with the 8d2c2e6 commit:
$ gsi -:tU
;;;;;;;;
(string->list "über") ;; -> (#\xfc #\b #\e #\r) ;; expected (#\ü #\b #\e #\r)
(list->string (string->list "über")) ;; -> "über" ;; ok
(map list->string (list (string->list "über"))) ;; -> ("\374ber") ;; expected ("über")
;;;;;;;;
I expected (string->list "über") to return (#\ü #\b #\e #\r) and that list->string does not behave different when used with map.
When setting readtable-max-unescaped-char-set in the gambcini file string->list and list->string work as expected.
Marc Feeley feeley@iro.umontreal.ca writes:
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 2:43 AM, Atticus atticus0@posteo.org wrote:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Marc
On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote:
Hello,
First and formost thanks for developing and maintaining gambit.
As a beginner I'm a litte confused about the handling of the character encoding in the repl.
$ gsi -:d,t8,f8 $> "ü" "\374"
$> (string-ref "ü" 0) #\xfc
$> (string->list "über") (#\xfc #\b #\e #\r)
$> (with-output-to-file "test2" (lambda () (##write-string "ü") $> ,q
$ cat test2 ü
$ gsi -:d,t8,f8 $> (with-input-from-file "test2" (lambda () (read-char))) #\xfc
With racket:
$ racket $>"ü" "ü"
$> (with-input-from-file "test2" (lambda () (read-char))) #\ü
$> (string->list "über") '(#\ü #\b #\e #\r)
Why does gambit return "\374" instead of "ü" in the repl when evaluating the string "ü"? I'm curious because in racket the repl returns the evaluated string as "ü"", also read-char in racket returns #\ü instead of #\xfc.
Looking at the documentation I could change the behaviour with: ;;;; (output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) #\U0010ffff)) ;;;;
Are there any advantages for having this default readtable setting in gambit? As a beginner I think it's confusing.
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
I encountered another problem.
'shell-command' in gambit release 4.7.6 version and 6b353f does not recognize the character encoding when used with the optional argument:
$ file testfile testfile: UTF-8 Unicode text
$ cat testfile Verhüllung
$ grep hüll testfile Verhüllung
Gambit 6b353f:
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #f) -> Verhüllung -> 0 ;; ok
$ gsi -:tA (shell-command "grep hüll testfile" #t) -> (256 . "")
Now with readtable-max-unescaped-char-set set to #\U0010ffff:
$ gsi -:tA (shell-command "grep hüll testfile" #t) (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) (0 . "Verhüllung\n")
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") (shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n")
Marc Feeley feeley@iro.umontreal.ca writes:
Ha ha! That’s a funny issue caused by an optimization of the pretty printer. Now fixed.
Marc
On Jun 5, 2015, at 3:57 PM, Atticus atticus0@posteo.org wrote:
Some observations about the escaping of characters with the 8d2c2e6 commit:
$ gsi -:tU
;;;;;;;;
(string->list "über") ;; -> (#\xfc #\b #\e #\r) ;; expected (#\ü #\b #\e #\r)
(list->string (string->list "über")) ;; -> "über" ;; ok
(map list->string (list (string->list "über"))) ;; -> ("\374ber") ;; expected ("über")
;;;;;;;;
I expected (string->list "über") to return (#\ü #\b #\e #\r) and that list->string does not behave different when used with map.
When setting readtable-max-unescaped-char-set in the gambcini file string->list and list->string work as expected.
Marc Feeley feeley@iro.umontreal.ca writes:
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 2:43 AM, Atticus atticus0@posteo.org wrote:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Marc
On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote:
Hello,
First and formost thanks for developing and maintaining gambit.
As a beginner I'm a litte confused about the handling of the character encoding in the repl.
$ gsi -:d,t8,f8 $> "ü" "\374"
$> (string-ref "ü" 0) #\xfc
$> (string->list "über") (#\xfc #\b #\e #\r)
$> (with-output-to-file "test2" (lambda () (##write-string "ü") $> ,q
$ cat test2 ü
$ gsi -:d,t8,f8 $> (with-input-from-file "test2" (lambda () (read-char))) #\xfc
With racket:
$ racket $>"ü" "ü"
$> (with-input-from-file "test2" (lambda () (read-char))) #\ü
$> (string->list "über") '(#\ü #\b #\e #\r)
Why does gambit return "\374" instead of "ü" in the repl when evaluating the string "ü"? I'm curious because in racket the repl returns the evaluated string as "ü"", also read-char in racket returns #\ü instead of #\xfc.
Looking at the documentation I could change the behaviour with: ;;;; (output-port-readtable-set! (repl-output-port) (readtable-max-unescaped-char-set (output-port-readtable (repl-output-port)) #\U0010ffff)) ;;;;
Are there any advantages for having this default readtable setting in gambit? As a beginner I think it's confusing.
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Atticus,
I see the printout and corrections you offer here but just for clarity, can you provide a definition of the problem you're seeing?
2015-06-16 15:23 GMT+05:30, Atticus atticus0@posteo.org:
I encountered another problem.
'shell-command' in gambit release 4.7.6 version and 6b353f does not recognize the character encoding when used with the optional argument:
$ file testfile testfile: UTF-8 Unicode text
$ cat testfile Verhüllung
$ grep hüll testfile Verhüllung
Gambit 6b353f:
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #f) -> Verhüllung -> 0 ;; ok
$ gsi -:tA (shell-command "grep hüll testfile" #t) -> (256 . "")
Now with readtable-max-unescaped-char-set set to #\U0010ffff:
$ gsi -:tA (shell-command "grep hüll testfile" #t) (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) (0 . "Verhüllung\n")
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") (shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n")
Marc Feeley feeley@iro.umontreal.ca writes:
Ha ha! That’s a funny issue caused by an optimization of the pretty printer. Now fixed.
Marc
On Jun 5, 2015, at 3:57 PM, Atticus atticus0@posteo.org wrote:
Some observations about the escaping of characters with the 8d2c2e6 commit:
$ gsi -:tU
;;;;;;;;
(string->list "über") ;; -> (#\xfc #\b #\e #\r) ;; expected (#\ü #\b #\e #\r)
(list->string (string->list "über")) ;; -> "über" ;; ok
(map list->string (list (string->list "über"))) ;; -> ("\374ber") ;; expected ("über")
;;;;;;;;
I expected (string->list "über") to return (#\ü #\b #\e #\r) and that list->string does not behave different when used with map.
When setting readtable-max-unescaped-char-set in the gambcini file string->list and list->string work as expected.
Marc Feeley feeley@iro.umontreal.ca writes:
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 2:43 AM, Atticus atticus0@posteo.org wrote:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like
\374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode
characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like
\374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode
characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Marc
> On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote: > > Hello, > > First and formost thanks for developing and maintaining gambit. > > As a beginner I'm a litte confused about the handling of the > character > encoding in the repl. > > $ gsi -:d,t8,f8 > $> "ü" > "\374" > > $> (string-ref "ü" 0) > #\xfc > > $> (string->list "über") > (#\xfc #\b #\e #\r) > > $> (with-output-to-file "test2" (lambda () (##write-string "ü") > $> ,q > > $ cat test2 > ü > > $ gsi -:d,t8,f8 > $> (with-input-from-file "test2" (lambda () (read-char))) > #\xfc > > > With racket: > > $ racket > $>"ü" > "ü" > > $> (with-input-from-file "test2" (lambda () (read-char))) > #\ü > > $> (string->list "über") > '(#\ü #\b #\e #\r) > > > Why does gambit return "\374" instead of "ü" in the repl when > evaluating > the string "ü"? I'm curious because in racket the repl returns the > evaluated string as "ü"", also read-char in racket returns #\ü > instead > of #\xfc. > > Looking at the documentation I could change the behaviour with: > ;;;; > (output-port-readtable-set! > (repl-output-port) > (readtable-max-unescaped-char-set > (output-port-readtable (repl-output-port)) > #\U0010ffff)) > ;;;; > > Are there any advantages for having this default readtable setting in > gambit? As a beginner I think it's confusing. > > _______________________________________________ > Gambit-list mailing list > Gambit-list@iro.umontreal.ca > https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
The problem is when you use 'shell-command' with grep on a file that contains non-ASCII characters (the testfile contains just one line with the word "Verhüllung"), for example searching for "hüll", grep can not find the line or word. I think the examples make this more clear:
With gsi -:tU
(shell-command "grep hüll testfile" #t)
returns (256 . "") but with the runtime option -:tU I expected (0 . "Verhüllung") and
(shell-command "grep Ver testfile" #t)
returns (0 . "Verhüllung\n") but I expected the return value (0 . "Verhüllung").
As I said in the previous post that with readtable-max-unescaped-char-set set to #\U0010ffff, 'shell-command' shows the expected behaviour but not with the -:tU runtime option. Shouldn't be both options equivalent?
Adam adam.mlmb@gmail.com writes:
Atticus,
I see the printout and corrections you offer here but just for clarity, can you provide a definition of the problem you're seeing?
2015-06-16 15:23 GMT+05:30, Atticus atticus0@posteo.org:
I encountered another problem.
'shell-command' in gambit release 4.7.6 version and 6b353f does not recognize the character encoding when used with the optional argument:
$ file testfile testfile: UTF-8 Unicode text
$ cat testfile Verhüllung
$ grep hüll testfile Verhüllung
Gambit 6b353f:
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #f) -> Verhüllung -> 0 ;; ok
$ gsi -:tA (shell-command "grep hüll testfile" #t) -> (256 . "")
Now with readtable-max-unescaped-char-set set to #\U0010ffff:
$ gsi -:tA (shell-command "grep hüll testfile" #t) (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) (0 . "Verhüllung\n")
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") (shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n")
Marc Feeley feeley@iro.umontreal.ca writes:
Ha ha! That’s a funny issue caused by an optimization of the pretty printer. Now fixed.
Marc
On Jun 5, 2015, at 3:57 PM, Atticus atticus0@posteo.org wrote:
Some observations about the escaping of characters with the 8d2c2e6 commit:
$ gsi -:tU
;;;;;;;;
(string->list "über") ;; -> (#\xfc #\b #\e #\r) ;; expected (#\ü #\b #\e #\r)
(list->string (string->list "über")) ;; -> "über" ;; ok
(map list->string (list (string->list "über"))) ;; -> ("\374ber") ;; expected ("über")
;;;;;;;;
I expected (string->list "über") to return (#\ü #\b #\e #\r) and that list->string does not behave different when used with map.
When setting readtable-max-unescaped-char-set in the gambcini file string->list and list->string work as expected.
Marc Feeley feeley@iro.umontreal.ca writes:
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 2:43 AM, Atticus atticus0@posteo.org wrote:
> By default the “write”, “pretty-print”, etc procedures (including the > REPL’s printer) assume that the output port can only display ASCII, so > it encodes non-ASCII characters as sequences such as \374 or #xfc > (which are composed of ASCII characters). This allows Scheme strings > that are printed to be processed by external tools which only support > ASCII. For example, if you pretty-print to a file a function that > contains the string "ü" you will end up with a file containing the > sequence of 6 characters "\374" which any editor can edit. In other > words, it makes minimal requirements on the features of external tools > (not just editors, but email clients if you want to email some Scheme > code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
> Should the external representation of strings depend on the character > encoding of the output port? In other words, if the character > encoding of the byte output port is > > - ASCII then characters whose code are >= 128 would use escapes like > \374 > - ISO-8859-1 then characters whose code are >= 256 would use escapes > - UCS-2 then characters whose code are >= 65536 would use escapes > - in all other cases escapes would not be used because all Unicode > characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
> By default the “write”, “pretty-print”, etc procedures (including the > REPL’s printer) assume that the output port can only display ASCII, so > it encodes non-ASCII characters as sequences such as \374 or #xfc > (which are composed of ASCII characters). This allows Scheme strings > that are printed to be processed by external tools which only support > ASCII. For example, if you pretty-print to a file a function that > contains the string "ü" you will end up with a file containing the > sequence of 6 characters "\374" which any editor can edit. In other > words, it makes minimal requirements on the features of external tools > (not just editors, but email clients if you want to email some Scheme > code, shell utilities, C compilers, other Scheme systems, etc). > > Note that this behavior was chosen a long time ago when UTF-8 was not > widespread. I’m open to changing this behavior, but I’d like to know > what other people think. > > Should the external representation of strings depend on the character > encoding of the output port? In other words, if the character > encoding of the byte output port is > > - ASCII then characters whose code are >= 128 would use escapes like > \374 > - ISO-8859-1 then characters whose code are >= 256 would use escapes > - UCS-2 then characters whose code are >= 65536 would use escapes > - in all other cases escapes would not be used because all Unicode > characters can be encoded > > By the way this would also affect the external representation of > symbols, such as 'über . > > Marc > > >> On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote: >> >> Hello, >> >> First and formost thanks for developing and maintaining gambit. >> >> As a beginner I'm a litte confused about the handling of the >> character >> encoding in the repl. >> >> $ gsi -:d,t8,f8 >> $> "ü" >> "\374" >> >> $> (string-ref "ü" 0) >> #\xfc >> >> $> (string->list "über") >> (#\xfc #\b #\e #\r) >> >> $> (with-output-to-file "test2" (lambda () (##write-string "ü") >> $> ,q >> >> $ cat test2 >> ü >> >> $ gsi -:d,t8,f8 >> $> (with-input-from-file "test2" (lambda () (read-char))) >> #\xfc >> >> >> With racket: >> >> $ racket >> $>"ü" >> "ü" >> >> $> (with-input-from-file "test2" (lambda () (read-char))) >> #\ü >> >> $> (string->list "über") >> '(#\ü #\b #\e #\r) >> >> >> Why does gambit return "\374" instead of "ü" in the repl when >> evaluating >> the string "ü"? I'm curious because in racket the repl returns the >> evaluated string as "ü"", also read-char in racket returns #\ü >> instead >> of #\xfc. >> >> Looking at the documentation I could change the behaviour with: >> ;;;; >> (output-port-readtable-set! >> (repl-output-port) >> (readtable-max-unescaped-char-set >> (output-port-readtable (repl-output-port)) >> #\U0010ffff)) >> ;;;; >> >> Are there any advantages for having this default readtable setting in >> gambit? As a beginner I think it's confusing. >> >> _______________________________________________ >> Gambit-list mailing list >> Gambit-list@iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
Gambit-list mailing list Gambit-list@iro.umontreal.ca https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
It is not clear what is and isn’t a character in the output of your “shell-command”. Could you do a string->list on the cdr of the result of shell-command?
Marc
On Jun 16, 2015, at 11:53 AM, Atticus atticus0@posteo.org wrote:
I encountered another problem.
'shell-command' in gambit release 4.7.6 version and 6b353f does not recognize the character encoding when used with the optional argument:
$ file testfile testfile: UTF-8 Unicode text
$ cat testfile Verhüllung
$ grep hüll testfile Verhüllung
Gambit 6b353f:
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #f) -> Verhüllung -> 0 ;; ok
$ gsi -:tA (shell-command "grep hüll testfile" #t) -> (256 . "")
Now with readtable-max-unescaped-char-set set to #\U0010ffff:
$ gsi -:tA (shell-command "grep hüll testfile" #t) (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) (0 . "Verhüllung\n")
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") (shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n")
Marc Feeley feeley@iro.umontreal.ca writes:
Ha ha! That’s a funny issue caused by an optimization of the pretty printer. Now fixed.
Marc
On Jun 5, 2015, at 3:57 PM, Atticus atticus0@posteo.org wrote:
Some observations about the escaping of characters with the 8d2c2e6 commit:
$ gsi -:tU
;;;;;;;;
(string->list "über") ;; -> (#\xfc #\b #\e #\r) ;; expected (#\ü #\b #\e #\r)
(list->string (string->list "über")) ;; -> "über" ;; ok
(map list->string (list (string->list "über"))) ;; -> ("\374ber") ;; expected ("über")
;;;;;;;;
I expected (string->list "über") to return (#\ü #\b #\e #\r) and that list->string does not behave different when used with map.
When setting readtable-max-unescaped-char-set in the gambcini file string->list and list->string work as expected.
Marc Feeley feeley@iro.umontreal.ca writes:
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 2:43 AM, Atticus atticus0@posteo.org wrote:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think.
Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is
- ASCII then characters whose code are >= 128 would use escapes like \374
- ISO-8859-1 then characters whose code are >= 256 would use escapes
- UCS-2 then characters whose code are >= 65536 would use escapes
- in all other cases escapes would not be used because all Unicode characters can be encoded
By the way this would also affect the external representation of symbols, such as 'über .
Marc
> On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote: > > Hello, > > First and formost thanks for developing and maintaining gambit. > > As a beginner I'm a litte confused about the handling of the character > encoding in the repl. > > $ gsi -:d,t8,f8 > $> "ü" > "\374" > > $> (string-ref "ü" 0) > #\xfc > > $> (string->list "über") > (#\xfc #\b #\e #\r) > > $> (with-output-to-file "test2" (lambda () (##write-string "ü") > $> ,q > > $ cat test2 > ü > > $ gsi -:d,t8,f8 > $> (with-input-from-file "test2" (lambda () (read-char))) > #\xfc > > > With racket: > > $ racket > $>"ü" > "ü" > > $> (with-input-from-file "test2" (lambda () (read-char))) > #\ü > > $> (string->list "über") > '(#\ü #\b #\e #\r) > > > Why does gambit return "\374" instead of "ü" in the repl when evaluating > the string "ü"? I'm curious because in racket the repl returns the > evaluated string as "ü"", also read-char in racket returns #\ü instead > of #\xfc. > > Looking at the documentation I could change the behaviour with: > ;;;; > (output-port-readtable-set! > (repl-output-port) > (readtable-max-unescaped-char-set > (output-port-readtable (repl-output-port)) > #\U0010ffff)) > ;;;; > > Are there any advantages for having this default readtable setting in > gambit? As a beginner I think it's confusing. > > _______________________________________________ > Gambit-list mailing list > Gambit-list@iro.umontreal.ca > https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
With gsi -:tU
(string->list (cdr (shell-command "grep Ver testfile" #t)) -> (#\V #\e #\r #\h #\Ã #\¼ #\l #\l #\u #\n #\g #\newline)
Marc Feeley feeley@iro.umontreal.ca writes:
It is not clear what is and isn’t a character in the output of your “shell-command”. Could you do a string->list on the cdr of the result of shell-command?
Marc
On Jun 16, 2015, at 11:53 AM, Atticus atticus0@posteo.org wrote:
I encountered another problem.
'shell-command' in gambit release 4.7.6 version and 6b353f does not recognize the character encoding when used with the optional argument:
$ file testfile testfile: UTF-8 Unicode text
$ cat testfile Verhüllung
$ grep hüll testfile Verhüllung
Gambit 6b353f:
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n") ;; expected (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #f) -> Verhüllung -> 0 ;; ok
$ gsi -:tA (shell-command "grep hüll testfile" #t) -> (256 . "")
Now with readtable-max-unescaped-char-set set to #\U0010ffff:
$ gsi -:tA (shell-command "grep hüll testfile" #t) (0 . "Verhüllung\n")
(shell-command "grep Ver testfile" #t) (0 . "Verhüllung\n")
$ gsi -:tU (shell-command "grep hüll testfile" #t) -> (256 . "") (shell-command "grep Ver testfile" #t) -> (0 . "Verhüllung\n")
Marc Feeley feeley@iro.umontreal.ca writes:
Ha ha! That’s a funny issue caused by an optimization of the pretty printer. Now fixed.
Marc
On Jun 5, 2015, at 3:57 PM, Atticus atticus0@posteo.org wrote:
Some observations about the escaping of characters with the 8d2c2e6 commit:
$ gsi -:tU
;;;;;;;;
(string->list "über") ;; -> (#\xfc #\b #\e #\r) ;; expected (#\ü #\b #\e #\r)
(list->string (string->list "über")) ;; -> "über" ;; ok
(map list->string (list (string->list "über"))) ;; -> ("\374ber") ;; expected ("über")
;;;;;;;;
I expected (string->list "über") to return (#\ü #\b #\e #\r) and that list->string does not behave different when used with map.
When setting readtable-max-unescaped-char-set in the gambcini file string->list and list->string work as expected.
Marc Feeley feeley@iro.umontreal.ca writes:
I have pushed a change which allows #f as the setting of the max-unescaped-char of a readtable, and it is now the default instead of (integer->char 127). When max-unescaped-char is #f the printer will take into account the character encoding of the output port and use escapes when the character can’t be encoded. I have also changed the default character encoding of the terminal, files, etc to use ASCII. This setting is actually equivalent to ISO-8859-1 (latin-1) at the lowest level of the runtime system, i.e. chars are encoded using 8 bits and the top bit is significant, but the printer will use escapes when the character code is above 127 (because ASCII is a 7 bit code).
This means that when no runtime options are passed to gsi the system will escape characters exactly as before. However, with
gsi -:tU (or gsi -:t8)
the system will not escape characters whose code is > 127.
For the users prefering this behavior, including you I assume, it can be made the default by setting the GAMBCOPT environment variable like this in your .profile:
GAMBCOPT=tU
This is better than changing the readtable in the gambcini file because it will apply to all Scheme programs compiled with Gambit, not just the interpreter, and it can be overriden with an explicit -:tA when starting gsi if you want to escape non-ASCII characters in a specific situation.
Marc
On May 30, 2015, at 2:43 AM, Atticus atticus0@posteo.org wrote:
> By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc).
Very interesting, thank you.
> Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is > > - ASCII then characters whose code are >= 128 would use escapes like \374 > - ISO-8859-1 then characters whose code are >= 256 would use escapes > - UCS-2 then characters whose code are >= 65536 would use escapes > - in all other cases escapes would not be used because all Unicode characters can be encoded
Imho yes. But to be clear, let's say the external representation of scheme strings depends now on the character encoding of the output port and I have the above explained situation, I have an utf-8 port and need to print to ASCII only, then I can still do that by changing the output port to ASCII manually before printing?
Marc Feeley feeley@iro.umontreal.ca writes:
> By default the “write”, “pretty-print”, etc procedures (including the REPL’s printer) assume that the output port can only display ASCII, so it encodes non-ASCII characters as sequences such as \374 or #xfc (which are composed of ASCII characters). This allows Scheme strings that are printed to be processed by external tools which only support ASCII. For example, if you pretty-print to a file a function that contains the string "ü" you will end up with a file containing the sequence of 6 characters "\374" which any editor can edit. In other words, it makes minimal requirements on the features of external tools (not just editors, but email clients if you want to email some Scheme code, shell utilities, C compilers, other Scheme systems, etc). > > Note that this behavior was chosen a long time ago when UTF-8 was not widespread. I’m open to changing this behavior, but I’d like to know what other people think. > > Should the external representation of strings depend on the character encoding of the output port? In other words, if the character encoding of the byte output port is > > - ASCII then characters whose code are >= 128 would use escapes like \374 > - ISO-8859-1 then characters whose code are >= 256 would use escapes > - UCS-2 then characters whose code are >= 65536 would use escapes > - in all other cases escapes would not be used because all Unicode characters can be encoded > > By the way this would also affect the external representation of symbols, such as 'über . > > Marc > > >> On May 29, 2015, at 2:12 PM, Atticus atticus0@posteo.org wrote: >> >> Hello, >> >> First and formost thanks for developing and maintaining gambit. >> >> As a beginner I'm a litte confused about the handling of the character >> encoding in the repl. >> >> $ gsi -:d,t8,f8 >> $> "ü" >> "\374" >> >> $> (string-ref "ü" 0) >> #\xfc >> >> $> (string->list "über") >> (#\xfc #\b #\e #\r) >> >> $> (with-output-to-file "test2" (lambda () (##write-string "ü") >> $> ,q >> >> $ cat test2 >> ü >> >> $ gsi -:d,t8,f8 >> $> (with-input-from-file "test2" (lambda () (read-char))) >> #\xfc >> >> >> With racket: >> >> $ racket >> $>"ü" >> "ü" >> >> $> (with-input-from-file "test2" (lambda () (read-char))) >> #\ü >> >> $> (string->list "über") >> '(#\ü #\b #\e #\r) >> >> >> Why does gambit return "\374" instead of "ü" in the repl when evaluating >> the string "ü"? I'm curious because in racket the repl returns the >> evaluated string as "ü"", also read-char in racket returns #\ü instead >> of #\xfc. >> >> Looking at the documentation I could change the behaviour with: >> ;;;; >> (output-port-readtable-set! >> (repl-output-port) >> (readtable-max-unescaped-char-set >> (output-port-readtable (repl-output-port)) >> #\U0010ffff)) >> ;;;; >> >> Are there any advantages for having this default readtable setting in >> gambit? As a beginner I think it's confusing. >> >> _______________________________________________ >> Gambit-list mailing list >> Gambit-list@iro.umontreal.ca >> https://webmail.iro.umontreal.ca/mailman/listinfo/gambit-list
I am more interested in the -:tA case. From what I can see, your ü is being passed as 2 bytes, and is returned as 2 bytes (in this case, 2 characters), and it is the console that is interpreting this 2 byte sequence and showing a ü on the terminal. So it seems to be correct, but in fact it is not. I just want to make sure I understand this before trying to find a solution to your problem. This confusion seems to be exactly what Mikael More was trying to avoid by forcing escapes on any characters <= 127.
Marc
On Jun 19, 2015, at 1:49 PM, Atticus atticus0@posteo.org wrote:
With gsi -:tU
(string->list (cdr (shell-command "grep Ver testfile" #t)) -> (#\V #\e #\r #\h #\Ã #\¼ #\l #\l #\u #\n #\g #\newline)
gsi -:tA
(string->list (cdr (shell-command "grep Ver testfile" #t))) -> (#\V #\e #\r #\h #\xc3 #\xbc #\l #\l #\u #\n #\g #\newline)
gsi -:tA (with readtable-max-unescaped-char-set set in gambcini)
(string->list (cdr (shell-command "grep Ver testfile" #t))) -> (#\V #\e #\r #\h #\Ã #\¼ #\l #\l #\u #\n #\g #\newline)
Same results for the 4.7.6 release version.
Marc Feeley feeley@iro.umontreal.ca writes:
I am more interested in the -:tA case. From what I can see, your ü is being passed as 2 bytes, and is returned as 2 bytes (in this case, 2 characters), and it is the console that is interpreting this 2 byte sequence and showing a ü on the terminal. So it seems to be correct, but in fact it is not. I just want to make sure I understand this before trying to find a solution to your problem. This confusion seems to be exactly what Mikael More was trying to avoid by forcing escapes on any characters <= 127.
Marc
On Jun 19, 2015, at 1:49 PM, Atticus atticus0@posteo.org wrote:
With gsi -:tU
(string->list (cdr (shell-command "grep Ver testfile" #t)) -> (#\V #\e #\r #\h #\Ã #\¼ #\l #\l #\u #\n #\g #\newline)
On Jun 19, 2015, at 10:16 PM, Marc Feeley feeley@iro.umontreal.ca wrote:
I am more interested in the -:tA case. From what I can see, your ü is being passed as 2 bytes, and is returned as 2 bytes (in this case, 2 characters), and it is the console that is interpreting this 2 byte sequence and showing a ü on the terminal. So it seems to be correct, but in fact it is not. I just want to make sure I understand this before trying to find a solution to your problem. This confusion seems to be exactly what Mikael More was trying to avoid by forcing escapes on any characters <= 127.
127 of course…
Marc