[gambit-list] Character encoding and the repl

Adam adam.mlmb at gmail.com
Fri May 29 15:33:23 EDT 2015


2015-05-30 0:38 GMT+05:30 Marc Feeley <feeley at iro.umontreal.ca>:

> By default the “write”, “pretty-print”, etc procedures (including the
> REPL’s printer) assume that the output port can only display ASCII, so it
> encodes non-ASCII characters as sequences such as \374 or #xfc (which are
> composed of ASCII characters).  This allows Scheme strings that are printed
> to be processed by external tools which only support ASCII.  For example,
> if you pretty-print to a file a function that contains the string "ü" you
> will end up with a file containing the sequence of 6 characters "\374"
> which any editor can edit. In other words, it makes minimal requirements on
> the features of external tools (not just editors, but email clients if you
> want to email some Scheme code, shell utilities, C compilers, other Scheme
> systems, etc).
>
> Note that this behavior was chosen a long time ago when UTF-8 was not
> widespread.  I’m open to changing this behavior, but I’d like to know what
> other people think.
>
> Should the external representation of strings depend on the character
> encoding of the output port?  In other words, if the character encoding of
> the byte output port is
>
> - ASCII then characters whose code are >= 128 would use escapes like \374
> - ISO-8859-1 then characters whose code are >= 256 would use escapes
> - UCS-2 then characters whose code are >= 65536 would use escapes
> - in all other cases escapes would not be used because all Unicode
> characters can be encoded
>
> By the way this would also affect the external representation of symbols,
> such as 'über .
>


Symbols should under ordinary conditions not contain chars >= 128 as
standard software practice is to implement all identifiers in English, or
englishify them at the level of alphabet use e.g. 'uber ;

If anybody with a non-English-alphabet wants to use Gambti for instance for
conveniency in educational purposes, then they can reconfigure themselves.


The amount of systems that are not UTF8-compliant is unlimited, and you
never know when data copied/pasted etc. would pass through that, so keeping
the default setting as sparse as possible makes enormous sense. For
instance;
 * Protocols: the SMTP and HTTP protcol is not UTF-8 by default, at least
with not-too-unrecent browsers. I guess the same applies with,
 * Filesystems (FAT),
 * Various text editors I guess - at least VIM last I checked - and
 * Various other tools.
 * Fonts: You don't even know what characters beyond 0..127 that fonts
implement, and absence of chars in the 128..256 range happens.


After having pondered extending the default even just from 127 to 255 by
looking at http://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout , I
think an extension would be a bad idea because of the above reasons.


The one who wants more than that for text output can just use the
non-escaping output routines e.g. print and display, that is really part of
minimum knowledge.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.iro.umontreal.ca/pipermail/gambit-list/attachments/20150530/d9c753e2/attachment.htm>


More information about the Gambit-list mailing list