[gambit-list] SRFI 14 and data structures in gambit/lib/gambit/char/char#.scm

John Cowan cowan at ccil.org
Sat May 20 03:34:57 EDT 2023


On Fri, May 19, 2023 at 6:27 PM Bradley Lucier <lucier at math.purdue.edu>
wrote:

Unicode has stretches of code-points that are invalid, doesn't it?  In
> which case complement would always be complement against the full valid
> Unicode set (which isn't a single contiguous range).

The ranges of valid Unicode characters are #x0000 to #xD7FF and #xE000 to
#x10FFFF.  As far as char-set-contains? is concerned, it doesn't matter if
the non-range #xD7FF to #xDFFF are included or excluded, as there is no way
to create a character whose char->int value is in this range.  However,
when it comes to enumeration (the cursor functions, -fold, -for-each, -map,
->list, ->string) you do need to special-case this non-range to avoid
traversing it.  Alternatively, the special-casing can be done in the set
operations, which is probably better because they are less used than the
enumerators.


> It appears to me that it would suffice to use John's representation for
> all character sets, but write
>
> char-set-contains?
>
> in terms of
>
> char-alphabetic?
> char-lower-case?
> char-numeric?
> char-upper-case?
> char-whitespace?
>
> for some of the standard character sets.
>

The difficulty for a portable implementation of SRFI 14 is that the above
functions often cover only ASCII or only Latin-1 in the native
implementation, or only cover a subset of the assigned characters
corresponding to an older version of Unicode (after the long-obsolete
version 1.1, the set of all assigned characters only grows with the
version, it never shrinks).  If you know that they are up-to-date, that's a
reasonable approach, but if they are incomplete, it's better to do it the
other way round: reimplement char-alphabetic? in terms of
char-set-contains? and char-set-letter?, etc.

Note that IMO this statement in the definition of ucs-range->charset

   - If the requested range includes unassigned UCS values, these are
   silently ignored (the current UCS specification has "holes" in the space of
   assigned codes).

should be disregarded, because it makes these functions unnecessarily
dependent on a specific version of Unicode.  However, attempts to specify
the surrogate codes #\xD800 to #\xDFFF should indeed be excluded, as they
cannot ever be assigned.

>
> Maybe a few more codes could be added for P*, S*, M* and N* above 199 in
> your encoding in char#.scm to have fast membership testing for all the
> "standard" classes.'


That makes sense to me.

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.iro.umontreal.ca/pipermail/gambit-list/attachments/20230520/5f35a0fe/attachment.htm>


More information about the Gambit-list mailing list