UTF-16 vs UTF-32

[John Tamplin]

On Mon, May 16, 2011 at 8:42 PM, Shawn Steele <Shawn.Steele at microsoft.com>wrote:

> It's clear why we want to support the full Unicode range, but it's less
> clear to me why UTF-32 would be desirable internally.  (Sure, it'd be nice
> for conversion types).
>
> What UTF-32 has that UTF-16 doesn't is the ability to walk a string without
> accidentally chopping up a surrogate pair.  However, in practice, stepping
> over surrogates is pretty much the least of the problems with walking a
> string.  Combining characters and the like cause numerous typographical
> shapes/glyphs to be represented by more than one Unicode codepoint, even in
> UTF-32.  We don't see that in Latin so much, especially in NFC, but in some
> scripts most characters require multiple code points.
>
> In other words, if I'm trying to find "safe" places to break a string,
> append text, or many other operations, then UTF-16 is no more complicated
> than UTF-32, even when considering surrogates.
>
> UTF-32 would cause a huge amount of ambiguity though about what happens to
> all of those UTF-16 sequences that currently sort-of work even though they
> shouldn't really because ES is nominally UCS-2.
>

Personally, I think UTF16 is more prone to error than either UTF8 or UTF32
-- in UTF32 there is a one-to-one correspondence, while in UTF8 it is
obvious you have to deal with multi-byte encodings.  With UTF16, most
developers only run into BMP characters and just assume that there is a
one-to-one correspondence between chars and characters.  Then,  when their
code runs into non-BMP characters they run into problems like restricting
the size of a field to a number of chars and it is no longer long enough,
etc.  The problems arise infrequently, which means many developers assume
the problem doesn't exist.

-- 
John A. Tamplin
Software Engineer (GWT), Google
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.mozilla.org/pipermail/es-discuss/attachments/20110516/9ffdd016/attachment.html>