New full Unicode for ES6 idea
Brendan Eich
brendan at mozilla.com
Sun Feb 19 00:33:49 PST 2012
Once more unto the breach, dear friends!
ES1 dates from when Unicode fit in 16 bits, and in those days, nickels
had pictures of bumblebees on 'em ("Gimme five bees for a quarter",
you'd say ;-).
Clearly that was a while ago. These days, we would like full 21-bit
Unicode character support in JS. Some (mranney at Voxer) contend that it
is a requirement.
Full 21-bit Unicode support means all of:
* indexing by characters, not uint16 storage units;
* counting length as one greater than the last index; and
* supporting escapes with (up to) six hexadecimal digits.
ES4 saw bold proposals including Lars Hansen's, to allow implementations
to change string indexing and length incompatibly, and let Darwin sort
it out. I recall that was when we agreed to support "\u{XXXXXX}" as an
extension for spelling non-BMP characters.
Allen's strawman from last year,
http://wiki.ecmascript.org/doku.php?id=strawman:support_full_unicode_in_strings,
proposed a brute-force change to support full Unicode (albeit with too
many hex digits allowed in "\u{...}"), observing that "There are very
few places where the ECMAScript specification has actual dependencies
upon the size of individual characters so the compatibility impact of
supporting full Unicode is quite small." But two problems remained:
P1. As Allen wrote, "There is a larger impact on actual
implementations", and no implementors that I can recall were satisfied
that the cost was acceptable. It might be, we just didn't know, and
there are enough signs of high cost to create this concern.
P2. The change is not backward compatible. In JS today, one read a
string s from somewhere and hard-code, e.g., s.indexOf("0xd800" to find
part of a surrogate pair, then advance to the next-indexed uint16 unit
and read the other half, then combine to compute some result. Such usage
would break.
Example from Allen:
var c = "😁" // where the single character between the quotes is the
Unicode character U+1f638
c.length == 2;
c === "\ud83d\ude38"; //the two character UTF-16 encoding of 0x1f683
c.charCodeAt(0) == 0xd83d;
c.charCodeAt(1) == 0xd338;
(Allen points out how browsers, node.js, and other environments blindly
handle UTF-8 or whatever incoming format recoding to UTF-16 upstream of
the JS engine, so the above actually works without any spec-language in
ECMA-262 saying it should.)
So based on a recent twitter/github exchange, gist recorded at
https://gist.github.com/1850768, I would like to propose a variation on
Allen's proposal that resolves both of these problems. Here are
resolutions in reverse order:
R2. No incompatible change without opt-in. If you hardcode as in Allen's
example, don't opt in without changing your index, length, and
char/code-at assumptions.
Such opt-in cannot be a pragma since those have lexical scope and affect
code, not the heap where strings and String.prototype methods live.
We also wish to avoid exposing a "full Unicode" representation type and
duplicated suite of the String static and prototype methods, as Java
did. (We may well want UTF-N transcoding helpers; we certainly want
ByteArray <-> UTF-8 transcoding APIs.)
True, R2 implies there are two string primitive representations at most,
or more likely "1.x" for some fraction .x. Say, a flag bit in the string
header to distinguish JS's uint16-based indexing ("UCS-2") from
non-O(1)-indexing UTF-16. Lots of non-observable implementation options
here.
Instead of any such *big* new observables, I propose a so-called "Big
Red [opt-in] Switch" (BRS) on the side of a unit of VM isolation:
specifically the global object.
Why the global object? Because for many VMs, each global has its own
heap or sub-heap ("compartment"), and all references outside that heap
are to local proxies that copy from, or in the case of immutable data,
reference the remote heap. Also because inter-compartment traffic is (we
conjecture) infrequent enough to tolerate the proxy/copy overhead.
For strings and String objects, such proxies would consult the remote
heap's BRS setting and transcode indexed access, and .length gets,
accordingly. It doesn't matter if the BRS is in the global or its String
constructor or String.prototype, as the latter are unforgeably linked to
the global.
This means a script intent on comparing strings from two globals with
different BRS settings could indeed tell that one discloses non-BMP
char/codes, e.g. charCodeAt return values >= 0x10000. This is the
*small* new observable I claim we can live with, because someone opted
into it at least in one of the related global objects.
Note that implementations such as Node.js can pre-set the BRS to "full
Unicode" at startup. Embeddings that fully isolate each global and its
reachable objects and strings pay no string-proxy or -copy overhead.
R1. To keep compatibility with DOM APIs, the DOM glue used to mediate
calls from JS to (typically) C++ would have to proxy or copy any strings
containing non-BMP characters. Strings with only BMP characters would
work as today.
Note that we are dealing only in spec observables here. It doesn't
matter whether the JS engine uses UTF-8 and the DOM UCS-2 (in which case
there is already a transcoding penalty; IIRC WebKit libxml and libxslt
use UTF-8 and so must transcode to interface with WebKit's DOM). The
only issue at this boundary, I believe, is how indexing and .length work.
Ok, there you have it: resolutions for both problems that killed the
last assault on Castle '90s-JS.
Implementations that use uint16 vectors as the character data
representation type for both "UCS-2" and "UTF-16" string variants would
probably want another flag bit per string header indicating whether, for
the UTF-16 case, the string indeed contained any non-BMP characters. If
not, no proxy/copy needed.
Such implementations probably would benefit from string (primitive
value) proxies not just copies, since the underlying uint16 vector could
be shared by two different string headers with whatever metadata flag
bits, etc., are needed to disclose different length values, access
different methods from distinct globals' String.prototype objects, etc.
We could certainly also work with the W3C to revise the DOM to check the
BRS setting, if that is possible, to avoid this non-BMP-string
proxy/copy overhead.
How is the BRS configured? Again, not via a pragma, and not by
imperative state update inside the language (mutating hidden BRS state
at a given program point could leave strings created before mutation
observably different from those created after, unless the implementation
in effect scanned the local heap and wrapped or copied any
non-BMP-char-bearing ones creatd before).
The obvious way to express the BRS in HTML is a <meta> tag in document
<head>, but I don't want to get hung up on this point. I do welcome
expert guidance. Here is another W3C/WHATWG interaction point. For this
reason I'm cc'ing public-script-coord.
The upshot of this proposal is to get JS out of the '90s without a
mandatory breaking change. With simple-enough opt-in expressed at
coarse-enough boundaries so as not to impose high cost or unintended
string type confusion bugs, the complexity is mostly borne by
implementors, and at less than a 2x cost comparing string
implementations (I think -- demonstration required of course).
In particular, Node.js can get modern at startup, and perhaps engines
such as V8 as used in Node could even support compile-time (#ifdef)
configury by which to support only full Unicode.
Comments welcome.
/be
More information about the es-discuss
mailing list