Directive prologue members, escapes, and retroactive syntax errors
jwalden+es at MIT.EDU
Mon Oct 18 09:53:46 PDT 2010
On 10/16/2010 02:30 AM, Peter van der Zee wrote:
>> So you suggest the tokenizer should have two modes of operation,
>> one to get a token using strict mode parsing and one to get it
>> using normal parsing?
> Doesn't the tokenizer already have this mode? How else is it able to know
> whether it should parse an octal in strings? And which additional keywords are
> additionally restricted in strict mode? You have to explicitly switch cfg's
> because of the second clause in 184.108.40.206.
The parser always "tries" to parse as octal (is the character after '\' in [1-7] or is it 0 followed by [0-7], that is -- or whatever the precise definition is, memory hazy). If it recognizes a sequence can only be an acceptable sequence if it is octal, *then* it checks whether strict mode is in force. So there's no mental or interface overhead to strict mode syntax checks, nor is there runtime overhead unless you use bad constructs. Thus you don't need both TokenStream::getTokenNormal and TokenStream::getTokenStrict -- just a TokenStream::getToken method that checks "are we strict?" when it encounters a potentially problematic token. The check can be hidden under the covers. This is why an octal escape, and then subsequent *retroactive* enabling of strict mode, is so problematic. The octal check already happened, so if early octal is to be included in strict mode restrictions that determination must be stashed away somewhere for potential future checking.
I don't believe SpiderMonkey has implemented strict mode keyword restriction (yet, it's in the bug list), but the same technique should be applicable. (Moreso, even, because strict mode invocation can't make a previously seen keyword invalid, because you can't invoke strict mode in such a syntactic position, by the nature of the directive prologue definition.)
More information about the es5-discuss