yield syntax

[David-Sarah Hopwood]

Brendan Eich wrote:
> This may be breaking a butterfly on a wheel, but I am game if it
> improves the state of the strawman proposals.
> 
> On May 15, 2009, at 7:16 PM, David-Sarah Hopwood wrote:
> 
>> My point was that the example of 'yield (foo)' (that is, yield as a
>> prefix operator applied to the expression '(foo)') shows that the prefix
>> operator syntax cannot possibly be easier to specify than the function
>> call syntax -- contrary to what you appeared to be arguing above.
> 
> Analogous to direct vs. indirect eval in ES5 (15.1.2.1.1), there is no
> purely syntactic specification for what Neil proposes. A runtime check
> is required. So I don't see why you are focusing only on syntax here.

Maybe I misunderstood what Neil was proposing, but I took him to mean
something equivalent to:

 - parse using essentially the current grammar.
 - if the text of the CallExpression in 'CallExpression Arguments',
   or of the MemberExpression in 'MemberExpression Arguments',
   is "yield" (excluding comments and whitespace), then the call
   is a yield-expression.

This is a purely syntactic specification.

It does have some quirks: if you define a function called 'yield' and
try to call it, then the call will still be treated as a yield-expression
rather than a function call. For that reason I'm going cold on the idea.
I would prefer something entirely unambiguous, like

  @yield expr

or an opt-in that makes 'yield' (and 'let') reserved everywhere.
Either would make parentheses around a yield expression unnecessary,
or at least not necessary in as many cases. ('yield' would behave
syntactically like 'typeof', perhaps with lower precedence.)

[...]
>> Show me an unambiguous grammar for the prefix yield operator, and
>> then I'll concede the point :-)
> 
> Python 2.5 has essentially the same issues,

Python does not attempt to contextually reserve 'yield'.
It uses an opt-in that makes 'yield' a keyword that is reserved
everywhere. From PEP 255:

#  A new statement is introduced:
#
#      yield_stmt:    "yield" expression_list
#
#  "yield" is a new keyword, so a future statement[8] is needed to phase
#  this in:  in the initial release, a module desiring to use generators
#  must include the line
#
#      from __future__ import generators
#
#  near the top (see PEP 236[8]) for details).  Modules using the
#  identifier "yield" without a future statement will trigger warnings.
#  In the following release, yield will be a language keyword and the
#  future statement will no longer be needed.

> and it has a formal grammar
> (several, the one CPython uses and at least one other at antlr.org).
> From CPython's Grammar/Grammar EBNF:
> 
> $ grep yield Grammar
> expr_stmt: testlist (augassign (yield_expr|testlist) |
>                      ('=' (yield_expr|testlist))*)
> flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt |
> yield_stmt
> yield_stmt: yield_expr
> atom: ('(' [yield_expr|testlist_gexp] ')' |
> yield_expr: 'yield' [testlist]

That would be ambiguous, if (hypothetically) 'yield' were not a
keyword in Python. For example

  (yield (foo))

would match both a parenthesised <yield_expr>, and a parenthesised
<NAME> 'yield' followed by a trailer giving function arguments.
(This involves the <power>, <trailer>, <arglist> and <argument>
productions that weren't found by your grep.)

Similarly for (yield - 1) or (yield + 1) or (yield /re/g) in Harmony.
If these are treated as being yield-expressions, then that is
not compatible with any ES3/5 code using 'yield' as a variable
name, but if they are treated as having their ES5 meaning, then
there are a bunch of ugly special cases for what token can
validly begin the expression after 'yield'.

> What we are discussing is almost trivial in terms of usability, but the
> difference is big enough to care about, IMHO.

Unambiguous grammar first, please.

-- 
David-Sarah Hopwood ⚥