yield* desugaring

Allen Wirfs-Brock allen at wirfs-brock.com
Mon May 13 08:58:03 PDT 2013

On May 13, 2013, at 1:22 AM, Andy Wingo wrote:

> Hi,
> On Sun 12 May 2013 21:29, Allen Wirfs-Brock <allen at wirfs-brock.com> writes:
>> 1) I specified yield* such that it will work with any iterator, not
>> just generators.  To me, this seems essential.  Otherwise client code
>> is sensitive to other peoples implementation decision (that are
>> subject to change) regarding whether to use a generator or an object
>> based iterator.  This was easy to accomplish and only requires a
>> one-time behavioral check to determine whether "next" or "send" should
>> be used to retrieve values from the delegated iterator and an behavior
>> guard on invoking "throw" on the delegated iterator.
> Are you checking structurally or are you checking the internal "brand"?
> I can think of situations where you would want to decorate a generator
> iterator, producing an object with the same interface but not actually a
> generator iterator.  Perhaps one could decorate with another generator,
> though.  Relatedly:

Structurally, of course. ES is a dynamic language.  yield* has no dependencies upon branded implementation details of the object it is delegating to. Even without the very minor accommodations I made for the differences between the Iterator and generator interfaces users, with the Wiki desugaring users could accomplishing the same thing simply by providing a "send" method (and "throw" for completeness") on the iterator they are delegating to.  They might even write themselves an ItrAsGen utility function.  

BTW, this is only an issue because Iterator and generators interfaces are more different then they need to be.  More latter...

>> 2) yield* invokes the @@iterator method on its expression to obtain the
>> iterator. This means you can say things like:
>>    yield * [2,4,6,8,10]; //individually yield positive integers <= 10.
> Is it wise to do this?

Every other place in the language (syntax and built-in functions) where we expect Iterables we transparently accept either iterator or generator objects.  Why should yield* be any different.  The generator wiki proposal even describes yield* as similar to to a for loop over the generator.  This is how regular uses are going to think about it, so yes it is wise.

> It would also be possible to define
>  function* iterate(iterable) { for (let x of iterable) yield x; }
> and that would provide a uniform interface to the RHS of a yield*, and a
> natural point at which a throw() to a generator suspended in a yield*
> would raise an exception.

In fact it "is" possible to define this and it is one way to implement the ItrAsGen I mention above. Or you could just write:
   yield * (for (Iet x of iterable) x);

But why force uses of yield * to defensive program in that way and why force the runtime overhead of a second level of generator iteration when (those context swaps must cost something) if in fact a simple object based iterator is being used.

>> 3) yield* yields the nextResult object produced by the inner
>> iterator. No unwrapping/rewrapping required.
> Does this ensure that the result is an object and has "value" and "done"
> properties?

Of course not, that wouldn't that would be an unnecessary extra runtime type check.  ES is a dynamic language and there is no guarantee that an iterator actually provides such an object.  However, if your nextResult object doesn't have or inherit those properties their value is a falsy undefined, so "done" is false.  BTW, this is another concern I have about the current nextResult object design.  I would prefer that a missing "done" property mean the same thing as done: true.

>> 4) I haven't (yet) provided a "close" method for generators.  I still
>> think we should.
> Let me try to summarize some thoughts on close().  I'll start from one
> of your use cases.
>>      b) Any time user code is manually draining a known generator that
>> it opened and decides that it is now done with the generator. They
>> really should close it.  Of course, they may not, but regardless they
>> should be provided with a means to do so and it should be encouraged as
>> a best practice.
> I think the question to ask is, why do you think this is a good
> recommendation?  It can't be for general resource cleanup issues,
> because otherwise iterators would also have a close method.  So I am
> inclined to think that it is because you see "finally" in the source
> code, and you treat that as a contract with the user that a finally
> block actually does run for its effects.

It's all about "finally" which up to now has been a strong guarantee. 
> But that's precisely what we can't guarantee: unlike function
> activations, the dynamic extent of a generator activation is unlimited.
> We don't have finalizers, so we can't ensure that a finally block runs.
> And once we allow for that possibility, it seems to me that close() is
> not only less useful, but that by making a kind of promise that we can't
> keep, it can be harmful.

The most common uses of generators (for-of, and similar contexts ) could fulfill the "finally" guarantees, even in the presence of early exits. Manual draining of generators ( case b) is the exceptional case. Providing "close" still doesn't guarantee the integrity of "finally" for that case but it does provide the necessary mechanism for building generator consuming abstractions that can make that guarantee. 

****** most important point:

However, there is another alternative to close that addresses the finally issue.  I can make it statically illegal for a "yield" statement to be nested within a block that is protected by a "finally".  This preserve the integrity of "finally" which is my only concern.  It also address the for-of early exit issues. This restriction is trivial to specify (and should be only slightly less trivial to implement). Since I don't currently have "close" in the spec.  I probably should have done this anyway.  

> close() also complicates a user's mental model of what happens when they
> see a "yield".  I see "yield x" and I think, OK, this suspends
> computation.  If it's in a position to yield a value, that tells me that
> it might also produce a value.  That it could throw an exception is less
> apparent, so you have to remember that.  But then you also have to
> remember that it might be like a "return"!  It's that second invisible
> behavior that tips my mental balance.

Note it really isn't like return from the user's perspecitve, it is more like a "break" out of the generator.  That's a better way to think about it.

> Of course a close() doesn't actually force a generator activation to
> finish; there is the possibility of exceptions or further yields in
> finally blocks.  In this case Python will re-queue the generator for
> closing, so you do finally run all the finallies -- but again, we don't
> guarantee that, so add that to the mental model of what "close" does...
> close() also adds restrictions on the use of generator objects.  For
> example it's very common to use a loop variable after a loop:
>   for (i = 0; i < N; i++) {
>     ...
>     if (foo) break;
>   }
>   ...
>   // here we use i   
> One can imagine situations in which it would be nice to use a generator
> object after a break, for example in lazy streams:
>  function* fib() {
>    var x = 0, y = 1;
>    yield x;
>    yield y;
>    while (1) {
>      let z = x + y;
>      yield x + y;
>      x = y;
>      y = z;
>    }
>  }
>  function for_each_n(f, iter, n) {
>    if (n) {
>      for (let x of iter) {
>        f(x);
>        if (--n == 0)
>          break;
>      }
>    }
>  }
>  var iter = fib();
>  for_each_n(x => console.log(x), iter, 10);  // first 10
>  for_each_n(x => console.log(x), iter, 10);  // next 10
> In summary my problems with close() are these:
> (1) It attempts to provide a reliable finally for unlimited-extent
>     activations, when we can't do that.
> (2) It complicates the mental model of what happens when you yield.
> (3) It makes common sugar like for-of inappropriate for some uses of
>     generator objects.

I think the static occurrence of "yield" within the control of a "finally" is the simplest solution.  Thanks for helping me understand that.


> WDYT?  Not to distract you too much from the new draft, of course :)
> Cheers,
> Andy

More information about the es-discuss mailing list