Iterators, generators, finally, and scarce resources (Was: April 10 2014 Meeting Notes)

Mark S. Miller erights at
Wed Apr 30 07:06:33 PDT 2014

On Tue, Apr 29, 2014 at 4:32 AM, Kevin Smith <zenparsing at> wrote:

> I agree with pretty much everything Andy had to say, and would like to add
> a meta-perspective:
> We should be viewing this as a last-minute feature request.  The churn
> that this request has introduced is (in my opinion) exactly the kind of
> problem that the ES7 process is meant to address.  In fact, I would go so
> far as to say that requirements churn has been the number one problem
> during ES6 development.
> If the "iterator-as-resource-manager" feature is truly desirable (and I'm
> not even convinced that it is), then the appropriate action is to figure
> out a way to defer it until ES7.  It is simply too late to add this feature
> now.

Btw, I still agree that *if* a way can be found to defer this to ES7
without breaking anything, then or now, we should. Suggestions?

To date, none of the suggested means of deferrals work. The *only* reason
we're considering this for ES6 is we do not see any other viable choice.
Please provide us one.

> On Tue, Apr 29, 2014 at 3:40 AM, Andy Wingo <wingo at> wrote:
>> On Fri 25 Apr 2014 16:22, Domenic Denicola <domenic at>
>> writes:
>> >> (2) not well-motivated according to some participants of the
>> > discussion (e.g., it's not necessarily a good idea to rely on
>> > finally-blocks for scarce resource management in the first place, since
>> > they provide only weak guarantees either way).
>> >
>> > This to me is the most worrying point. I feel like Andy summed things
>> up well in this old thread:
>> >
>> >
>> >
>> > I do not feel that his arguments were ever rebutted.
>> So in the meeting notes the consensus was to send the issue to
>> "generator champions" -- brendan and david herman, and somehow I ended
>> up on CC.  We had some backs and forths but as it seems that TC39
>> members are choosing to discuss this issue here, I'll repost my initial
>> note here.  Please read the meeting notes for a description of Jafar's
>> use case.  I'm a bit grumpy that this is being brought up again, and
>> this late, and in multiple forums, but as it seems that people want to
>> talk about it again, that talking about it again is the thing to do...
>>                             *  *  *
>> If I may summarize Jafar's argument, it's that the iterator in a for-of
>> may hold a scarce resource, like a file descriptor, and because of that,
>> for-of should be able to release this scarce resource on an early exit
>> via "break".  The provisional consensus elaborates a method to do this.
>> Is this a fair summary?
>> I sympathise with Jafar's plight but I think that the current setup is
>> the best we can do.  The summary of my argument is this:
>>   (1) calling return() on iterators is rarely appropriate;
>>   (2) return() in generators is semantically weird; and
>>   (3) making for-of call return() on early exit is expensive at run-time.
>> I should note first that this situation is not limited to generators, so
>> the starting point mention of "finally" blocks is something of a
>> distraction.  To the extent that this issue applies to generators, it
>> also applies to other kinds of iterators.  Indeed I expect that in
>> practice most iterators in an ES6 program will be map, set, and array
>> iterators, which in practice will not be implemented with generators.
>> Incidentally I think that if TC39 decided to re-add this method, it
>> should be called close() instead, because it doesn't make sense to
>> "return" from a non-generator iterator.
>> == Calling return() on iterators is rarely appropriate
>> Again I do sympathise with the use case, but we should start with a
>> discussion of what is the common case.
>> If we knew that the @@iterator call in the for-of would return a fresh
>> iterator, then it would make more sense to provide some means for
>> closing on early exit.  Jafar argues that this is in fact the common
>> case, which sounds about right to me.
>> However, holding a scarce resource is also likely to be uncommon.  It
>> certainly doesn't come up in the browser, for example.  I think it's
>> reasonable in that rare case to require some thought on the part of the
>> user as to what scarce resources they have acquired, and to arrange to
>> release them as appropriate.
>> Granted, if you are a user of an iterator, you might not know that it
>> has a scarce resource.  So there are two cases here: one in which the
>> iterator was created by its consumer, and one in which the consumer is
>> decoupled from the producer.
>> The first case is the one Jafar gives in his notes:
>>   for (var line of openFile("foo.txt"))
>>     if (line == '-- mark --')
>>       break;
>> However in this case it is possible to arrange to close the iterator,
>> with a different interface:
>>   var file = openFile("foo.txt");
>>   try {
>>     for (var line of lines(file))
>>       if (line == '-- mark --')
>>         break;
>>   } finally {
>>     file.close();
>>   }
>> Among other possibilities.  Something like Python's "with" might be
>> appropriate here.  The point is that although in this case, calling
>> return() on the iterator may indeed be appropriate, the desired behavior
>> can still be implemented.
>> Note that there is nothing special about for-of or iterators in this
>> example; any abstraction that captures a scarce resource has to do the
>> same thing.  It is not that generators are unable to abstract over IO --
>> it is that they are unable to transparently abstract over scarce
>> resource acquisition.  No surprise there.
>> The other case is when you have an iterator consumer which is decoupled
>> from the code that created the iterator, as in:
>>   function (iterable) {
>>     ...
>>     for (var x of iterable) {
>>       if foo(x) break;
>>     }
>>     ...
>>   }
>> But it is precisely in this case when you would *not* want to close the
>> iterator, because you don't know its lifetime.
>> == return() in generators is semantically weird
>> I know the argument has already been made, but I would like to repeat my
>> point (2) from
>> namely that close() "complicates the mental model of what happens when
>> you yield."  It's really strange to consider a yield as not only an
>> expression that produces a value, or possibly a point at which an
>> exception could be thrown, but also a "return".  Bizarre.  It's a hazard
>> to reading generator functions.
>> Also, the insistence on a return() that doesn't run catch blocks seems
>> to me to be ill-placed.  I think it's telling that the counter-examples
>> are from Python, which has a different semantic model, as it has
>> finalization.  Implementing abstractions over scarce resources in JS is
>> going to necessarily involve different design patterns than those used
>> by Python.  For the given use-case, throw() is entirely sufficient.  If
>> you don't trust your generators to do the right thing on an exception,
>> you shouldn't be acquiring scarce resources!
>> Finally, the given use-case is incompletely specified; a loop can exit
>> prematurely through exceptions as well as through "break".  So really
>> what is proposed is a finally block in every for-of statement, which
>> brings me to my next point...
>> == Calling return() on early exit from for-of is expensive
>> Wrapping a try/finally around each for-of is going to be really
>> expensive in all engines right now.  I'm skeptical about our ability to
>> optimize this one away.  Avoiding try/catch around for-of was one reason
>> to move away from StopIteration, and it would be a pity to re-impose
>> this cost on every for-of because of what is, in the end, an uncommon
>> use case.  I think the expected result of doing this would be
>> performance lore to recommend using other iteration syntaxen instead of
>> for-of.
>> There's no perfect answer when it comes to abstractions over scarce
>> resources.  Given the constraints of what JS is, its finalization model,
>> its deployment in the browser, and its engines, for me the status quo is
>> the best we can do.  I know that for people that open file descriptors,
>> that's somewhat unsatisfying, but perhaps such a cross is what goes with
>> the crown of being a true Unix hacker ;)
>> Regards,
>> Andy
>> _______________________________________________
>> es-discuss mailing list
>> es-discuss at

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the es-discuss mailing list