Immediate closing of iterators

Brendan Eich brendan at mozilla.org
Thu Dec 14 23:51:58 PST 2006


On Dec 14, 2006, at 9:01 PM, Jeff Thompson wrote:

> This is a follow-up to a bugzilla discussion at:
> https://bugzilla.mozilla.org/show_bug.cgi?id=349326

Jeff, thanks for writing.  I'll try to add relevant detail below,  
since this question is complicated by a bug you hit when testing in  
the js shell built from SpiderMonkey sources, I think.  Plus, the  
whole topic is complicated, period.

> var was_closed = false;
> function gen() {
>         try {
>                 yield 1;
>         } finally {
>                 was_closed = true;
>         }
> }
>
> for (var i in gen())
>         break;
> print("was_closed=" + was_closed);
>
> Right now, this prints "was_closed=false" because when it breaks  
> out of the for loop,
> Javascript does not close the iterator, and does not execute the  
> finally clause to
> set was_closed true.

Only if you run the above in the js shell, which shows up a bug  
(https://bugzilla.mozilla.org/show_bug.cgi?id=363917).  I just fixed  
that bug in my build, and with this variation on your testcase:

var was_closed = false;
function gen() {
         try {
                 yield 1;
         } finally {
                 was_closed = true;
                 print("closed!");
         }
}

for (var i in gen())
         break;
print("was_closed=" + was_closed);

gc();  // <=== force a GC here

I get this output:

was_closed=false
global
closed!

The gc() call is needed, otherwise the shell does a final GC after  
its global object has become unreachable, and any generator scoped by  
an unreachable global object is not closed, which is intentional (see  
below for why).

This HTML version of your test:

<textarea id="t" rows="4"></textarea>
<script type="application/javascript;version=1.7">
var tarea = document.getElementById('t');
var print = function (s) { t.value += s + '\n'; }
var was_closed = false;
function gen() {
         try {
                 yield 1;
         } finally {
                 was_closed = true;
                 print("closed!");
         }
}

for (var i in gen())
         break;
print("was_closed=" + was_closed);

function toString() { return "global"; }
print(this);
</script>

works as expected (if not as desired), with no forced GC (Firefox  
runs a GC soon enough after page load that you can see the "closed!"  
text in the textarea appear, but off the page-load critical path).

This may seem cold comfort, since the finalization happens well after  
the loop terminates.

IOW, what's wanted is not a guarantee that close always runs the  
generator (a finally clause).  If ES4 did require that, trivial  
denial-of-service attacks exist; this is why JS1.7 doesn't close  
generators that are unreachable if their scope is unreachable.

What's wanted is that close run promptly *only* in the case of a for- 
in loop where no reference to the iterator-generator escapes to the  
heap.

This is a reasonable thing to want, and it's what https:// 
bugzilla.mozilla.org/show_bug.cgi?id=349326 requests.  The question  
is, should ES4 require such prompt finalization in the case where the  
for-in loop creates the iterator and it never escapes to the heap?

Here's another variation on your testcase:

var was_closed = false;
function gen() {
         try {
                 yield 1;
         } finally {
                 was_closed = true;
                 print("closed!");
         }
}

var i = gen().next();
print("was_closed=" + was_closed);

With the js shell bugfix, but without the explicit gc() call at the  
end, this too will fail to close the iterator returned by gen().  But  
there's no for-in loop here, so if you think this case shows a bug,  
then you are not just asking for for-in loops to close non-escaping  
generator-iterators -- you are asking all ES4 implementations to use  
reference counting or something equivalent, which promptly finalizes  
unreachable generator-iterators.  That's a harsher requirement for a  
standard to make on all implementations.

>   I want to argue that Javascript should close the iterator:
> * No guarantee is made that the 'finally' clause will ever be
>   executed, so if you need it to close a file, etc. it might never  
> happen.

You can't count on deferred scripted functions such as timeouts  
running in the browser.  The page may be unloaded and the timeout  
canceled.  The situation with close hooks (finally clauses in  
generators) is entirely analogous.

Also (and this is really an aside -- I'm not arguing about the for-in- 
loops-should-close-non-escaping-generator-iterators point):  
finalization should never be required to promptly free scarce  
resources that have been explicitly allocated by a program.  ECMA  
requires some kind of GC, but not promply finalizing GC.

Note that finalization is never canceled, but finally clauses in a  
scripted finalize hook (that's what a generator that yields from a  
try with a finally is) may have to be canceled, just as timeouts may  
be canceled.

> * This seems the opposite of how 'finally' works, which usually  
> means that you can
>   be *sure* it will execute, even if the 'try' block throws an  
> exception, etc.

Python actually does not guarantee that finally clauses in all  
generators run -- if the generator misbehaves by yielding while being  
closed, then an outer finally may not be run.  This is considered the  
best way to deal with a misbehaving generator.  The ES4 design avoids  
this by throwing an exception from yield during close, which runs  
finallys on the way out if uncaught.

Anyway, the js shell bug aside, the issue for es4-discuss is not  
whether finally must always run (it can't or DOS attacks are trivial  
using generators); it's whether for-in loops should promptly finalize  
non-escaping iterators.

> * Python 2.5 and C# both close the iterator and execute the  
> 'finally' clause (as one would expect)
> * In fact, before version 2.5, Python gives a compiler error for  
> this code:
>   >>> 'yield' not allowed in a 'try' block with a 'finally' clause
>   I guess this is because before 2.5, Python did not always close  
> the iterator when
>   leaving a for loop from an exception, etc., so they didn't let  
> you write code that
>   wouldn't work like you expect.

Right; we prototyped this but then tracked 2.5; later, we eliminated  
GeneratorExit (see the python-dev thread mentioned above).

> So, could the spec require Javascript to always close the iterator  
> when leaving a loop?

So long as it's a non-escaping generator-iterator created by the  
loop, then the spec could mandate that.  It requires some extra work  
by non-reference-counting implementations.  They need to keep track  
of such generator-iterators across nested loops in each live function  
or script activation, and close each generator-iterator as control  
exits its loop.

We'll talk about it in the current TG1 meeting, which finishes  
tomorrow (day 3).  More then, or here in this thread of TG1'ers prefer.

/be




More information about the Es4-discuss mailing list