Day 2 meeting notes

[Brendan Eich]

On Jul 30, 2010, at 9:06 PM, felix wrote:

> it seems odd to me that if 'a' is an iterator, it will iterate over the iterator's value stream instead of the iterator's properties, unless you define the two to be identical, which would be strange.  eg, if you have an input stream iterator f, would f.hasOwnProperty('bacon') work or not?

Sorry, I misspoke -- without having specified keys fully (we'll get to that task on the wiki), let me try to make up for it by implementing some variations on keys using proxies, in TraceMonkey.

The object returned by keys, called "a" in your revision, is a proxy. Let's call it "p" to avoid confusion.

Let's say that the handler for proxy "p" has no traps emulating properties, only an iterate trap. The iterate trap is a derived trap (it is optional; if missing, the default enumerate behavior is supplied by the proxy support in the runtime). The for-in loop calls the iterate trap on "p", which returns an iterator object.

This iterator, call it "it", is an object with a 'next' property whose value is a function taking no arguments and returning the next value in the iteration. This iterator object "it" is not necessarily a proxy -- it could be, but it need not be.

Ok, so the proxy denoted by "p" returned by keys(o) has been subject to for-in by being on the right of 'in', so its iterate trap has been called, and now the for-in runtime has the returned iterator "it".

The for-in loop runtime therefore calls it.next() implicitly at the top of each iteration of the loop body. If it.next() throws StopIteration, the for-in loop terminates normally (any other exception thrown by the body terminates the loop and is of course propagated).

Here's a keys implementation in bleeding edge SpiderMonkey (http://hg.mozilla.org/tracemonkey):

js> function keys(o) {
    const handler = {
        iterate: function () { for (let i in o) yield i; }
    };
    return Proxy.create(handler);
}
js> let o = {a:1, b:2, c:3};
js> for (let k in keys(o))
    print(k);
a
b
c
js> 

(The strawman at http://wiki.ecmascript.org/doku.php?id=strawman:iterators provides a convenience Iterator.create function to make a proxy given just the handler iterate trap function. Using it, we could simplify keys to just

function keys(o) {
    return Iterator.create(function () { for (let i in o) yield i; });
}

but the full expansion above, using Proxy.create, is not much longer.)

What if you wanted a proxy whose handler uses other traps in addition to iterate to emulate enumerable properties in full? That is doable too, thanks to the power of proxies:

js> function keys2(o) {
    const handler = {
        iterate: function () { for (let i in o) yield i; },
        getPropertyDescriptor: function (name) {
            if (o.hasOwnProperty(name))
                return Object.getOwnPropertyDescriptor(o, name); 
            return undefined;
        },
        getOwnPropertyDescriptor: function (name) {
            return Object.getOwnPropertyDescriptor(o, name); 
        },
        defineProperty: function (name, pd) {
            return Object.defineProperty(o, name, pd);
        },
        getOwnPropertyNames: function () {
            return Object.getOwnPropertyNames(o);
        },
        delete: function (name) {
            return delete o[name];
        },
        /*enumerate:*/
        fix: function () {
            return Object.freeze(o);
        }
    };
    return Proxy.create(handler);
}
js> let o = {a:1, b:2, c:3};
js> let p = keys2(o);
js> for (let k in p)
    print(k);
a
b
c
js> print(p.a);
1
js> print(p.b);
2
js> print(p.c);
3
js> delete p.a;
true
js> for (let k in p)
    print(k);
b
c

There's a workaround in this handler's getPropertyDescriptor trap, because we have not yet implemented Object.getPropertyDescriptor (it was ratified only this week at the TC39 meeting). I took the easy way out and instead of walking o's prototype chain, being careful to shadow, I handle only "own" properties (let's assume that Object.prototype has not been monkey-patched ;-).

This example shows how delete p.a followed by another for-in loop over p reveals that 'a' has been deleted. The proxy is implementing native object semantics. Of course it is possible to implement inconsistent object semantics with proxies -- this is true of host objects in general (especially prior to ES5, but even in ES5 -- see ES5 8.6.2, the paragraphs toward the end -- and of course implementations may defy the standard, or simply be buggy or old).

Proxies are thus an in-language facility for implementing host objects. This is an evolutionary path toward cleaning up host objects in various DOM implementations. TC39 thinks this is important; it may save us from the worst of the browser DOMs (the IE DOM can claim that title, but all browsers have quirky host objects in my experience).

I also implemented the iterate trap, which is lazy, in preference to implementing the commented-out enumerate trap, which is eager and used only to return property names, not arbitrary values. But since this example implements native object semantics, and since o has few properties, it would have been better to implement enumerate as well as iterate -- or instead of iterate if there's no reason to be lazy.

Why two traps, enumerate and iterate? Because a proxy can be a prototype of a non-proxy object, and for-in goes up the prototype chain. Having started with the enumerable properties of the non-proxy, going up the prototype chain to the proxy must not loop over arbitrary values, e.g. Fibonacci numbers. The for-in loop started enumerating property names, so it should continue doing so even if, somewhere up the prototype chain, it hits a proxy.

So, having started by enumerating a non-proxy, the for-in runtime finds a proxy up the prototype chain and calls the enumerate trap, not the iterate trap. This ends the prototype chain traversal, since proxies handle prototype delegation fully in the Harmony proxies design (this has been discussed here, see first cited text and reply at https://mail.mozilla.org/pipermail/es-discuss/2009-December/010313.html).

Implementing both iterate and enumerate allows a proxy to be iterated directly by for-in, but enumerated if on a prototype chain. The enumerate trap would return all the keys eagerly, in an array, that the iterate trap returns one-by-one via the 'next' method of the iterator it constructs.

So what you want can be implemented, but it need not be if only an iterator is needed. Besides the use-cases for iterators that entail lazy computation of a stream of values, the "large proxy" problem raised in the proxies proposal (near the bottom, look for "Waldemar") motivated the development of the iterate trap.

A large proxy is one emulating an object with a great many properties (possibly an unbounded number). A large proxy would have both iterate as well as enumerate in its handler, so if it were directly referenced on the right of 'in' by a for-in loop, the large proxy's keys would not be eagerly stuffed in a large array via its enumerate trap -- instead the iterate trap would be called and provide an iterator returning the keys lazily.

(The large proxy's enumerate trap could indeed fail to complete, being subject to script runtime and memory quotas, if "large" really means "infinite"; such a large proxy should not be the prototype of non-proxy objects, clearly!)

But I expect, as is common (with necessary changes) in Python, that proxies with only an iterate trap will be sufficient for many useful iteration protocols. Such an iterable proxy could use no-ops for its fundamental traps, in order to appear to be an empty object. Or as in the first example, if only the iterate trap is supplied in the handler, the proxy is empty but attempts to get or test its properties (including 'toString') will throw.

We'll work out the remaining details, with help from es-discuss and even JS authors who experiment with Firefox 4 betas. I hope that this message helps by showing some of the options, as well as the rationale for extending proxies to support iteration in the way we've proposed.


> yes, I'd like generalized iterators like python, so this is not really an argument.

What's not an argument?

I'm not arguing that we'd like iterators under just about any syntax, details to be decided. I'm arguing specifically, for several reasons, that we would be better off allowing for-in to be customizable as in Python:

1. Allows developers, not just TC39, to improve for-in semantics in parallel and at scale.

2. Avoids enlarging the language with another for-in-like construct.

3. Re-uses Python brainprint

I don't have a crystal ball, so I can't prove that these benefits override possible drawbacks. I'm encouraged by our experience with JS1.7 and up, since 2006. Both SpiderMonkey and Rhino support iterators not as proxies, but with essentially the same semantics with respect to for-in. The Python precedent is also significant and relevant to many developers.

In all this time, no one has complained about for-in sometimes returning keys (or values for sequence types in Python, but keys for other types), other times returning custom iterator results. Note also that proxies as proposed for Harmony, and Python's unstratified meta-programming support, allow all sorts of fundamental opeations, not just for-in, to be customized.

But reason (1) is IMHO most compelling. Between my own mistakes made in haste in 1995 and TC39 TG1's failure in 1996 and 1997 with ES1 to clean up for-in and fully specify it, we have a legacy of confusion in a fundamental looping statement.

Meanwhile the mass of Ajax developers has coped by building some nice libraries on top of the language, including for-in, but of course expressing iteration functionally for want of a way to augment or reform the syntax.

From this history I conclude that giving developers the ability to reform for-in is likelier to bear good fruit than TC39 taking another rare and probably one-time-only, hard-coded-in-the-spec, stab at the problem, using novel yet less desirable syntax.

The Web is a very large evolving system. Giving developers evolutionary paths toward better ways of doing everyday tasks *in practice* works better than dictating "final answers" via de-jure standards bodies every few years or decades. As noted above, iteration may let authors reform for-in, even as the full power of proxies helps JS authors and DOM developers (certainly Mozilla's, and I believe also IE's) reform the DOM.

But we need these evolutionary pathways. They are necessarily meta-programming APIs, so JS developers can write programs determining the meaning of program elements including objects, properties, and for-in loops.

Without such meta-programming facilities, not much will change, and what does change will come in the form of highly constrained (over-constrained, in my experience), infrequently released, and costly products of design by committee.

/be