Two interoperable implementations rule

[Maciej Stachowiak]

On Jul 11, 2008, at 5:03 PM, Allen Wirfs-Brock wrote:

> A few thoughts on the general topic and various points that are been  
> raised:
>
> Overall, I think this is a good idea.  My personal opinion is that  
> standardization should follow proven utility, not the other way  
> around.  However, it's difficult to get meaningful use of proposed  
> web standards until there is ubiquitous implementation. If we, as a  
> community, can find a way to meaningfully work together to advance  
> web technology it will be a very good thing.
>
> Realistically, I think it has to be real browser-based  
> implementations. However, Maciej's at least one browser  
> implementation suggestion may be good enough. My perception is that  
> we have far more unresolved "will it break the web" arguments then  
> we do about the actual utility of features.  Let's just demonstrate  
> it on the web, one way or another.  BTW, I think this puts us  
> (Microsoft) at a disadvantage because we have self imposed  
> restrictions that currently make it much harder for us to publicly  
> demonstrate (or even discuss) browser changes or enhances than it  
> would be for any sort of standalone implementation we did. We'll  
> have to learn how to deal with it.

There are a few purposes for requiring interoperable implementations  
to advance the specification:

1) Demonstrate practical implementability - that the spec can be  
implemented correctly in a production-quality implementation without  
truly excessive implementation effort.
2) Show that the spec can be implemented without compromising Web  
compatibility, through implementor review and widespread testing.
3) Help flush out performance, security and usability issues through  
implementation and use of the resulting implementations.
4) Show that the spec language is unambiguous enough that multiple  
independent implementations are compatible.

A reference implementation, even if browser-hosted, would have limited  
utility for purposes 1 and 3 (since by definition it does not aim to  
be production-quality or high-performance). It may still help with  
goal #2 a lot if browser-hosted.

> Conversely, I don't think a "reference implementation" really makes  
> the cut, even if it is hosted in a browser.  However, there isn't  
> necessarily a sharp line between a reference implementation and a  
> simplistic or naïve "production" implementation so maybe the browser  
> hosted requirement is as close as we can come to pinning that down.
>
> I'm ambivalent on the single feature or entire specification  
> question. For a spec. on the order what is being proposed as ES3.1 I  
> don't think an entire spec. requirement would be unreasonably  
> burdensome.  For more complex feature sets I'm less sure.  A related  
> question is what does it take for a feature to even get into a  
> proposed specification? You can't require complete implementation of  
> a spec. that is not yet complete.  Is a public, browser based  
> implementation a pre-requisite for even getting to the state of a  
> feature proposal? I could argue both sides of that one.

I think it would be overkill to require an implementation (or even a  
test suite) to even make a proposal. We should expect that at some  
point in the standards development process, there will be a cycle of  
feedback among implementations and the spec, where implementors  
identify problems and propose changes, and the spec is adapted as a  
result.

The IETF and W3C, the two other major standards organizations that  
operate in the area of Web and Internet standards, have a two  
implementation rule combined with a sequence of maturity levels. Here  
is a summary of their levels:


IETF <http://www.ietf.org/rfc/rfc2026.txt>:

* Internet-Draft -- Initial published version, for comment and review  
(typically there are multiple iterations as an Internet-Draft, the  
minimum comment period is 2 weeks)

* Proposed Standard --  Has resolved known design choices, is believed  
to be well-understood, has received significant community review. But  
further experience might result in a change or even retraction of the  
specification before it advances. This is the stage at which wide  
implementation is encouraged. Minimum 6 month duration.

* Draft Standard -- Two independent and interoperable implementations  
from different code bases have been developed. Well-understood and  
known to be quite stable. Minimum 4 month duration.

* Internet Standard -- Characterized by a high degree of technical  
maturity and by a generally held belief that the specified protocol or  
service provides significant benefit to the Internet    community.  
(Not all implementations reach this stage.)

(There is also a minimum 2 week Last Call period between any state  
transition.)


W3C <http://www.w3.org/2005/10/Process-20051014/tr.html#maturity- 
levels>:

* Working Draft -- A signal to the community to begin reviewing the  
document. (Typically multiple iterations of this step.)

* Last Call Working Draft -- Final Working Draft that, if no serious  
issues are found, proceeds to Candidate Recommendation. All technical  
requirements are believed to be satisfied. Minimum 3 week duration.

* Candidate Recommendation -- The Director calls for implementations.  
Features may be identified as "at risk" in which case they are dropped  
if interoperable implementations are not produced, instead of sending  
the spec back to Working Draft. Implementation and test suite work  
occurs. A minimum duration must be stated before entering CR.

* Proposed Recommendation -- Two interoperable implementations are  
required, and the spec should be considered mature and widely  
reviewed, with significant experience. After this final review period,  
the Director and the W3C Membership vote whether to advance to a full  
Recommendation. Minimum 4 week duration.

* Recommendation -- Final stage. The spec is believed to have  
significant support and the ideas in it are appropriate for the web.


I would say the drafts of ES3.1 and ES4 published so far are  
equivalent in maturity to an IETF Internet-Draft or a W3C Working  
Draft. I would suggest that before seeking final ECMA approval, we  
should reach a maturity level equivalent to IETF Draft Standard or W3C  
Proposed Recommendation. This would also imply that we need a stage  
equivalent to Proposed Standard / Candidate Recommendation - where the  
spec is considered largely complete and stable, but we are focusing on  
finishing implementations and developing tests. (Of course, this does  
not mean that implementation work cannot start early, as is being done  
for ES4.)

> Feature interactions are often a source of unanticipated problems.   
> That argues for testing an entire specification.

I agree.

> Is there a time limit in which implementations must occur?  How do  
> we actually reach the point of "shipping" a revised specification.

If we do something analogous to W3C or IETF process, there is no time  
limit but the spec can't "ship" in final form until the  
implementations are there. However, if it sits in this state for a  
long time, features may be dropped or we go back to the drawing board  
more generally.

> An implementation, without some way to validate it seems of limited  
> use.  Should we also expect those who propose features to also  
> provide a validation suite?

That would be a pretty high bar to pass to even make a suggestion, but  
I do think it should be strongly encouraged.

> How fine a granularity do we push this to. Some of the sorts of  
> clarify changes to existing specification language or changes to  
> specification models that are needed to accommodate new features may  
> not even be directly testable if their intent is no net change to  
> the existing feature set.

Anything in the spec that has no observable effect on behavior does  
not need to be tested. Only normative conformance criteria need to be  
tested.

> Don't take this a "vote" yet, as we certainly need to have some  
> internal discussion on the topic.  However, I don't see how such a  
> requirement would be in any way inconsistent with how Microsoft  
> currently thinks about the process of web evolution.

Cool, I look forward to hearing more from you on this.

Regards,
Maciej