Announcing TC39 ECMAScript Test Suite Project - Test262

Allen Wirfs-Brock Allen.Wirfs-Brock at microsoft.com
Tue Jun 22 17:29:51 PDT 2010


(I'm going to try to shift this thread to the test262-discuss list, by bccing: es-discuss.)

I think we all agree that a web based test runner is part of what we want to have.  It seems inevitable that some sort of pass % or similar score has to display even though it's not clear whether such a score is very meaningful for this style of test suite.  You can get big fluctuation because a single bugs can cause many individual tests to fail while in other cases a really significant bugs  might only show up in a small number.

If anybody has any ideas for a meaningful metric that doesn't have those sorts of sensitivities, let's hear it.

Allen

> -----Original Message-----
> From: Christian Plesner Hansen [mailto:plesner at quenta.org]
> Sent: Tuesday, June 22, 2010 4:00 PM
> To: Sam Tobin-Hochstadt
> Cc: Allen Wirfs-Brock; test262-discuss at mozilla.org; Jonathan Chetwynd; es-
> discuss at mozilla.org
> Subject: Re: Announcing TC39 ECMAScript Test Suite Project - Test262
> 
> Yes, there is an acidic element to this project.  There will eventually be a website
> where anyone can test their browser's conformance against the test262 suite
> like they can today against the acid tests.
> 
> 
> -- Christian
> 
> On Tue, Jun 22, 2010 at 3:47 AM, Sam Tobin-Hochstadt <samth at ccs.neu.edu>
> wrote:
> > On Mon, Jun 21, 2010 at 1:42 PM, Allen Wirfs-Brock
> > <Allen.Wirfs-Brock at microsoft.com> wrote:
> >> The intent is a suite that tests an implementation against the actual
> requirement of the Ecma-262 standard. It will be based upon and similar to what
> you can already see in the Sputnik and ES5conform suites.  Those tests generally
> identify a specific requirement of a specific section of the spec. and then try to
> prove that the requirement is/isn't met by an implementation.
> >>
> >> Give that the standard specification itself  is not particularly understandable
> to the general public, it probably isn't reasonable to expect that all the tests of
> the specification would be understandable to the same public.
> >>
> >> However, reasonable understandability is a good goal.  What are some of the
> characteristics of the W3C suites that make them "acidic"?
> >
> > I believe that the various "Acid" tests for rendering are what's being
> > referred to here [1].  I think the relevant characteristc is that
> > anyone can run them in their browser, and they display a visual
> > rendering of how "good" a job the browser does, making them into
> > easily-distinguishable markers of conformance to whatever is tested.
> > It's not clear to me whether this would be good for this test suite.
> >
> > [1] http://en.wikipedia.org/wiki/Acid3
> > --
> > sam th
> > samth at ccs.neu.edu
> > _______________________________________________
> > test262-discuss mailing list
> > test262-discuss at mozilla.org
> > https://mail.mozilla.org/listinfo/test262-discuss
> >



More information about the es-discuss mailing list