<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText">Thanks for your suggestions David. Comments below:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> Hi,<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Here are a couple of suggestions which, in my opinion, would help the
<o:p></o:p></p>
<p class="MsoPlainText">> community work:<o:p></o:p></p>
<p class="MsoPlainText">> - test262.ecmascript.org<o:p></o:p></p>
<p class="MsoPlainText">> -- In the Results page, it would be useful to have a link to the test
<o:p></o:p></p>
<p class="MsoPlainText">> file in the <a href="http://hg.ecmascript.org/tests/test262/">
http://hg.ecmascript.org/tests/test262/</a> website (may help <o:p></o:p></p>
<p class="MsoPlainText">> to spot differences between the run test suite and the latest )<o:p></o:p></p>
<p class="MsoPlainText">I love the idea. It's just a bit non-trivial to implement until we get tests from all external contributions living happily in the same predictable directory. For example, instead of having "test/suite/ietestcenter/chapter07/*" and
"test/suite/sputnik_converted/07_Lexical_Conventions/7.6_Identifiers/*", we'd simply have "test/suite/ch07/*". Any ways, once this directory cleanup occurs I think we should go ahead and do this. Filed as
<a href="https://bugs.ecmascript.org/show_bug.cgi?id=44">https://bugs.ecmascript.org/show_bug.cgi?id=44</a><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> -- A "report a bug" link next to each bug result (or in the "source"<o:p></o:p></p>
<p class="MsoPlainText">> popup?)would be useful too. If there is such a feature in Bugzilla,
<o:p></o:p></p>
<p class="MsoPlainText">> the link could carry a preformatted bug title and pre-fill the bug report to point to the "Tests"<o:p></o:p></p>
<p class="MsoPlainText">> component. Uniform bug titles make text-search easier.<o:p></o:p></p>
<p class="MsoPlainText">I definitely agree some sort of a bug template mechanism would be useful. E.g., click a link and a new browser window pops up and pre-populates a new Test262 'tests' bug with the test suite version, test harness version, etc. Making
this available for each and every failure for a browser only makes sense though if a high percentage of the test cases are actually invalid. Such a link might actually detract from the credibility of Test262 as well if applied to every failing test case.
Filed as <a href="https://bugs.ecmascript.org/show_bug.cgi?id=45">https://bugs.ecmascript.org/show_bug.cgi?id=45</a>
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> - Test searching<o:p></o:p></p>
<p class="MsoPlainText">> There are now about 10000 tests. By default, they are sorted in the
<o:p></o:p></p>
<p class="MsoPlainText">> repo by source(Sputnik | ietestcenter)/chapter/section/subsection/...<o:p></o:p></p>
<p class="MsoPlainText">> Even if it certainly makes sense from a test suite producer point of
<o:p></o:p></p>
<p class="MsoPlainText">> view (and directory choices has to be made anyway), it doesn't cover all search use cases.<o:p></o:p></p>
<p class="MsoPlainText">> In order to "identify test holes" more efficiently as suggested under
<o:p></o:p></p>
<p class="MsoPlainText">> "Community Contributions" it would be helpful to have some tool to
<o:p></o:p></p>
<p class="MsoPlainText">> search for tests. For instance, it's currently hard to find if a test has been forgotten regarding:<o:p></o:p></p>
<p class="MsoPlainText">> * native prototype objects<o:p></o:p></p>
<p class="MsoPlainText">> For instance, in 15.4.2.2 is written "The [[Prototype]] internal
<o:p></o:p></p>
<p class="MsoPlainText">> property of the newly constructed object is set to the original Array
<o:p></o:p></p>
<p class="MsoPlainText">> prototype object, the one that is the initial value of Array.prototype".<o:p></o:p></p>
<p class="MsoPlainText">> There is clearly a test to write based on that to make sure that
<o:p></o:p></p>
<p class="MsoPlainText">> Monkey-patched Array.prototype aren't used. Such a pattern can be
<o:p></o:p></p>
<p class="MsoPlainText">> found for functions (15.3.3, 15.3.4), objects (15.2.2.1 step 4) and certainly in other places.<o:p></o:p></p>
<p class="MsoPlainText">> * length properties<o:p></o:p></p>
<p class="MsoPlainText">> * Errors (make sure that code that should throw errors have a test for
<o:p></o:p></p>
<p class="MsoPlainText">> them)<o:p></o:p></p>
<p class="MsoPlainText">> * strict mode (I assume it's not going to be covered in a chapter)<o:p></o:p></p>
<p class="MsoPlainText">> * Any other cross-chapter topic you could think of.<o:p></o:p></p>
<p class="MsoPlainText">> Several different approaches could be taken to tackle that issue:<o:p></o:p></p>
<p class="MsoPlainText">> -- text-search<o:p></o:p></p>
<p class="MsoPlainText">> -- tags search<o:p></o:p></p>
<p class="MsoPlainText">There will be far more tag usage in the future in the form of either test object metadata or JSON metadata separate from the actual test files. This is needed in the harness to properly support features like Strict Mode, syntax errors
existing outside of an eval, etc. Already have bugs filed on these.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Hadn't considered the ability to search through the tags directly through the website though. I do like this idea; it just seems a bit beyond what we have the resources to do in the short term. Covered by
<a href="https://bugs.ecmascript.org/show_bug.cgi?id=46">https://bugs.ecmascript.org/show_bug.cgi?id=46</a>.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> - Run partial test suite<o:p></o:p></p>
<p class="MsoPlainText">> Running the test suite is currently long. I'm currently reviewing FF4
<o:p></o:p></p>
<p class="MsoPlainText">> fails. When I close my web browser, I have to re-run all tests.
<o:p></o:p></p>
<p class="MsoPlainText">> However, most of the time, I don't care about most of them. I just
<o:p></o:p></p>
<p class="MsoPlainText">> want all those that I haven't reviewed yet. I wish the website could offer an easy way to run for instance:<o:p></o:p></p>
<p class="MsoPlainText">> - all chapter 15.2 tests (or any chapter/section/subsection obviously)<o:p></o:p></p>
<p class="MsoPlainText">> - all tests found through a search (if/when some search feature will
<o:p></o:p></p>
<p class="MsoPlainText">> be<o:p></o:p></p>
<p class="MsoPlainText">> available)<o:p></o:p></p>
<p class="MsoPlainText">> - all tests that has been added or changed between two test suite
<o:p></o:p></p>
<p class="MsoPlainText">> versions. If I have reviewed all tests of version x, I may not care
<o:p></o:p></p>
<p class="MsoPlainText">> about them when the x+1 version comes out.<o:p></o:p></p>
<p class="MsoPlainText">Agreed for the most part. We've talked internally at Microsoft about requiring an opt-in for test chapters via a check-box to reduce test suite execution time. Filed as
<a href="https://bugs.ecmascript.org/show_bug.cgi?id=47">https://bugs.ecmascript.org/show_bug.cgi?id=47</a>.
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Ability to search for tests and run them alone would require a major re-write of the existing test harness though. Version 2.0 maybe:) Filed as
<a href="https://bugs.ecmascript.org/show_bug.cgi?id=46">https://bugs.ecmascript.org/show_bug.cgi?id=46</a>.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> - Accept user contributed tests<o:p></o:p></p>
<p class="MsoPlainText">> I have read that "Ecma's intellectual property policies, permit only
<o:p></o:p></p>
<p class="MsoPlainText">> Ecma members to directly contribute code to the project.". I however
<o:p></o:p></p>
<p class="MsoPlainText">> think that this test suite would benefit from accepting contributions
<o:p></o:p></p>
<p class="MsoPlainText">> from the community. No one ignore how big is the JavaScript community
<o:p></o:p></p>
<p class="MsoPlainText">> on Github. No one ignore how crucial is JavaScript to the world as it
<o:p></o:p></p>
<p class="MsoPlainText">> is the most used programming language. I think there is a need to open up the way tests can be contributed.<o:p></o:p></p>
<p class="MsoPlainText">> If the Ecma's intellectual property policies cannot be changed/adapted
<o:p></o:p></p>
<p class="MsoPlainText">> to make that happen, could ECMA TC-39 members find a way to accept
<o:p></o:p></p>
<p class="MsoPlainText">> user contributions, review them and then submit them to Ecma? For
<o:p></o:p></p>
<p class="MsoPlainText">> instance, by creating a TC-39 github account and a test repo, people
<o:p></o:p></p>
<p class="MsoPlainText">> create tests, submit them to this repo and when there are enough of
<o:p></o:p></p>
<p class="MsoPlainText">> them or after a certain amount of time, TC-39 members officially
<o:p></o:p></p>
<p class="MsoPlainText">> submit the tests to ECMA. There are certainly intellectual
<o:p></o:p></p>
<p class="MsoPlainText">> property/copyright issues that has to be considered, by I'm drawing some sort of main idea.<o:p></o:p></p>
<p class="MsoNormal">We’ll take this into consideration for the next TC-39 meeting;)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My best,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Dave<o:p></o:p></p>
</div>
</body>
</html>