<br><br><div class="gmail_quote">On Mon, Jan 17, 2011 at 2:59 PM, Mark S. Miller <span dir="ltr"><<a href="mailto:erights@google.com">erights@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br><br><div class="gmail_quote"><div><div></div><div class="h5">On Mon, Jan 17, 2011 at 2:49 PM, Dave Fugate <span dir="ltr"><<a href="mailto:dfugate@microsoft.com" target="_blank">dfugate@microsoft.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Mark, basically what's going on is the original test harness expected either a return value of 'true' or falsy for all IE Test Center-based test cases. The problem with this is Sputnik tests don't return anything so the harness was altered at some point in time to accept 'true' or 'undefined' as success values. This change on its own is harmless provided all of Microsoft's tests explicitly return true or false - as you've discovered they do not. If you're interested, Chrome 8.0 also had the following false positives:<br>
- 11.1.5_4-4-b-1<br>
- 11.1.5_4-4-b-2<br>
- 11.1.5_4-4-c-1<br>
- 11.1.5_4-4-c-2<br>
- 11.1.5_4-4-d-1<br>
- 11.1.5_4-4-d-2<br>
- 11.1.5_4-4-d-3<br>
- 11.1.5_4-4-d-4<br>
- 11.4.1-4.a-5<br>
- 11.4.1-4.a-7<br>
- 15.2.3.3-4-178<br>
- 15.2.3.3-4-179<br>
- 15.2.3.3-4-180<br>
- 15.2.3.3-4-188<br>
- 15.4.4.14-1-1<br>
- 15.4.4.14-1-2<br>
- 15.5.4.20-1-1<br>
- 15.5.4.20-1-2<br>
- 15.5.4.20-4-10<br>
- 15.5.4.20-4-18<br>
- 15.5.4.20-4-34<br>
- 15.10.6<br>
- 15.10.7.1-1<br>
- 15.10.7.2-1<br>
- 15.10.7.3-1<br>
- 15.10.7.4-1<br>
- 15.10.7.5-1<br>
<br>
I just fixed the harness to differentiate between expected Sputnik/Test Center return values based on the filename (i.e., all Sputnik test cases begin with "S" where Test Center tests do not), and will upload this to the live website.<br>
</blockquote><div><br></div></div></div><div>This seems like the wrong fix. Why not fix all tests to return either true or false?</div></div></blockquote><div><br></div><div>And to have the test harness differentiate only truthy and falsy. Grouping true and undefined into the same bucket seems like a mistake.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="gmail_quote"><div><div></div><div class="h5"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
My best,<br>
<font color="#888888"><br>
Dave<br>
</font><div><br>
-----Original Message-----<br>
From: Dave Fugate<br>
Sent: Monday, January 17, 2011 12:20 PM<br>
To: <a href="mailto:test262-discuss@mozilla.org" target="_blank">test262-discuss@mozilla.org</a><br>
</div><div><div></div><div>Subject: RE: test262-discuss Digest, Vol 4, Issue 2<br>
<br>
Thanks for the report Mark! Looks like something is up with the test harness here (I see the same issue under Chrome 8.0.552.237), and will investigate.<br>
<br>
My best,<br>
<br>
Dave<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Mon, 17 Jan 2011 01:41:37 -0800<br>
From: Mark Miller <<a href="mailto:erights@gmail.com" target="_blank">erights@gmail.com</a>><br>
To: <a href="mailto:test262-discuss@mozilla.org" target="_blank">test262-discuss@mozilla.org</a><br>
Subject: test262 shows a pass for a failing testcase<br>
Message-ID:<br>
<AANLkTi=rA+roQpYGDhTD=<a href="mailto:1McpkNgVqXNtfhEaUpc%2BXf8@mail.gmail.com" target="_blank">1McpkNgVqXNtfhEaUpc+Xf8@mail.gmail.com</a>><br>
Content-Type: text/plain; charset=ISO-8859-1<br>
<br>
After running test262 on Chrome 10.0.634.0 dev, the results page shows<br>
<br>
> 15.10.7.2-1 RegExp.prototype.global is of type Boolean [source] pass<br>
<br>
Clicking on the source gives<br>
<br>
<br>
> Test?15.10.7.2-1<br>
><br>
> Description<br>
><br>
> RegExp.prototype.global is of type Boolean<br>
><br>
><br>
> Testcase<br>
><br>
> function testcase() {<br>
> if((typeof(RegExp.prototype.global)) === 'boolean')<br>
> return true;<br>
> }<br>
><br>
> Path<br>
><br>
> TestCases/chapter15/15.10/15.10.7/<a href="http://15.10.7.2/15.10.7.2-1.js" target="_blank">15.10.7.2/15.10.7.2-1.js</a><br>
><br>
<br>
But on Chrome 10.0.634.0 dev, RegExp.prototype is an object, not a RegExp, and it has no 'global' property. The above typeof returns the string 'undefined',so true is not returned, the function falls off the end returning undefined which is falsy.<br>
<br>
Why does this show as a passing test?<br>
<br>
--<br>
<br>
? ? Cheers,<br>
? ? --MarkM<br>
<br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
test262-discuss mailing list<br>
<a href="mailto:test262-discuss@mozilla.org" target="_blank">test262-discuss@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/test262-discuss" target="_blank">https://mail.mozilla.org/listinfo/test262-discuss</a><br>
<br>
<br>
End of test262-discuss Digest, Vol 4, Issue 2<br>
*********************************************<br>
<br>
_______________________________________________<br>
test262-discuss mailing list<br>
<a href="mailto:test262-discuss@mozilla.org" target="_blank">test262-discuss@mozilla.org</a><br>
<a href="https://mail.mozilla.org/listinfo/test262-discuss" target="_blank">https://mail.mozilla.org/listinfo/test262-discuss</a><br>
</div></div></blockquote></div></div></div><font color="#888888"><br><br clear="all"><br>-- <br> Cheers,<br> --MarkM<br>
</font></blockquote></div><br><br clear="all"><br>-- <br> Cheers,<br> --MarkM<br>