Thoughts on IEEE P754
Sam Ruby
rubys at intertwingly.net
Fri Aug 22 16:26:59 PDT 2008
On Fri, Aug 22, 2008 at 7:07 PM, Waldemar Horwat <waldemar at google.com> wrote:
> Sam Ruby wrote:
>> I find it easier to talk about real examples than abstractions. I've
>> done the following quickly, so forgive me if I get some detail wrong.
>>
>> A binary floating point number has 52 bits of fraction, and by
>> assuming an implicit leading one, they get an additional bit. This
>> means that 1.1 is stored as (for brevity, I'll use hex)
>>
>> [1].1999999999999
>>
>> A conversion of that to decimal128 would be equivalent to computing
>>
>> 4953959590107545m / 4503599627370496m
>>
>> Which would produce
>>
>> 1.099999999999999866773237044981215
>
> That's incorrect. The correct answer is 1.100000000000000088817841970012523, which is closer mathematically to 1.1.
Apparently, I rounded incorrectly. Rounding the first number
correctly produces:
1.199999999999A
4953959590107546m / 4503599627370496m
1.100000000000000088817841970012523
>> Repeating that for 1.2 produces
>>
>> 0x13333333333333L
>> 5404319552844595m / 4503599627370496m
>> 1.199999999999999955591079014993738
>
> This one is correct.
One ramification of requiring up-conversion to decimal128 of mixed
mode operations would mean that the following would be true
(1.1 < 1.1m) && (1.2m < 1.2)
> Waldemar
> _______________________________________________
> Es-discuss mailing list
> Es-discuss at mozilla.org
> https://mail.mozilla.org/listinfo/es-discuss
>
More information about the Es-discuss
mailing list