Thoughts on IEEE P754

Waldemar Horwat waldemar at google.com
Fri Aug 22 16:07:08 PDT 2008


Sam Ruby wrote:
> I find it easier to talk about real examples than abstractions.  I've
> done the following quickly, so forgive me if I get some detail wrong.
> 
> A binary floating point number has 52 bits of fraction, and by
> assuming an implicit leading one, they get an additional bit.  This
> means that 1.1 is stored as (for brevity, I'll use hex)
> 
> [1].1999999999999
> 
> A conversion of that to decimal128 would be equivalent to computing
> 
> 4953959590107545m / 4503599627370496m
> 
> Which would produce
> 
> 1.099999999999999866773237044981215

That's incorrect.  The correct answer is 1.100000000000000088817841970012523, which is closer mathematically to 1.1.

> Repeating that for 1.2 produces
> 
> 0x13333333333333L
> 5404319552844595m / 4503599627370496m
> 1.199999999999999955591079014993738

This one is correct.

    Waldemar


More information about the Es-discuss mailing list