Thoughts on IEEE P754
waldemar at google.com
Fri Aug 22 16:07:08 PDT 2008
Sam Ruby wrote:
> I find it easier to talk about real examples than abstractions. I've
> done the following quickly, so forgive me if I get some detail wrong.
> A binary floating point number has 52 bits of fraction, and by
> assuming an implicit leading one, they get an additional bit. This
> means that 1.1 is stored as (for brevity, I'll use hex)
> A conversion of that to decimal128 would be equivalent to computing
> 4953959590107545m / 4503599627370496m
> Which would produce
That's incorrect. The correct answer is 1.100000000000000088817841970012523, which is closer mathematically to 1.1.
> Repeating that for 1.2 produces
> 5404319552844595m / 4503599627370496m
This one is correct.
More information about the Es-discuss