|
Re: Glitch in floor() function?
|
Matt Lewis
|
Jun 26, 2008 13:40 PDT
|
posted by: Matt Lewis <matthewwalkerlewis at gm?i?.com>
Arthur Crump wrote:
| |
Leaving in just the relevant parts of Mike Wever's message:
Mike Wever wrote:
| |
procedure round(atom x, integer precision)
integer mult
mult = power(10, precision)
printf(1, "initial: %f\n", {x})
x *= mult
printf(1, "shifted right: %f\n", {x})
x += 0.5
printf(1, "plus rounding half: %f\n", {x})
x = floor(x)
printf(1, "floor: %f\n", {x})
x /= mult
printf(1, "shifted left: %f\n", {x})
end procedure
round(1.005,2)
produces these results:
...
initial: 1.005000
shifted right: 100.500000
plus rounding half: 101.000000
floor: 100.000000
shifted left: 1.000000
It seems that floor(101) gave a result of 100. Is this an error or am I
missing something?
|
Effectively the result of:
floor( 1.005*100 + .5 )
is required.
The problem arises because 1.005 is converted into binary.
|
Arthur is correct. It's just a standard limitation of floating point
numbers. To see what's going on, change your format specifier from
"%f" to "%0.15f".
Matt
|
|
 |
|
