Home/Support/Support Forum/floating point problems
Welcome to Digi Forum, where you can ask questions and receive answers from other members of the community.

floating point problems

0 votes
I am encountering some strange floating point problems.
Two notable cases: 0.01 is represented as .09. Also, using the ftoa function, 3.0 is returned properly as 3.0 (30 -8). 5.0 is returned as 50.0 (50 -7). I haven't determined yet if the second problem is an issue with the ftoa function or the way the floating value is being stored. Has anyone else encountered this? And more importantly, has anyone found a solution?

Chuck
asked Oct 30, 2008 in Rabbit by ChasLAmy New to the Community (0 points)

Please log in or register to answer this question.

1 Answer

0 votes
What version of DC are you seeing this under?
answered Nov 6, 2008 by bsprouse Seasoned Professional (202 points)
...