Tuesday, March 15, 2011

floating point numbers and representation error

Floating point numbers can be a great source of error. When we think about pure mathematics it includes the concept of infinity. For example, between the numbers 0 and 1 are infinite fractions.

When you store a floating point number in memory it has a limited number of values. This means that it is impossible to represent all possible floating point numbers. Different languages have different ways to represent numbers. The more accurate the floating point numbers are, the more expensive, computationally, using them becomes.

This means that operations like 1.47 - 1.00 might result in 0.46999999997 because the math library cannot represent 0.47 and 0.46999999997 is the closest match.

If the program you are testing is financial and it only deals with dollar and cents, this representation error can be problematic.

As a programmer there is a simple solution to this problem. Use only integers (long or double long if possible) and treat them as cents. So the above example would become:

long n1 = 147;           // $1.47
long n2 = n1 - 100;      // $1.47 - $1.00
long dollars = n2 / 100; // 0
long cents = n2 % 100;   // 47
System.out.printf("$%d.%02d", dollars, cents);

This means if you have a program which uses dollars and cents, you want to check for representation error (assuming the programmer used floating point variables) but you also want to check for issues which might be integer related. So you want to consider things like integer overflow and underflow. See the article Are most binary searches broken? for a discussion of overflow.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.