CitectVBA Programming Reference > Understanding CitectVBA Language Basics > Numbers > Floating Point Calculation Rules

Floating Point Calculation Rules

Often precision, rounding, and accuracy in floating-point calculations can generate unexpected results. To avoid this situation, follow these rules:

  1. In a calculation involving both single and double precision, the result will not usually be any more accurate than single precision. If double precision is required, be certain all terms in the calculation, including constants, are specified in double precision.
  2. Never assume that a simple numeric value is accurately represented in the computer. Most floating-point values can't be precisely represented as a finite binary value. For example .1 is .0001100110011... in binary (it repeats forever), so it can't be represented with complete accuracy on a computer using binary arithmetic, which includes all PCs.
  3. Never assume that the result is accurate to the last decimal place. There are always small differences between the "true" answer and what can be calculated with the finite precision of any floating point processing unit.
  4. Never compare two floating-point values to see if they are equal or not-equal. This is a corollary to rule three. There are almost always going to be small differences between numbers that "should" be equal. Instead, always check to see if the numbers are nearly equal. That is, check to see if the difference between them is very small or insignificant.

See Also

Published June 2018