ROUNDING ERROR, AN UNEXPECTED FAULT IN THE OUTPUT FROM A RECORDING SPECTROPHOTOMETER - IMPLICATIONS FOR MODEL DISCRIMINATION

Although commonly ignored in discussions of experimental error, roundi ng may sometimes be the major source of error, especially with modem p recision instruments: some recording spectrophotometers are optically and photometrically capable of making absorbance measurements with err ors less than 0.0003, but provide no numerical information more precis e than +/-0.001. The problem may be diagnosed by a characteristic arra ngement of points in a residual plot, which resembles the result of cu tting a stroboscopic picture of a bouncing ball into several strips an d modifying it by sliding the strips relative to one another to bring the points closer to the axis. Harmful effects of rounding error can b e critical in experiments designed for model discrimination.