TOPICS
Search

Precision


Precision is the number of significant digits carried in a numerical computation. In significance arithmetic, a nonzero approximation x with estimated absolute uncertainty delta>0 has decimal precision p=-log_(10)(delta/|x|). Thus precision measures relative uncertainty, whereas accuracy a=-log_(10)delta measures absolute uncertainty, with p=a+log_(10)|x|. These measures depend on the uncertainty assigned to a number, not merely its displayed digits.

In many programming languages, numerical computations are done with some fixed precision.


See also

Absolute Error, Accuracy, Arbitrary Precision, Error Propagation, Relative Error, Significance Arithmetic, Significant Digits

Explore with Wolfram|Alpha

References

Wolfram Research. "Precision." Wolfram Language & System Documentation Center. https://reference.wolfram.com/language/ref/Precision.html.

Referenced on Wolfram|Alpha

Precision

Cite this as:

Weisstein, Eric W. "Precision." From MathWorld--A Wolfram Resource. https://mathworld.wolfram.com/Precision.html

Subject classifications