Understanding Precision Measurement

precision measurement

Here, we will take a look at the concept of measurements at a high precision level. If you have a career in metrology, precision measurement is already an everyday part of your life. By nature, the world of measurements is complex. That said, the following will be a rather simplistic view of the principles of measurement, how it works, important factors in measuring, and how tools should be correctly used for measuring.

For a most fundamental measurement understanding, there are three all-important terms that are frequently used incorrectly, confused as having the same meaning, and that get mixed up – but which should be aptly understood – resolution, precision, and accuracy.


A measurement tool’s smallest distinguishable value is what is meant by resolution. Example: 1/10 of an inch is the resolution of a ruler which is divided up into tenths. On a more precision level, 1/10 thousandth of an inch is the resolution of a micrometer that is read to 1/10 thousandth. Having a fine resolution, however, does not necessarily make for precise or accurate measurements.

Precision and accuracy have distinguishing differences. Measurements can be:

  • Neither accurate nor precise, or
  • Precise but not accurate, or
  • Accurate but not precise, or
  • Both accurate and precise

In the world of metrology, accuracy and preciseness must go hand-in-hand.


A measurement of repeatability is what is meant by precision. Example: take an object and measure it five times. How close to one another are the five measurements? This is what is meant by precision. Another example: how often can a tool for measurement be read reliably and repeatedly? The finiteness of this measurement tool is its precision.


How close is a measurement to an object’s “true value? That closeness is what is known as accuracy. Example: let’s say a valve stem is precisely 5.000 mm or 0.1969 inches. Now, how close is what is reading out on the measurement tool to that precise value? That is the tool’s accuracy.

What Can Affect Precision and Accuracy?

As accurate, or precise, as a measuring tool may be, there are certain things that can lead to variations affecting both the precision and accuracy of any tool. These three things are as follows:

  • User error – Unfortunately, this happens all the time. The person doing the measuring is either inexperienced or simply having an off day. Two different people can measure the same part, five times each, and one person will come up with five different variations, while the other individual gets the same reading all five times.
  • Temperature – Though not always achievable, this is why temperatures should always be kept to neither one extreme nor the other – not too hot, not too cold. Consistency is equally as important. Temperatures should remain as constant as possible. Particularly in the case of metrology, extreme, fluctuating temperatures can have a direct effect on the size (or measurement) of a piece of metal. What is an ideal temperature when it comes to metrology? Standard is 20°C or 68°F.
  • Tool quality and the type of tool being used for measurement – Various ranges of accuracy apply to various tools of measurement. If accuracy is a requirement, a strict requirement, keep this in mind when you are shopping for the tools that you will be using on-the-job. Tool descriptions should include the manufacturer’s accuracy, specifically, if the measurement tool is of good quality.

Sometimes, the tool that you are using for a specific task may not be accurate enough for the preciseness of the measurement required. Be aware of how accurate the tool needs to be for whatever job is at hand.

Measuring tools can lose their accuracy or have to be adjusted. You should frequently check the tools that you use for measuring to make sure that they are supplying you with an accurate reading. Some measuring tools require calibration, regardless of their initial price. Recalibrating a measuring tool can be as simple as setting it to zero when the tips of the tool are clean and put together or as involved as using specifically manufactured gauges meant solely for checking calibration.

And of course, knowing exactly how to use a measuring tool correctly is paramount. Even the most accurate tool, when operated incorrectly, cannot supply the user with the right information. Take your time, use a firm but sensitive touch, read correctly and write your results down. These steps can help assure an accurate reading.

by George Chitos