Precision in Six Sigma: Definition, Formula, and Examples

Precision measures how consistent repeated measurements are, not whether they are correct. If you measure the same part five times and get nearly the same number each time, your measurement system is precise.

Precision says nothing about accuracy, which measures how close your measurements land to the true value. A measurement system can be precise and wrong at the same time, if it consistently measures the same part 2 mm too small every time.

Six Sigma teams check precision using repeatability (does one person get the same result twice) and reproducibility (do two people get the same result), combined into a study called Gage R&R. A precise measurement system is a requirement before trusting any data collected with it.

Quick-Reference Table: Precision Concepts at a Glance

Term	What It Measures	How It’s Assessed
Precision	Consistency between repeated measurements	Standard deviation, variance, coefficient of variation
Accuracy	Closeness of a measurement to the true value	Bias, comparison to a known reference standard
Repeatability	Consistency when one person measures the same part repeatedly	Equipment variation (EV)
Reproducibility	Consistency when different people measure the same part	Appraiser variation (AV)
Gage R&R	Combined repeatability and reproducibility	%R&R compared against total process variation

Key Takeaways

• Precision is about consistency, not correctness. A precise measurement system gives the same answer repeatedly, whether that answer is right or wrong.
• Precision and accuracy are two separate properties. A measurement system needs both, and having one does not guarantee the other.
• Repeatability and reproducibility together define precision in practice. One checks single-operator consistency; the other checks operator-to-operator consistency.
• Gage R&R is the standard tool for measuring precision. It produces a %R&R number that tells a team whether their measurement system is trustworthy.
• A %R&R under 10% is generally acceptable. Above 30% usually means the measurement system needs repair before any data from it can be trusted.
• Precision problems hide inside process data. A process that looks unstable might actually have a stable process and an imprecise gage.

The Bullseye Model: Precision vs. Accuracy

The clearest way to separate these two concepts uses a target with four possible outcomes.

Pattern	Precision	Accuracy
Shots clustered tightly, but off-center	High	Low
Shots scattered widely, but centered on average	Low	High
Shots clustered tightly, centered on the bullseye	High	High
Shots scattered widely, off-center	Low	Low

Only the third pattern represents a trustworthy measurement system. A tightly clustered but off-center pattern is precise, but every measurement carries the same hidden error, called bias. A widely scattered but centered pattern is accurate on average, but no single measurement can be trusted on its own.

Why Precision Alone Is Not Enough?

A highly precise measurement system can still mislead an entire project. Imagine a caliper that measures the same part as 10.02 mm every single time, five times in a row. That result looks excellent. Precision is nearly perfect.

Now imagine the part’s true measurement is actually 10.50 mm. The caliper has high precision and low accuracy at the same time. Every reading is consistent, and every reading is wrong by the same fixed amount.

This is why Six Sigma checks precision and accuracy as two separate questions, never assuming that one implies the other.

Repeatability: The First Half of Precision

Repeatability measures whether one person, using one gage, gets the same result when measuring the same part multiple times. It is also called equipment variation, because differences here usually trace back to the measuring device itself.

Poor repeatability often points to a specific mechanical cause:

• A worn or damaged gage
• Inconsistent fixturing that holds the part differently each time
• A gage with insufficient resolution for the tolerance being measured

Reproducibility: The Second Half of Precision

Reproducibility measures whether different people, using the same gage, get the same result when measuring the same part. It is also called appraiser variation, because differences here usually trace back to how each person operates the equipment.

Poor reproducibility often points to a specific human cause:

• Inconsistent technique between operators
• Unclear or missing measurement instructions
• Operators applying different amounts of pressure or force during measurement

Gage R&R: Measuring Precision with a Formal Study

A Gage R&R study combines repeatability and reproducibility into one number, called %R&R. This study is the standard method Six Sigma teams use to formally assess precision before trusting any dataset.

A typical Gage R&R study follows this structure:

1. Select 8 to 10 parts that represent the full range of expected variation.
2. Select 2 to 3 operators who normally perform this measurement.
3. Have each operator measure each part 2 to 3 times, in random order, without knowing their previous readings.
4. Calculate the variation from repeatability (within one operator, across repeated measurements).
5. Calculate the variation from reproducibility (between different operators).
6. Combine both into a %R&R score, comparing measurement variation against total observed variation.

Also Read: Six Sigma in Precision Agriculture to Maximize Crop Yields

How to Read a %R&R Score

The Automotive Industry Action Group (AIAG) publishes widely accepted thresholds for interpreting a %R&R result.

%R&R Score	Interpretation	Action Needed
Under 10%	Excellent measurement system	None, system is acceptable
10% to 30%	Marginally acceptable	May be acceptable depending on application, cost, and risk
Over 30%	Unacceptable	Measurement system needs improvement before use

A %R&R above 30% means the measurement system itself contributes too much noise to trust any data collected with it. Fixing the measurement system usually comes before fixing the process it is measuring.

The Precision-to-Tolerance Ratio: A Second Way to Judge Precision

The precision-to-tolerance ratio (P/T ratio) compares a measurement system’s precision directly against the tolerance it needs to detect.

P/T Ratio = (Precision measurement, usually 5.15 × standard deviation) ÷ (Tolerance range)

A lower P/T ratio is better. A common rule of thumb accepts a P/T ratio below 10%, meaning the measurement system’s own variation uses up less than 10% of the allowed tolerance window. A P/T ratio above 30% usually signals the gage cannot reliably distinguish good parts from bad ones within that tolerance.

This ratio matters because a measurement system that works fine for a loose tolerance can fail completely for a tight one, even with no change to the gage itself.

A Practical Example: Checking Precision on a Caliper

A quality team wants to confirm whether a digital caliper is precise enough for a part with a tolerance of plus or minus 0.05 mm.

Step 1: Run a Gage R&R study. Two operators each measure the same 10 parts, three times each, in random order.

Step 2: Calculate the results.

Source	Contribution to Variation
Repeatability (equipment)	4%
Reproducibility (operators)	3%
Total %R&R	7%

Step 3: Interpret the result. A 7% total %R&R falls comfortably under the 10% threshold. The caliper and the operators together produce a precise, trustworthy measurement system for this tolerance.

If the same caliper were used on a tighter tolerance, say plus or minus 0.01 mm, the P/T ratio recalculated for that tighter window might exceed 30%, even though nothing about the caliper itself changed. Precision is not a fixed property of a gage. It depends on the tolerance being measured against.

Also Read: Key Differences Between Precision and Accuracy Explained

Coefficient of Variation: A Simple Precision Metric

For situations that do not require a full Gage R&R study, the coefficient of variation (CV) offers a quick precision check.

CV = (Standard deviation ÷ Mean) × 100

A lower CV indicates tighter, more precise measurements relative to the size of the value being measured. This metric works well for comparing precision across measurements with very different scales, since it expresses variation as a percentage rather than a raw number.

How Precision Fits Into a Six Sigma Project

Precision checks belong at the very start of the Measure step in a DMAIC project, before any process data gets collected for analysis. This order matters more than most teams realize.

A process that looks unstable in the data might actually be perfectly stable, with an imprecise gage creating false variation. Chasing a root cause based on imprecise data wastes weeks investigating a problem that does not exist in the real process, only in the measurement noise.

Total Observed Variation = Process Variation + Measurement Variation. A Gage R&R study separates these two sources before a team commits to analyzing the wrong one.

Common Mistakes When Assessing Precision

Assuming a calibrated gage is automatically precise. Calibration checks accuracy against a known standard. It does not test repeatability or reproducibility, which require a separate Gage R&R study.

Skipping the Gage R&R study to save time. Teams under deadline pressure sometimes skip this step, then spend far more time later chasing a root cause hidden inside measurement noise instead of real process variation.

Confusing a good %R&R with a good measurement system for every tolerance. A %R&R score is only valid for the specific tolerance it was tested against. A tighter tolerance on the same part may require a new study.

Treating precision and accuracy as the same concept. A system can pass a precision check and still produce consistently wrong numbers if it has never been checked for bias against a true reference value.

Frequently Asked Questions on Precision in Six Sigma

What does precision mean in Six Sigma?
Precision means how consistent repeated measurements are with each other, regardless of whether those measurements are correct. A precise measurement system gives nearly the same result every time it measures the same thing.

What is the difference between precision and accuracy?
Precision measures consistency between repeated measurements. Accuracy measures how close those measurements land to the true value. A measurement system can be precise without being accurate, and accurate on average without being precise.

What is Gage R&R?
Gage R&R is a formal study that measures precision by combining repeatability, which checks one operator’s consistency, and reproducibility, which checks consistency between different operators, into a single %R&R score.

What is a good %R&R score?
A %R&R under 10% is generally considered excellent. A score between 10% and 30% may be acceptable depending on the application, and a score above 30% usually means the measurement system needs improvement before its data can be trusted.

What is the precision-to-tolerance ratio?
The precision-to-tolerance ratio compares a measurement system’s precision directly against the tolerance range it needs to detect, expressed as a percentage. A lower ratio, typically under 10%, indicates the measurement system uses up little of the allowed tolerance window with its own variation.

Can a measurement system be precise but not accurate?
Yes. A precise but inaccurate system gives consistent results that are all wrong by roughly the same amount, a pattern often called bias. This is one of the four possible combinations shown by the classic precision-versus-accuracy bullseye model.

Why does precision matter before analyzing process data?
Total observed variation combines both real process variation and measurement variation. Checking precision first, usually through a Gage R&R study, prevents a team from mistaking measurement noise for a genuine process problem.

Final Words

Precision answers a narrow but essential question: does this measurement system produce the same result when nothing about the part has actually changed? It says nothing about whether that result is correct, which is why accuracy has to be checked separately. Repeatability and reproducibility, combined into a Gage R&R study, give a team a defensible %R&R score before any process data gets trusted for analysis.

Skipping this check risks the most expensive kind of mistake in a Six Sigma project: chasing a root cause that exists only in the measurement system, not in the actual process.

Build Your Measurement Skills with SSDSI

Trusting your data starts with trusting the system that measures it. SSDSI delivers IASSC-accredited Green Belt and Black Belt training that covers precision, accuracy, and full Measurement System Analysis, including Gage R&R, through onsite, live virtual, public, and self-paced formats.

SSDSI has certified 5,322+ professionals across 600+ organizations in 52 cities, backed by a 5-star Google review rating. If you want to build this foundation properly, a conversation about the right certification path is a solid next step.

Lean Six Sigma Green Belt Training Brochure

Related Articles