The confusion matrix: precision, recall and who pays for each mistake

Sooner or later every binary classifier meets the same question: should it favour precision or recall? The stock answer is a slogan (recall for medicine, precision for spam), and it stops helping as soon as real numbers arrive. This note works through what the confusion matrix says, why precision collapses when positives are rare, how costs become a threshold, and how three settings resolve the trade-off: cancer screening, card fraud, and fake accounts on a short-video platform.

Takeaway. Recall is a property of the model and its threshold. Precision is a property of those and of the population the model runs on. Which one to favour is a property of the action that follows a positive. All three settings below end up with several thresholds and graded actions, not one trade-off.

The problem

A model produces a score. A product needs a decision, so someone puts a threshold on the score, and from then on the model makes two kinds of mistake. It flags things that are fine, and it lets through things that are not. The two mistakes are rarely paid for by the same person, and rarely cost the same.

“Precision or recall?” compresses three separate questions:

1. How rare is the positive class where the model will run?
2. What does each mistake cost, and who pays?
3. What happens after the model says “positive”? A second test, a human review, or an irreversible action?

The confusion matrix is the place where all three become visible, so I start there.

One table, read three ways

The four cells

Rows are the truth; columns are the model’s call. The four cells are the true positives (TP), false negatives (FN), false positives (FP) and true negatives (TN). Before reading anything else, pin down which class is “positive”. By convention it is the rare one that triggers an action: the cancer, the fraud, the fake account. Every name below flips if that choice flips.

Here is a real matrix: 1,682,504 screening mammograms read in U.S. community practice from 2007 to 2013 (the unit is an exam, not a woman), from the Breast Cancer Surveillance Consortium (BCSC). “Positive” means cancer diagnosed within a year of the exam.

Across a row: what the model does to each class

Divide along a row and the denominator is the number of real positives or real negatives.

• Recall (sensitivity): $8{,}529 / 9{,}812 = 86.9\%$ of cancers were called back.
• Specificity: $1{,}486{,}553 / 1{,}672{,}692 = 88.9\%$ of healthy exams were read as normal. Its complement, the false positive rate (FPR), is 11.1%.

These two numbers describe the test. They do not depend on how many of the people screened have cancer.

Down a column: what a result means

Divide down a column and the denominator is the number of exams that got a given answer.

• Precision (positive predictive value, PPV): $8{,}529 / 194{,}668 = 4.4\%$ of call-backs were cancer.
• Negative predictive value: $1{,}486{,}553 / 1{,}487{,}836 = 99.91\%$ of normal readings were right.

These are what the person holding the result needs, and they depend on prevalence as much as on the test.

The whole table: accuracy

Accuracy adds the diagonal: 88.9%. A reader who called every exam normal would score 99.4% and find no cancers. Accuracy weights each cell by its count, so with 0.58% prevalence it is almost entirely a statement about healthy people.

The four rates, one at a time

The matrix has four cells and four natural rates. Each takes one of the two correct cells and divides it by its row or by its column, and each is the question of a different person.

Recall: of the real positives, how many did I catch?

This is the question of whoever is responsible for the positives: the screening programme, the fraud team that owns the losses. Its complement is the miss rate; here 13.1% of cancers were read as normal.

• Blind to: false alarms. FP is not in the formula, so recall says nothing about what catching those positives cost.
• Cheap way to 100%: flag everything.
• Leads when: a miss is the expensive error and something cheap follows a positive. Screening, a safety filter, removing names from clinical notes.
• Caveat: recall is an average over the positives the model happened to see. Faint cases are easier to miss than obvious ones, so a population with a different mix of easy and hard cases has a different recall, even with the same model.

Specificity: of the real negatives, how many did I leave alone?

This is the question of whoever pays for false alarms in bulk. Its complement is the false positive rate, 11.1%.

• Blind to: misses.
• Cheap way to 100%: flag nothing.
• Leads when: the action after a positive is costly or hard to undo: a confirmatory test, a declined payment, a removed account.
• Caveat: percentages flatter it. Negatives are the huge majority, so one point of specificity here is 16,727 exams. A specificity of 99% sounds near-perfect; where one payment in a thousand is fraud, it means ten false alarms for every fraud in the stream. When positives are rare, count false alarms per thousand or per million negatives instead.

Precision: when I raise an alarm, how often is it real?

This is the question of whoever receives the alarm: the woman called back, the analyst working an alert queue, the user whose account was flagged.

• Blind to: misses. A model can be precise by being timid.
• Cheap way to 100%: flag only the one case you are surest of.
• Leads when: a person absorbs every alarm: a review queue with fixed capacity, or an action taken against a customer.
• Two ways to read it: against the base rate, and upside down. A call-back raises the chance of cancer from 0.58% to 4.4%, a 7.5-fold lift. And $1/\text{precision}$ is workload: 23 call-backs for each cancer found.

NPV: when I give the all-clear, how often is it right?

This is the question of whoever is sent home: the woman told her mammogram is normal, the payment that goes through, the account left alone.

• Blind to: false alarms.
• Cheap way to look good: when positives are rare, clear everyone. That scores 99.42% here without reading a single image.
• Leads when: a negative ends the process: rule-out tests, triage, automatic approval.
• How to read it: on the complement, against the base rate. Before the reading, the chance of a cancer diagnosis within the year is 1 in 171. After a normal reading it is 1 in 1,160. The reading cut the risk about sevenfold; that is the real content of “99.91%”.

The complement of NPV, the false omission rate, is what a system leaves behind: cancers among the cleared, fraud among approved payments, fake accounts still active. “How much is left on the platform” is a number of this kind.

How the four fit together

Two pairings are worth remembering.

Through the threshold, recall trades against specificity. Lowering the threshold raises recall and lowers specificity, and nothing else about the model changes. The section on thresholds below shows it.

Across the diagonal, recall protects NPV and specificity protects precision. High recall leaves few false negatives to hide among the all-clears; high specificity leaves few false positives to dilute the alarms. Clinicians remember this as SnNOut and SpPIn: a sensitive test, when negative, rules out; a specific test, when positive, rules in. Pewsner and colleagues add the caution that ruling out also depends on specificity, and ruling in also on sensitivity. Prevalence matters as well: 88.9% specificity cannot rule anything in when only 0.58% of exams are positive.

No one of the four can be reported alone. Each has a trivial way to reach its maximum, and each is blind to one kind of error. A useful report pairs a rate that is blind to false alarms with one that is blind to misses (recall with precision is the machine-learning habit, sensitivity with specificity the medical one) and adds the prevalence. Those three numbers are enough to rebuild the whole matrix.

Eight rates, many names

Each rate has a complement: the other cell of the same row or column, over the same total. So the matrix holds eight rates, in four pairs that add up to 100%. Medicine, statistics, signal detection and machine learning named them independently, which is why one number goes by several names.

Two more rates describe the table rather than the model’s accuracy: prevalence, (TP + FN) / N = 0.58%, and the flag rate, (TP + FP) / N = 11.6%, the share of cases called positive (also alert rate or positive rate).

Same word, different number

The names collide, and a few collisions are worth knowing before reading anyone else’s metrics.

• “Recall rate” in mammography is not recall. Radiology audits use it for the share of screening exams called back: the BCSC defines it as the percentage of exams with an abnormal initial interpretation. Here that is 11.6%, the flag rate. What machine learning calls recall is sensitivity there: 86.9%.
• “Detection rate” usually means recall in security and fraud work. In screening, the cancer detection rate is cancers found per 1,000 exams, TP / N: 5.1 here.
• “Hit rate” is recall in signal detection. Fraud teams also use it for alert quality, including the ratio of good customers flagged to fraud caught.
• False positive rate and false positive ratio. The rate divides by the real negatives. The ratio, common in fraud operations, counts false positives per true fraud. For screening it is 22 to 1.
• “Our false positive rate is 90%”, said of an alert queue, usually means that 90% of alerts were false. That is the false discovery rate, 1 − precision. The false positive rate of the same system is often well under 1%.

The safe habit is to ask for the formula, or better, the four counts.

One-number summaries

Accuracy was the first attempt to fold the table into one number, and it fails when positives are rare. The other common ones:

• Balanced accuracy, the mean of recall and specificity, is 87.9% here. It ignores prevalence, which fixes the main flaw of accuracy, but it still treats a miss and a false alarm as equally bad.
• $F_1$, the harmonic mean of precision and recall, is 0.08 here. A harmonic mean stays close to the smaller of its two inputs, so $F_1$ is high only when both are. It ignores true negatives, and it depends on which class is called positive: score the same readers with “normal” as the positive class and $F_1$ is 0.94.
• $F_\beta$ lets recall count $\beta$ times as much as precision, without saying where $\beta$ comes from.
• Youden’s J, recall + specificity − 1, is 0.76 here. It is how far the operating point sits above the diagonal of the ROC plot (receiver operating characteristic: recall against false positive rate, introduced below), and a common way to choose a threshold when costs are unknown. Its other name is informedness; its column twin, precision + NPV − 1, is called markedness and is 0.043 here.
• Matthews correlation uses all four cells and is 0.18 here. It equals the geometric mean of informedness and markedness, so it is low whenever either reading of the table is weak.
• Diagnostic odds ratio, the positive likelihood ratio LR+ = recall / FPR divided by the negative likelihood ratio LR− = (1 − recall) / specificity, is 53 here: the odds of a positive reading are 53 times higher with cancer than without.

None of these says whether screening is worth doing, because none knows what a miss or a false alarm costs. I would rather write the costs down, which a later section does.

Why precision collapses when positives are rare

The screening test above catches 87% of cancers and clears 89% of the exams with no cancer, and still 96 of every 100 call-backs are false alarms. Nothing is wrong with the test. There are about 170 cancer-free exams for every exam with cancer, so an 11% false positive rate applied to the large group swamps an 87% recall applied to the small one.

Bayes’ rule in odds form makes this a single multiplication:

For screening, the odds before are 1 to 170 and the likelihood ratio is $0.869 / 0.111 = 7.8$. The flag makes cancer 7.8 times more likely, which is a strong test, and the odds after are still only 1 to 22. Written as a probability:

Three things follow.

The false positive rate has to be about as small as the prevalence. Precision reaches 50% only when $\text{FPR} \le \text{recall} \cdot \pi / (1 - \pi)$. With positives at 0.1% and recall at 80%, that means flagging fewer than 0.08% of negatives. A false positive rate of 1% sounds excellent and gives 7% precision there.

Precision measured on a rebalanced test set does not transfer. A model with 90% recall and a 5% false positive rate shows 94.7% precision on a 50/50 test set and 15.4% where positives are 1%. Recall and specificity carry over, as long as the kinds of cases stay the same; precision has to be recomputed at the prevalence of deployment with the formula above.

The same model is a different product in each population. Move the screening test to a group where 4% are positive and its precision is 25%; move it to card-fraud prevalence and it is 0.5%.

The matrix is a snapshot of one threshold

A confusion matrix belongs to a model at a threshold. Sliding the threshold moves cases within a row: true positives become false negatives, or false positives become true negatives. So recall and the false positive rate rise and fall together, and precision is a by-product of where both land and of the prevalence.

That is why two curves that describe the same model can tell such different stories.

The ROC curve plots the two row rates against each other, so it is blind to prevalence. Its area here is 0.95 whether positives are half the data or one in 170. The precision-recall curve plots a row rate against a column rate. At the same operating point precision is 89%, 29% or 4.4% as positives go from 50% to 5% to 0.58%.

Saito and Rehmsmeier argue that this makes ROC plots easy to misread on imbalanced data: a large pool of negatives makes specificity look reassuring. Davis and Goadrich prove the two spaces agree about dominance (a curve that wins everywhere in one wins everywhere in the other), and also that optimising ROC area does not optimise precision-recall area. My reading: use ROC to compare models, and the precision-recall curve at deployment prevalence to decide whether a model is usable.

From costs to a threshold

Give each error a cost and the expected cost per case is

If the model outputs a calibrated probability $p$ that a case is positive, flagging costs $(1-p)\,c_{FP}$ on average and not flagging costs $p\,c_{FN}$. Flag when the first is smaller, which gives the threshold from Elkan’s cost-sensitive learning paper:

This reframes the slogan. “Favour recall” means misses are expensive relative to false alarms, so the threshold is low. “Favour precision” means the opposite. And the costs attach to the action, not to the field: a call-back for more imaging and a biopsy are both “medicine”, and they sit at very different thresholds.

Four practical complications:

• The threshold assumes calibrated probabilities. Training on resampled data breaks calibration. If only a fraction $\beta$ of negatives is kept, the model’s odds are inflated by $1/\beta$; the corrected probability is $p = \beta p_s / (\beta p_s + 1 - p_s)$.
• 0.5 is a special case. It is right only when the two errors cost the same and the probabilities are calibrated.
• Costs can differ per case. A missed $2,000 fraud is not a missed $20 fraud. Then every case has its own threshold.
• Capacity can set the threshold instead. If people review every alert and can handle 500 a day, the operating point is “the 500 highest scores”, and the useful metrics are precision within that budget and the recall it buys.

Three settings

Medical screening: recall first, then a second test

In the BCSC benchmarks, 11.6% of screening exams were called back and 5.1 cancers were found per 1,000 exams. Precision at that step was 4.4%. Among exams that ended in a biopsy recommendation, it was 25.6%.

A precision of 4.4% is acceptable because of what a positive triggers: more imaging, not treatment. The false alarm costs an appointment and some anxious days. The miss costs a cancer found a year or more later. With a cost ratio that lopsided, the threshold belongs low.

The more important point is that screening is not one decision. Each stage raises the prevalence for the next: 0.58% in the screened population, 4.4% among those called back, about 26% among those sent to biopsy. A sensitive first test followed by a specific second test gets recall and precision out of a pipeline, which no single threshold could.

Recall is still not pushed to 100%. False alarms repeat at every screening round, follow-up has its own harms, and readers’ time is finite. In the same BCSC data only about six in ten radiologists met the recommended range for the call-back rate; the profession audits the false-alarm side as well.

Card fraud: both errors are priced in money

Fraud is rarer still. The Nilson Report puts 2023 card fraud losses at $33.83 billion worldwide, which its press release expresses as 6.58 cents per $100 of volume: under 0.1% by value.

Take 1,000,000 payments of which 1,000 are fraudulent, and a model with 80% recall. At a 0.5% false positive rate it raises 4,995 false alarms for 800 frauds caught: 13.8% precision, six interrupted customers per fraud. Cutting the false positive rate to 0.1% lifts precision to 44.5%. That is the prevalence rule from earlier, in a setting where each false alarm is a customer standing at a checkout.

Here both errors are in dollars, so the cost formula applies almost literally, one payment at a time. As an illustration with made-up but plausible numbers, let a miss cost the amount $A$ and a false decline cost 2% of $A$ in lost margin plus $3 of friction. Then $p^*$ is 14.5% for a $20 payment, 3.4% for $200 and 2.1% for $2,000. Larger payments deserve lower thresholds, which is also why fraud teams report recall weighted by value (the share of fraud dollars stopped) and not only the share of fraud cases.

Three things make the fraud matrix harder to measure than the medical one:

• Labels are late. A fraud is confirmed when the customer reports it or a chargeback arrives. Last week’s matrix is still filling in.
• Labels are selective. A declined payment never reveals whether it was fraud. Only alerted payments are checked quickly, so the fastest feedback comes from the cases the model already suspected. Dal Pozzolo and colleagues call these verification latency and the alert-feedback interaction, and note that for investigators, who can check only a limited number of alerts, the precision of those alerts is the measure that matters.
• The positives fight back. Fraud patterns change in response to the model, so an operating point chosen in January drifts.

And as in screening, the answer is more than one threshold: approve, ask for a one-time code, hold for review, decline. The one-time code plays the role of the second mammogram. It is a cheap extra test that a real customer passes in seconds, so it can sit at a low, recall-friendly threshold.

Fake accounts: scale turns small rates into crowds

On a platform, the positives are less rare and much harder to count. Meta estimates the share of fake accounts on Facebook by sampling monthly active users and labelling them, and has recently put it at about 4%. TikTok publishes removals rather than a prevalence estimate: 146.8 million fake accounts removed worldwide in the second quarter of 2026.

Suppose a platform has a billion accounts, 4% of them fake, and a detector with 90% recall and a 0.1% false positive rate. Precision is 97.4%, a number any classifier would be proud of. It is also 960,000 real people actioned by mistake. Precision as a percentage hides the crowd. The authors of Meta’s Deep Entity Classification system make the same point about why abuse classifiers need very high precision.

The two errors also land differently. A missed fake account spreads its cost thinly: slightly distorted rankings, inflated follower counts, less trustworthy ad metrics. A false positive lands on one person, who may have lost an audience or an income. So the action ladder matters more here than anywhere:

• Quiet, reversible actions run at high recall. Not counting a suspicious account’s likes, or asking for a CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart), costs a real user almost nothing.
• Removal runs at very high precision, usually with an appeal path, because it is the one action the user cannot undo.

Measurement is the hardest of the three settings, because nobody hands over the truth. Precision is estimated by having people review a sample of actioned accounts; overturned appeals give a floor on false positives. Recall cannot be observed at all. It is inferred from the prevalence that sampling finds among the accounts still standing.

There is also a third axis the matrix has no cell for: time. An account removed after it has posted its spam is a true positive in the table and a failure in practice. Blocking at sign-up and removing after the fact are different products even at identical precision and recall.

What the three have in common

None of the three picks precision over recall once. Each has rare positives and lopsided costs, and each answers with the same design: several thresholds on one score, with actions that get more expensive and less reversible as the score rises. The early, cheap stage is tuned for recall and tolerates terrible precision. The late, costly stage is tuned for precision and tolerates missing cases, because the earlier stage already caught the attention of a human or a second test. The trade-off is real at each threshold; the system as a whole gets both.

What I check before reporting a classifier

1. Name the positive class and the unit. Per exam or per patient, per payment or per card, per account or per action.
2. Use the prevalence where the model will run, not the one in the dataset.
3. Say what follows a positive and who pays for each error. That fixes the cost ratio, and the cost ratio fixes the threshold.
4. Report the matrix at the operating point, as counts. Rates without counts hide small samples: 27 hits out of 30 positives is 90% recall with a 95% interval from 74% to 97%.
5. Compare with doing nothing. “Always negative” scored 99.4% accuracy above.
6. Show the precision-recall curve at deployment prevalence, and check calibration before thresholding a probability.
7. Ask where the labels came from and how late they are.
8. Re-measure after launch. Prevalence moves, and in fraud and abuse the positives adapt.

From my own work

Three earlier projects of mine read differently with this in hand.

RadQNLI: accuracy fell, recall did not. In RadQNLI I tested ChatGPT on four radiology test sets whose negatives shared more and more words with the question. Accuracy fell from 0.81 to 0.62, which looks like a difficulty effect. But recall on the positive class stayed between 0.52 and 0.53 on every set, while the share of positives rose from 21% to 70% and precision rose from 0.56 to 0.89. A control with the same number of random negatives fell almost as far, from 0.83 to 0.64. The matrix separates two effects. Specificity did slip on the word-overlap negatives, from 0.89 to 0.84, while on random negatives it stayed near 0.92: that part is real difficulty. The rest is class mix. Accuracy is $\pi \cdot \text{recall} + (1-\pi) \cdot \text{specificity}$, so with recall stuck at 0.53, raising $\pi$ from 21% to 70% drags accuracy down toward recall.

Heart disease: which class was positive? In my heart-disease course project the data file codes target = 1 as no disease. Every per-class metric for class 1 therefore describes healthy patients, and “recall” for that class is what a clinician would call specificity. The 61 test patients split into 36 without disease and 25 with it, so recall for the disease class rests on 25 cases: 22 right out of 25 would be 88%, with a 95% interval from 70% to 96%. And the headline number was ROC AUC, the area under the ROC curve, which says how well a model ranks but not where to cut.

De-identification: a recall task with a misleading floor. In open-LLM de-identification, which asks open large language models (LLMs) to strip protected health information from clinical notes, the costly error is a leaked name, so the task is recall-first; over-redaction only costs readability. The score I used counted the share of protected strings removed, and because of how matches were counted, a note that had not been touched at all already scored 0.597. Every result has to be read against that floor. It is the same lesson as “always negative gets 99.4%”: check what doing nothing scores before trusting a metric.

References

1. Lehman CD, Arao RF, Sprague BL, et al. National Performance Benchmarks for Modern Screening Digital Mammography: Update from the Breast Cancer Surveillance Consortium. Radiology 2017;283(1):49–58. Counts from the BCSC benchmark tables; the false positive count is non-cancers minus true negatives.
2. Saito T, Rehmsmeier M. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE 2015;10(3):e0118432.
3. Davis J, Goadrich M. The Relationship Between Precision-Recall and ROC Curves. ICML 2006.
4. Elkan C. The Foundations of Cost-Sensitive Learning. IJCAI 2001.
5. Dal Pozzolo A, Boracchi G, Caelen O, Alippi C, Bontempi G. Credit Card Fraud Detection: A Realistic Modeling and a Novel Learning Strategy. IEEE Transactions on Neural Networks and Learning Systems 2018;29(8):3784–3797.
6. The Nilson Report. Card Fraud Losses Worldwide in 2023. December 2024; the per-$100 figure is from its press release of 6 January 2025.
7. Meta Transparency Center. Prevalence: how the share of fake accounts is estimated. The 4% figure as reported by Social Media Today.
8. TikTok Community Guidelines Enforcement Report, Q2 2026, as reported by the Dhaka Tribune.
9. Xu T, Goossen G, Cevahir HK, et al. Deep Entity Classification: Abusive Account Detection for Online Social Networks. USENIX Security 2021.
10. Pewsner D, Battaglia M, Minder C, Marx A, Bucher HC, Egger M. Ruling a diagnosis in or out with “SpPIn” and “SnNOut”: a note of caution. BMJ 2004;329:209–213.

The figures and every number computed in this note come from one script, tools/notes/precision_vs_recall_figs.py.