06 — Metrics

The number your model is judged by is a choice, and it is a choice you should be able to defend against the obvious alternative.

For every common metric, three things matter:

1. What question it actually answers.
2. What has to be true of your data before that answer means what people think it means.
3. The specific way it lies to you when that condition fails.

By the end you should be able to take an unfamiliar problem, name one metric, and state why the alternative is worse.

What goes in and what comes out

A metric is a small function with a fixed shape. Two equal-length lists go in, one number comes out.

• y — the true answer for each row of your evaluation data. A row is one example: one transaction, one patient, one search query.
• p — the model’s prediction for that same row.

That is the entire interface. Many different functions accept those same two lists, and they do not agree on which model is better.

A metric is a loss you agreed to be judged by

A metric is a loss function you have agreed to be judged by, and every loss function is minimised by a particular summary of the true spread of answers. So choosing a metric is choosing what your model will converge to.

A loss function is a rule that scores how bad a single prediction was. Training a model means adjusting it until the average of that score over the training rows is as small as possible.

Two pieces of notation carry the rest of the chapter, so pin them down now.

p(y|x) is the spread of true answers y you would see among rows that look like x. For one particular customer there is not a single true spend; there is a distribution of plausible spends.

A functional is one summary number pulled out of that spread — its average, its middle value, its 90th percentile, or the probability itself.

Here is that abstraction on three actual rows. Suppose three customers look completely identical to the model — same age band, same tenure, same everything it can read — and their actual spends turn out to be $10, $20, and $90:

the spread p(y|x) for these rows:   10, 20, 90

mean   = (10 + 20 + 90) / 3 = 40      <- one functional
median = 20                           <- a different functional

The model can emit only one number for all three rows, because it cannot tell them apart. Is that number 40 or 20? The metric decides. Squared error forces 40; absolute error forces 20.

Most metric arguments are really arguments about which functional of p(y|x) the business wants, without noticing that is the question.

The short names, expanded once

These abbreviations appear throughout and are used freely after this list. Each one gets its own section and its own worked arithmetic below.

Where the chapter goes

That single idea — a metric is a loss, and a loss picks out one summary of p(y|x) — generates every disagreement in the chapter:

• MSE and MAE disagree on skewed targets because one asks for a mean and the other asks for a median.
• ROC-AUC and PR-AUC disagree under imbalance because one normalizes by a huge denominator and the other does not.
• AUC and log loss disagree because one measures order and the other measures the numbers.

Each disagreement is derivable, and each is derived below.

1. Every metric names an optimal prediction

One procedure makes every later disagreement predictable: reduce a metric to the score it gives one row, ask which single number minimizes that score on average, and you have the target the metric will drag your model toward.

The trick, in three steps

1. Take any metric and strip it to its per-row loss l(y, p) — the score it assigns to one prediction on one row.
2. Ask: what constant c minimizes E[l(Y, c)]? Here E[...] means the expected value, the long-run average over rows drawn from the data.
3. That c is the answer. It is the number the model would be forced to emit if it could see nothing at all.

Why does a constant matter when your model is not a constant? Because your model is a constant inside every group of rows it cannot tell apart — every group whose features look identical to it. Within each such group, the metric drags the prediction to that group’s c. This is exactly the three-identical-customers situation from the intro.

A feature is one input column the model reads — the transaction amount, the merchant category, the hour of day — as distinct from y, the answer, and p, the prediction.

One piece of notation has to be fixed before the diagram, because the diagram uses it. That constant is written argmin over c of E[l(Y, c)]. argmin names the input that makes an expression smallest, not the smallest value itself:

argmin over c of (c - 3)^2  =  3      <- the input that minimizes it
min      over c of (c - 3)^2  =  0      <- the minimum value itself

The five metrics and the five targets

The diagram below is the chapter’s spine. Follow one path top to bottom: you pick a metric, that fixes a per-row loss, and the loss fixes what the model converges to. The five leaves are the five possible answers.

flowchart TD
    Q["You pick a metric"] --> L["which fixes a per-row loss l of y and p"]
    L --> A["the model converges toward<br/>argmin over c of the mean of l of Y and c"]
    A --> M["squared error<br/>-> conditional MEAN"]
    A --> D["absolute error<br/>-> conditional MEDIAN"]
    A --> P["pinball loss at tau<br/>-> conditional tau-QUANTILE"]
    A --> R["percentage error<br/>-> 1/y-weighted MEDIAN"]
    A --> LL["log loss or Brier<br/>-> conditional PROBABILITY"]

    style A fill:#1d3557,color:#fff
    style M fill:#2d6a4f,color:#fff
    style D fill:#40916c,color:#fff
    style LL fill:#bc6c25,color:#fff

The word conditional in every leaf means “computed within a group of rows the model cannot distinguish.” Here is what each leaf says, with the terms defined:

Three derivations follow, and they prove three of the five leaves — mean, median, and quantile. The other two are proved where they are used: the percentage-error leaf in Mape asymmetric by construction undefined at zero, and the log-loss/Brier leaf in Proper scoring rules log loss and brier.

Each derivation is the same move. Write down the average loss, differentiate with respect to c, set the derivative to zero, and read off which c solves it.

Squared error gives the mean. It answers “what value is right on average, so that overs and unders cancel?” Differentiate E[(Y - c)^2] with respect to c:

d/dc E[(Y - c)^2] = -2 E[Y - c] = 0   ->   c = E[Y]

Absolute error gives the median. It answers “what value is right for a typical row?” The function |Y - c| has derivative -1 when Y > c and +1 when Y < c, so pushing c upward is worth it exactly while more rows sit above it than below:

d/dc E|Y - c| = P(Y < c) - P(Y > c) = 0   ->   P(Y < c) = P(Y > c) = 0.5

Check both claims on the three customers from the intro. Their spends were 10, 20, 90, with mean 40 and median 20. Score the two candidate constants under both losses:

c = 40 (the mean)      c = 20 (the median)
  squared: (30^2 + 20^2 + 50^2)/3       squared: (10^2 + 0^2 + 70^2)/3
         = (900 + 400 + 2500)/3                = (100 + 0 + 4900)/3
         = 1266.67   <- LOWER                  = 1666.67

  absolute: (30 + 20 + 50)/3            absolute: (10 + 0 + 70)/3
          = 33.33                               = 26.67    <- LOWER

Squared error prefers 40. Absolute error prefers 20. Two metrics, same three rows, two different “correct” predictions.

Pinball loss gives any quantile you ask for. It answers “what value will the truth fall below tau of the time?”, which is the question behind every prediction interval. With l_tau(y, q) = max(tau·(y - q), (tau - 1)·(y - q)), the derivative in q is -tau when y > q and (1 - tau) when y < q. Setting the average derivative to zero, and writing F for the cumulative distribution function — F(q) is the fraction of rows whose true value falls at or below q:

-tau·P(Y > q) + (1 - tau)·P(Y < q) = 0
(1 - tau)·F(q) = tau·(1 - F(q))
F(q) = tau                       -> q is the tau-quantile

Set tau = 0.5 and pinball loss is MAE scaled by 1/2, which is why MAE gives the median — it is the symmetric special case. Quantile regression is not a different model family; it is the same model under a different metric.

Three assumptions sit behind those derivations, and all three have to hold.

1. The averages have to exist. A target with genuinely heavy tails — a distribution whose extreme values stay influential no matter how many rows you average, rather than being diluted by the bulk — can have no finite mean at all. Then squared error has no minimizer to converge to.
2. The evaluation rows must come from the same distribution as production rows. Every expectation above is taken over that distribution. If you resample, filter, or deduplicate your test set, you have changed the target the metric points at.
3. The median argument tacitly treats Y as continuous. On a small discrete sample, any value between the two middle observations achieves the same minimum, so the minimizer is a short interval rather than a point.

What interviewers probe: “MSE and MAE ranked my two models differently — which is right?” Neither. They are answering different questions. Ask what the number is used for: a budget that must be right on average needs the mean and therefore MSE; a promised delivery time that should be right for a typical order needs the median and therefore MAE.

2. Regression, with the disagreement computed

The mean-versus-median split becomes concrete with two constant predictors on five rows, ranked in opposite orders by two respectable metrics — and the arithmetic shows exactly why. Regression here just means the model outputs a number rather than a class label.

Two models, five rows, opposite rankings

Five actual insurance claim amounts: y = [1, 2, 2, 3, 50]. Four are small and one is large — that is the whole setup.

Two candidate models, each of which predicts the same number for every row. Model A predicts the mean, 11.6. Model B predicts the median, 2.0. The block below scores both under three metrics; read it by comparing the two columns row by row.

y       = [1, 2, 2, 3, 50]
mean(y) = 58 / 5 = 11.6        median(y) = 2.0

MODEL A predicts 11.6 for every row
  errors  |y - 11.6|:   10.6    9.6    9.6    8.6    38.4
  squared:             112.36  92.16  92.16  73.96  1474.56
  MSE  = (112.36 + 92.16 + 92.16 + 73.96 + 1474.56) / 5 = 1845.20 / 5 = 369.04
  RMSE = sqrt(369.04)                                                 =  19.21
  MAE  = (10.6 + 9.6 + 9.6 + 8.6 + 38.4) / 5           =   76.80 / 5 =  15.36

MODEL B predicts 2.0 for every row
  errors  |y - 2.0|:     1.0    0.0    0.0    1.0    48.0
  squared:               1.00   0.00   0.00   1.00  2304.00
  MSE  = (1 + 0 + 0 + 1 + 2304) / 5                     = 2306.00 / 5 = 461.20
  RMSE = sqrt(461.20)                                                  =  21.48
  MAE  = (1 + 0 + 0 + 1 + 48) / 5                       =   50.00 / 5 =  10.00

RMSE is the root mean squared error — the square root of MSE, taken so the number comes back into the units of the thing you are predicting (dollars, not dollars squared). It answers the same question MSE does; it is only easier to read.

The table collects those six numbers. The winner column is what matters: it flips.

The two metrics rank the two models in opposite orders, and neither is wrong.

MSE weights each row by its own error, so the single row at 50 contributes 1474.56 / 1845.20 = 80% of Model A’s total squared error and drags the fit toward itself.

MAE weights every row equally regardless of error size, so the four small rows outvote the one large one.

Note also that MSE(A) = 369.04 is exactly Var(y) — the variance of y, the average squared distance of the rows from their own mean. That is not a coincidence: Model A predicts the mean, so its squared error is the definition of variance. This identity is what R^2 is built on (R2 negative is meaningful and cross dataset comparison is not).

The RMSE/MAE ratio is a free diagnostic

RMSE is never smaller than MAE. Jensen’s inequality says that for a curved-upward function such as squaring, the average of the transformed values is at least the transform of the average. Applied to the absolute errors it gives E[e^2] >= (E|e|)^2, and taking square roots gives RMSE >= MAE.

Equality holds only when every residual has the same magnitude. A residual is the gap between a row’s true value and its prediction.

So the ratio of the two is free information about the shape of your errors:

RMSE / MAE = 19.21 / 15.36 = 1.25   (model A)
RMSE / MAE = 21.48 / 10.00 = 2.15   (model B)

A ratio near 1.0 means errors are uniform in size. A ratio above roughly 2 means a small number of rows own most of the squared error. Model B’s 2.15 is that signature: one row (the claim at 50) is carrying the metric.

Check those rows before touching the model, because a ratio that high is usually a data problem rather than a modelling one. The usual culprits:

• a unit error — grams recorded where kilograms were expected
• a sentinel value — a placeholder such as -999 or 9999 standing in for “missing” and being read as a real quantity
• a duplicated event counted twice

Huber loss is the deliberate compromise between the two. It is squared inside a band of width delta and linear outside it, so its minimizer sits between the mean and the median and slides toward the median as delta shrinks.

Here are the assumptions RMSE and MAE rest on.

Both assume the target lives on a scale where a fixed-size error means the same thing everywhere: that being $10 off is equally bad on a $20 claim and a $20,000 claim. When that is false — and for money, demand, and latency it usually is — you want a relative error instead, which is the subject of the next section.

Both also assume every row deserves an equal vote. If some rows are worth more (bigger customers, more recent weeks) you must weight the metric explicitly, because neither metric will do it for you.

3. MAPE — asymmetric by construction, undefined at zero

The most commonly requested error metric is the one you should most often talk someone out of. MAPE is the mean absolute percentage error: for each row, take the miss as a fraction of the true value, then average. It answers “on average, by what percentage was the forecast off?” — and it answers it badly, in two independent ways.

MAPE = mean(|y - p| / |y|). The per-row quantity |y - p| / |y| is the absolute percentage error, abbreviated APE below.

Pathology 1: it is bounded below and unbounded above

Fix the true value at 100 and vary the prediction. The block below reads as four separate predictions on that one row; note what happens to APE on the low side versus the high side.

truth y = 100 in every line below

predict   0  ->  APE = |100 -   0| / 100 = 1.00  -> 100%   <- worst possible under-forecast
predict  50  ->  APE = |100 -  50| / 100 = 0.50  ->  50%      (half the truth)
predict 200  ->  APE = |100 - 200| / 100 = 1.00  -> 100%      (double the truth)
predict 300  ->  APE = |100 - 300| / 100 = 2.00  -> 200%   <- no ceiling on this side

Predicting zero is the most wrong you can possibly be on the low side, and it costs 100%. Predicting double costs 100% too. Predicting triple costs 200%, and nothing stops it climbing.

So a 2x over-forecast is penalized twice as hard as a 2x under-forecast, and any model tuned on MAPE learns to forecast low. In a demand-planning system that bias is not a rounding artifact. It is systematic under-stocking, produced by the metric and invisible in the metric.

Which constant does MAPE pull toward?

Run the §1 trick on MAPE and you land on the 1/y-weighted median: the middle value after each row’s vote has been scaled by one over its true value. Here is that computation on the five claims from §2, y = [1, 2, 2, 3, 50]:

y:        1      2      2      3      50
weight
 = 1/y:   1.000  0.500  0.500  0.333  0.020      sum = 2.353
running
  total:  1.000  1.500  2.000  2.333  2.353

half the total weight = 2.353 / 2 = 1.177
the running total first passes 1.177 at y = 2   ->  weighted median = 2

The row at 1 casts a full vote. The row at 50 casts one fiftieth of a vote. The largest claim in the dataset has 2% of the influence of the smallest one — a strange thing to want from a claims model.

Scoring the two §2 models under MAPE confirms it, and the ranking is the reverse of MSE’s:

Model A (predicts 11.6):
  APEs = 10.6/1, 9.6/2, 9.6/2, 8.6/3, 38.4/50
       = 10.600  4.800  4.800  2.867   0.768     sum = 23.835
  MAPE = 23.835 / 5 = 4.767  ->  476.7%

Model B (predicts 2.0):
  APEs =  1/1,  0/2,  0/2,  1/3,  48/50
       = 1.000  0.000  0.000  0.333   0.960      sum =  2.293
  MAPE =  2.293 / 5 = 0.459  ->   45.9%

MAPE puts B (45.9%) far ahead of A (476.7%), because the four cheap rows outvote the expensive one by 50 to 1.

Pathology 2: it explodes near zero

Now a dataset of 1,000 rows where the model is almost perfect. 999 rows are predicted exactly right, each with y ≈ 100. One row has y = 0.02 and is predicted 0.50 — an absolute miss of 48 cents.

The denominator of that one row does all the damage:

APE on that row = |0.02 - 0.50| / 0.02 = 0.48 / 0.02 = 24.0  ->  2,400%

MAPE  = (2,400% + 999 zeros) / 1,000       = 2,400% / 1,000  =  2.4%
WAPE  = sum|y - p| / sum|y| = 0.48 / 99,900                  =  0.00048%

MAPE reports 2.4% error where WAPE reports 0.00048%. That is a 5,000x inflation (2.4 / 0.00048 = 5,000) produced by a single row, on a model whose total error across the whole dataset is 48 cents.

One row near zero can dominate the metric for the entire dataset. And y = 0 makes MAPE undefined outright, because the division then has a zero denominator.

The three standard repairs

• WAPE, the weighted absolute percentage error: sum the misses, sum the truths, divide once at the end. One division instead of a thousand.
• SMAPE, the symmetric mean absolute percentage error: divide by the average of the true and predicted values instead of by the truth alone.
• MAE on log(1+y): drop percentages entirely, since a fixed gap in log space is a fixed ratio in the original units.

The table prices each one. The last column is the one people skip.

The last row of that table contains a trap, and it needs two terms spelled out.

Monotone-equivariant means a summary commutes with any order-preserving transform. The median of log(1+y) is exactly the log of the median of y. So if you take a median in log space and transform back, you land on the plain median of y, unchanged. Taking MAE on the log therefore buys you nothing new about the target — you were already getting the median from plain MAE.

The mean has no such property, and that is what makes MSE on the log a genuinely different metric. Its minimizer back-transforms to the geometric mean — the n-th root of the product of n values, equivalently exp(mean(log y)) — which sits below the arithmetic mean on any spread-out positive target.

Reach for WAPE by default when someone asks for a percentage error. It is the only one of the three that cannot be destroyed by a single small denominator.

Here are the assumptions every percentage error rests on.

1. The target is strictly positive and comfortably away from zero, because dividing by the truth is the whole construction.
2. The target is on a ratio scale — “off by 10%” is a meaningful statement about the quantity. True of demand and revenue; false of temperature in Celsius, and false of any signed quantity.
3. You genuinely want small rows to carry as much weight as large ones. Where you do not, WAPE is the metric that says so out loud.

4. R^2 — negative is meaningful, and cross-dataset comparison is not

What R^2 really compares your model against explains both why it can go negative and why the same model can score 0.96 on one segment and 0.36 on another without getting any worse. R^2, the coefficient of determination, is often read aloud as “the fraction of variance explained.” It answers exactly one question: how much better is my model than a model that always predicts the average of the data I am scoring on?

R^2 = 1 - SSE / SST        SSE = sum (y_i - p_i)^2
                           SST = sum (y_i - ybar)^2     ybar = mean of the EVALUATION set

SSE is the sum of squared errors your model made. SST is the total sum of squares, the same quantity for the always-predict-the-average baseline. ybar is that average. So the ratio SSE / SST is “your squared error as a fraction of the baseline’s,” and R^2 is one minus it: 1.0 for a perfect model, 0.0 for a model exactly as good as the average, and negative for a model worse than the average.

R^2 compares your model to one specific baseline: the mean of the set you are scoring on. That baseline is not available at prediction time — you would have to already know the answers to compute it — which is what makes negative values possible and informative.

Trap 1: negative R^2 is a level shift, not a broken model

Take a test set y = [10, 12, 14, 16, 18]. Its mean is ybar = 14, so:

SST = (10-14)^2 + (12-14)^2 + (14-14)^2 + (16-14)^2 + (18-14)^2
    =    16     +     4     +     0     +     4     +    16      = 40

Now score three constant models against it. Only the predicted level changes between them.

predicts 14:  errors -4,-2, 0, 2, 4  ->  SSE =  16+ 4+0+ 4+16 =  40
              R^2 = 1 -  40/40 =  0.00     <- exactly as good as the baseline

predicts 15:  errors -5,-3,-1, 1, 3  ->  SSE =  25+ 9+1+ 1+ 9 =  45
              R^2 = 1 -  45/40 = -0.125    <- one point off, already negative

predicts 20:  errors -10,-8,-6,-4,-2 ->  SSE = 100+64+36+16+4 = 220
              R^2 = 1 - 220/40 = -4.50

Look at that last line next to its RMSE: sqrt(220/5) = sqrt(44) = 6.6. A model trained on a period when the mean was 20 and evaluated on a period when the mean is 14 scores a catastrophic-looking R^2 = -4.5 while being off by an ordinary 6.6 units.

Negative R^2 almost always means a shifted intercept, not a broken model. The intercept is the constant offset the model adds to every prediction, so a shifted one moves all predictions up or down together. The ranking may be fine and the errors may be small; only the level is wrong.

That is a drift signature — drift being the slow change of the live data away from the data you trained on — and it is chased down in 07 — Imbalanced Data, Calibration & Drift.

Trap 2: R^2 measures the target’s variance as much as your error

Same model, same RMSE of 2.0, scored on two customer segments. sd(y) is the standard deviation of the target — the typical distance of a row from the average — and SST/n is its square, the variance.

The only thing that changes between the two lines is how spread out the segment’s target is:

RMSE = 2.0 in both, so the model's mean squared error is 2.0^2 = 4.0

segment A:  sd(y) = 10.0  ->  SST/n = 10.0^2 = 100.00
            R^2 = 1 - 4/100.00 = 1 - 0.04 = 0.96
segment B:  sd(y) =  2.5  ->  SST/n =  2.5^2 =   6.25
            R^2 = 1 - 4/  6.25 = 1 - 0.64 = 0.36

Identical errors, R^2 differing by 0.60. Segment B is not a worse model. It is a segment whose target barely varies, so there is little variance available to explain, and a fixed error eats a much bigger share of it.

Never compare R^2 across datasets, across time periods, or across segments. Compare RMSE or MAE, which are in the units of the thing you predict.

Adjusted R^2 fixes a third, smaller problem

Training-set R^2 can only rise when you add a feature, even a purely random one, because the fit has more freedom and nothing penalizes the extra parameter. Adjusted R^2 puts that penalty back:

adjusted R^2 = 1 - (1 - R^2)(n - 1) / (n - k - 1)

  n = number of rows
  k = number of features

(Textbooks usually write p for the feature count. This chapter already uses p for the model’s prediction, so k it is.)

Adjusted R^2 is a within-dataset model-selection aid. It fixes neither of the two traps above.

Here are the assumptions R^2 rests on.

1. You want to be compared against the mean of the evaluation set, and that mean is a meaningful competitor. This fails the moment the evaluation set’s mean differs from the training set’s — trap 1.
2. Squared error is the right loss, since the whole identity is built from sums of squares.
3. The comparison stays inside one dataset. The moment two R^2 numbers come from sets with different target variance, the comparison is measuring the targets, not the models — trap 2.

5. The confusion matrix is the root; everything else is a ratio of its cells

Classification metrics are, all of them, ratios of four counts — and every later disagreement between accuracy, F1, ROC and PR traces back to which of the four counts each one uses. Classification means the model’s job is to sort rows into classes; take the binary case, where each row is either positive (fraud, disease, spam) or negative.

The four counts

Run the model over your evaluation rows. Each row is either positive or negative in truth, and the model either flagged it or did not, so each row falls into exactly one of four buckets.

The confusion matrix is those four counts arranged in a square. Columns are the truth, rows are what the model said. Read the diagonal (top-left, bottom-right) as “got it right” and the off-diagonal as the two kinds of mistake:

Everything else in this section — accuracy, precision, recall, F1, ROC, PR — is a ratio built from those four numbers and nothing more. The diagram below shows which slice of the square each metric divides by.

flowchart TD
    CM["Confusion matrix<br/>TP · FP · FN · TN"] --> R1["row: predicted positive<br/>TP + FP"]
    CM --> R2["column: actual positive<br/>TP + FN"]
    CM --> R3["column: actual negative<br/>FP + TN"]
    R1 --> PR["Precision = TP / TP+FP<br/>of what I flagged, how much was real"]
    R2 --> RC["Recall = TPR = TP / TP+FN<br/>of what was real, how much did I catch"]
    R3 --> FP["FPR = FP / FP+TN<br/>of the innocent, how many did I bother"]
    PR --> F1["F1 = harmonic mean<br/>of precision and recall"]
    RC --> F1
    RC --> ROC["ROC curve<br/>TPR vs FPR"]
    FP --> ROC
    PR --> PRC["PR curve<br/>precision vs recall"]

    style PR fill:#2d6a4f,color:#fff
    style RC fill:#40916c,color:#fff
    style FP fill:#bc6c25,color:#fff

Each of the three core rates takes TP or FP and divides it by one slice of the matrix — either a whole row or a whole column:

The rest of the diagram is what you build on top of those three:

• F1 is the harmonic mean of precision and recall — one number that demands both.
• The ROC curve plots TPR against FPR, so it lives entirely on the two columns.
• The PR curve (precision-recall curve) plots precision against recall, so it uses a row and a column.

Notice the fourth cell. TN feeds none of precision, recall, or F1. That looks like an oversight; it turns out to be the whole point later in this section.

The structural fact everything else follows from

Precision’s denominator is a row of the matrix. Recall’s and FPR’s denominators are columns.

This is the load-bearing fact for the next three sections.

The columns are fixed by the data: how many rows are truly positive and how many are truly negative is decided before your model runs. So recall and FPR are computed within a class, and adding more negatives to your dataset cannot change recall at all.

The top row is fixed by the model’s decisions, and it mixes both classes — TP from the positive column and FP from the negative one. So precision depends on how many negatives exist. Flood the dataset with negatives and precision falls even though the model is unchanged.

Every disagreement in Pr auc vs roc auc under heavy imbalance and the comparison pr auc is not allowed to make falls out of that. It is also the argument that cuts against PR-AUC: a metric built on precision inherits precision’s dependence on how rare positives are, so it cannot be compared across datasets with different rarities.

Accuracy under imbalance, in arithmetic

Accuracy is the fraction of all rows the model got right, (TP + TN) / total — the two diagonal cells over everything. It answers “how often is the model correct?”

Prevalence is the share of rows that are positive. The arithmetic below shows why accuracy is nearly worthless when prevalence is low.

The setup: 100,000 card transactions, 300 of them fraud. Prevalence is 300 / 100,000 = 0.003, or 0.3%. Compare a model that does nothing with a model that works.

Model NULL -- predict "not fraud" always, never flag anything
    TP =   0, FP =     0, FN = 300, TN = 99,700
    accuracy = (0 + 99,700) / 100,000 = 0.9970

Model X -- an actual fraud model
    TP = 240, FP = 1,960, FN =  60, TN = 97,740
    accuracy = (240 + 97,740) / 100,000 = 0.9798

Model X’s four counts, in the square from the top of this section:

The useless model beats the useful one by 1.7 accuracy points (0.9970 - 0.9798 = 0.0172).

The reason is in the weights. Accuracy is a weighted average of the per-class recall, with each class weighted by its prevalence. At 99.7% negatives, accuracy is 99.7% a measurement of the negative class and 0.3% a measurement of the thing you built the model for.

Read this before the numbers below start moving. The 100,000-transaction, 300-fraud population is reused twice more in this chapter, and each reuse is a different model, not Model X at a different threshold.

• Here in §5: Model X, recall 0.80, so TP = 240.
• In Pr auc vs roc auc under heavy imbalance and the comparison pr auc is not allowed to make: Models A and B, two other scorers, both held at recall 0.90, so TP = 270 each.
• In Choosing a threshold from the cost matrix: a third scorer whose default threshold gives recall 0.40, so TP = 120.

Same population, three different models, three different recalls. The counts are not meant to reconcile with each other, and a change in counts across sections is never a threshold move on one fixed model.

Two metrics survive this population where accuracy did not. F1 you have seen named; MCC, the Matthews correlation coefficient, is the correlation between the predicted labels and the true labels. It runs from -1 (perfectly wrong) through 0 (no better than chance) to +1 (perfect), and unlike F1 it reads all four cells.

Here they are on Model X. Every number below comes out of the four cells in the square above — 2,200 is the flagged row, 300 is the fraud column, 99,700 is the legitimate column, 97,800 is the not-flagged row:

precision = TP / (TP + FP) = 240 / 2,200 = 0.1091
recall    = TP / (TP + FN) = 240 /   300 = 0.8000

F1  = 2·P·R / (P + R)
    = 2(0.1091)(0.8) / (0.1091 + 0.8)
    = 0.17456 / 0.9091
    = 0.1920

MCC = (TP·TN - FP·FN) / sqrt( (TP+FP)(TP+FN)(TN+FP)(TN+FN) )
    = (240 · 97,740 - 1,960 · 60) / sqrt(2,200 · 300 · 99,700 · 97,800)
    = (23,457,600 - 117,600)      / sqrt(6.4354e15)
    = 23,340,000 / 80,221,000
    = 0.291

Model X catches 80% of the fraud, and its precision is 11% — so 9 out of every 10 things it flags are innocent. Both of those facts are visible in F1 = 0.192 and MCC = 0.291. Neither is visible in accuracy = 0.98.

Accuracy rests on two assumptions, and both are strong. It means what people think it means only when the classes are roughly balanced and the two kinds of mistake cost about the same. Both conditions fail in fraud, disease screening, and content moderation, which is most of the interesting classification problems.

Why F1 uses a harmonic mean, and what it deliberately ignores

F1 is the harmonic mean of precision and recall. The harmonic mean of two numbers is 2ab / (a + b) — a kind of average that sits much closer to the smaller of the two than an ordinary average does. F1 answers “is this model good at both flagging real cases and not crying wolf?”, and it refuses to accept a good score on one as payment for a bad score on the other.

F1 = 2PR / (P + R), writing P for precision and R for recall.

The harmonic mean is dominated by the smaller argument. Here is the extreme case — a model with perfect precision and almost no recall:

P = 1.00, R = 0.01

arithmetic mean = (1.00 + 0.01) / 2         = 0.505
F1 (harmonic)   = 2(1.00)(0.01) / (1.00 + 0.01) = 0.02 / 1.01 = 0.0198

That is a model that flags exactly one transaction and happens to be right about it. Perfect precision, useless. The arithmetic mean rates it “average” at 0.505. F1 rates it at 0.02.

F1 asks for both, and refuses to let one buy off the other.

The cell F1 cannot see

Now the property that makes F1 the imbalance metric: TN appears nowhere in its formula. Precision uses TP and FP; recall uses TP and FN. The fourth cell never enters.

Test it. Take Model X and dump in 900,000 extra easy negatives — legitimate transactions the model classifies correctly. Nothing about the model changed; you only enlarged the boring corner of the matrix.

             TP    FP     FN     TN        accuracy      F1      MCC
before      240  1,960    60    97,740      0.97980    0.1920   0.2909
after       240  1,960    60   997,740      0.99798    0.1920   0.2950
                              ^^^^^^^^^     ^^^^^^^    ^^^^^^
                              only this     jumps      does not
                              changed                  move

Accuracy improved by 1.82 points, from 0.97980 to 0.99798, purely because you added rows the model handles trivially. F1 did not move at all.

F1 survives imbalance precisely because it is blind to the cell that imbalance inflates.

MCC barely moves either — 0.291 -> 0.295 — even though it does read TN. It normalizes by all four marginal totals, so free true negatives inflate the numerator and the denominator together. That is the argument for MCC as your single number when you genuinely care about the negative class: it cannot be inflated by easy negatives the way accuracy just was, and it does not have to ignore TN to achieve that the way F1 does.

F_beta: buying recall with precision

F_beta = (1 + beta^2)·P·R / (beta^2·P + R) weights recall beta times as heavily as precision. Setting beta = 1 recovers F1.

• beta = 2 for screening, where missing a case is expensive.
• beta = 0.5 when a false alarm wastes an expensive human.

Choosing a threshold from the cost matrix shows that choosing beta is choosing a cost ratio, whether or not you meant to.

F1 rests on three assumptions.

1. The positive class is the one you care about and the negative class is uninteresting background. That is exactly why ignoring TN is a feature and not a bug.
2. The decision threshold is fixed. Precision and recall are both computed after the model has committed to a hard yes or no, so a model that looks worse on F1 may simply be sitting at a different point on the same curve.
3. A cost ratio between the two mistakes has already been chosen — silently, by whatever threshold you happen to be at. That is the trap taken apart in The threshold you chose by accident.

Macro, micro, and weighted averaging over classes

Everything above is binary. With three or more classes you compute precision or recall once per class and then have to combine those per-class numbers into one. How you combine them is a second metric choice hiding inside the first.

There are three conventions:

• Micro-averaging pools the raw counts across all classes first, then computes the ratio once at the end. Every row gets one vote.
• Macro-averaging computes the metric separately per class, then takes a plain unweighted average. Every class gets one vote regardless of size.
• Weighted averaging is the macro average with each class weighted by its row count, which lands back near the micro number.

They can disagree wildly. Here is a 1,000-row, three-class problem where class A dominates and class C is tiny. Compute both averages from the same three per-class recalls:

class   rows   correct   recall
  A      900     855      855/900 = 0.95
  B       80      40       40/80  = 0.50
  C       20       2        2/20  = 0.10

micro: pool the counts first
  = (855 + 40 + 2) / (900 + 80 + 20) = 897 / 1,000 = 0.897

macro: average the three recalls
  = (0.95 + 0.50 + 0.10) / 3         = 1.55 / 3    = 0.5167

Micro says 0.897, macro says 0.5167, on identical predictions.

Micro is 90% a report on class A, because class A is 90% of the rows. Macro gives the 20 rows of class C the same weight as the 900 rows of class A, so class C’s 0.10 recall drags the average down hard.

Neither is wrong. They answer different questions:

• Micro answers “what fraction of incoming rows will be handled correctly.” That is the operational question.
• Macro answers “does the model work for every class, including the rare ones.” That is the fairness and coverage question, and it is the one that catches a model quietly failing on a small but important category.

Report both, or state which one you mean. A bare “F1 = 0.87” on a multi-class problem is not a number.

This is the same weighting error as averaging per-region averages instead of re-aggregating the underlying totals, which shows up in SQL as the average-of-averages trap (Aggregation).

6. ROC-AUC is a probability, and here is the derivation

The most quoted classification number is worth deriving twice — once by counting pairs and once by measuring area — because seeing the two agree tells you exactly which model changes AUC can and cannot see.

The ROC curve — receiver operating characteristic, a name inherited from wartime radar — is what you get by sweeping the decision threshold from strict to lenient and plotting TPR against FPR at every setting. AUC is the area under that curve.

But the interview answer is not “area under the ROC curve.” It is:

AUC = P( score(a random positive) > score(a random negative) )

In words: pick one positive row and one negative row at random. AUC is the probability the model gave the positive the higher score.

So AUC answers “does this model put positives above negatives?” and nothing else. AUC 0.5 is coin-flip ordering. AUC 1.0 is perfect separation.

Why the area equals that probability

Sort all n rows by score, descending, and walk down the list drawing the ROC curve one step per row. Write P for the number of positives and N for the number of negatives.

• Each positive you pass moves you up by 1/P (you just recovered one more real case).
• Each negative you pass moves you right by 1/N (you just took one more false alarm).

Area accumulates only on the horizontal moves. A horizontal move has width 1/N, and it happens at height k_j / P, where k_j counts the positives already passed when you reach negative j. Add up the rectangles:

AUC = sum over negatives j of  (1/N) · (k_j / P)
    = (1 / (N·P)) · sum_j k_j
    = (number of (positive, negative) pairs with s_pos > s_neg) / (N · P)
    = P(s_pos > s_neg)                                  ties count 1/2

The middle line is the key step: sum_j k_j is literally a count of positive-negative pairs in which the positive scored higher, because each negative j contributes the number of positives sitting above it. Divide a count of winning pairs by the total number of pairs and you have a probability.

That count divided by N·P is the Mann-Whitney U statistic, a classical rank test that counts exactly these winning pairs.

AUC depends only on the ranks of the scores, and that has two consequences you will be asked about:

1. It is invariant under any strictly increasing transform of the score — any relabelling that preserves order. So calibrating a model, meaning rescaling its scores so that a score of 0.30 really does mean a 30% chance (the subject of Proper scoring rules log loss and brier), cannot change its AUC by even a thousandth.
2. It is invariant to class prevalence, because both of its axes are normalized within a single class — TPR by the positive column, FPR by the negative column, exactly the structural fact from The confusion matrix is the root everything else is a ratio of its cells.

The same seven rows, counted and then measured

Take three positives with scores 0.90, 0.60, 0.35 and four negatives with scores 0.80, 0.55, 0.40, 0.20.

First by counting pairs. There are 3 × 4 = 12 positive-negative pairs. Go through each positive and count how many negatives it outscores.

0.90 beats 0.80, 0.55, 0.40, 0.20   ->  4
0.60 beats       0.55, 0.40, 0.20   ->  3      (loses to 0.80)
0.35 beats             0.20         ->  1      (loses to 0.80, 0.55, 0.40)
                                        --
concordant pairs                         8     AUC = 8/12 = 0.6667

Now by geometry, walking the sorted list and accumulating area. Sorted descending, the labels read P N P N N P N. There are 3 positives, so each P steps up by 1/3 = 0.333; there are 4 negatives, so each N steps right by 1/4 = 0.25.

Read the trace below as the path of a staircase from (0,0) to (1,1):

step      point (FPR, TPR)
start     (0.00, 0.000)
0.90 P    (0.00, 0.333)
0.80 N    (0.25, 0.333)
0.60 P    (0.25, 0.667)
0.55 N    (0.50, 0.667)
0.40 N    (0.75, 0.667)
0.35 P    (0.75, 1.000)
0.20 N    (1.00, 1.000)

one rectangle per rightward move, width 0.25, height = TPR at that moment:
area = 0.25(0.333) + 0.25(0.667) + 0.25(0.667) + 0.25(1.000)
     = 0.0833     + 0.1667     + 0.1667     + 0.2500     = 0.6667

8/12 = 0.6667 by counting, 0.6667 by area. Identical, as the derivation says they must be.

Say the probabilistic form first in an interview, then note it equals the area. That ordering signals you know why the equality holds.

The function below is the pair-counting definition, written out literally rather than efficiently. Two of its lines deserve a close look.

def auc_by_pairs(scores, labels):
    """AUC as P(score(pos) > score(neg)), ties at half credit. O(n^2), for clarity."""
    pos = [s for s, y in zip(scores, labels) if y == 1]
    neg = [s for s, y in zip(scores, labels) if y == 0]
    if not pos or not neg:
        return float("nan")
    total = sum((a > b) + 0.5 * (a == b) for a in pos for b in neg)
    return total / (len(pos) * len(neg))

The double loop for a in pos for b in neg does one comparison per positive-negative pair. That is the definition rather than a shortcut, which is what O(n^2) means here: the work grows with the square of the number of rows.

The 0.5 * (a == b) term is the tie rule from the derivation. When a positive and a negative carry the same score, the model has expressed no preference between them. Half a pair is the only award that leaves a model with all-identical scores at AUC 0.5 rather than 0 or 1.

Production implementations use the Mann-Whitney identity instead — AUC = (sum of positive ranks - P(P+1)/2) / (N·P) after sorting once. Same count, O(n log n), the cost of a single sort.

Here are the assumptions AUC rests on.

1. Both classes are present in the evaluation set. With zero positives or zero negatives there are no pairs and the number is undefined, which is why the code returns nan.
2. You care about ordering across the whole score range, including the region above your operating threshold where you will never actually work. A model can buy AUC in a region you never visit.
3. The scores are comparable across rows. This fails if different rows were scored by different models, or under different feature availability.

7. PR-AUC vs ROC-AUC under heavy imbalance, and the comparison PR-AUC is not allowed to make

When should you prefer the precision-recall curve over the ROC curve? The answer — and, just as important, the one comparison PR-AUC is not allowed to make — comes out of a single algebraic identity.

Half of what follows is the condition. PR-AUC is the curve to prefer when two things are both true:

• the positives are rare and you operate at low FPR, and
• every model you are comparing was measured at the same prevalence.

Outside those conditions it is the wrong curve. The recommendation table at the end of this section gives ROC-AUC the first row, precisely because ROC-AUC is the one that survives a change of prevalence, and the section closes by showing that a PR-AUC quoted without its prevalence is not a number at all.

One identity generates everything here

Write precision in terms of the two rates and the prevalence pi. Start from TP / (TP + FP) and divide top and bottom by the row count n. Since TP/n = TPR · pi and FP/n = FPR · (1 - pi):

precision = (TPR · pi) / (TPR · pi + FPR · (1 - pi))

That formula is the whole section. Read what is in it and what is not: pi appears three times in precision, and it appears nowhere in TPR or FPR. Precision depends on prevalence; the ROC axes do not.

Two models the ROC curve says are nearly identical

The population is the one The confusion matrix is the root everything else is a ratio of its cells used — 300 positives, 99,700 negatives, pi = 0.003 — but Model A and Model B below are two new models, neither of them §5’s Model X.

Both are held at TPR = 0.90, so each catches TP = 270 of the 300 (against Model X’s 240 at recall 0.80). The only thing that differs between them is the false positive rate. Compare the last two columns:

On the ROC axes those two operating points are 0.040 - 0.010 = 0.030 apart. That is the entire visible difference between the two models on those axes.

The reason the ROC curve barely notices is the denominator. Model B produces 3,988 - 997 = 2,991 more false positives than Model A, but FPR divides that by 99,700, so 2,991 extra false alarms move the x-axis by three hundredths.

Precision divides the same 2,991 by a base of a few thousand, so it moves from 0.213 to 0.063. In operational terms: an analyst who reviewed 4.7 cases per caught fraud under Model A now reviews 15.8 — a 3.4x increase in workload for the same fraud caught.

The same story after aggregating the curves

Aggregating means integrating over all thresholds rather than reading one operating point. That is what turns a curve into a single number.

ILLUSTRATIVE -- see the note below before quoting any of these

                ROC-AUC     PR-AUC
Model A          0.982       0.31
Model B          0.971       0.12
difference       0.011       0.19

Those AUC figures are illustrative, not the output of a named scorer, and the same applies to the downsampled row further down. They are chosen to show the shape of the disagreement. Some score distribution does produce them, but not the one you would reach for first, so a reader who checks them against the obvious model will think the section is wrong.

Here is that check, so you do not have to run it. Take the textbook bi-normal model: negatives scored N(0,1), positives scored N(mu,1), which gives ROC-AUC = Phi(mu/sqrt 2). Set mu = 2.966 to land ROC-AUC on 0.982. That scorer has FPR 0.046 at TPR 0.90, not the 0.010 in the table above, and PR-AUC 0.51 at prevalence 0.003, not 0.31.

Use the numbers here for the argument they carry. Recompute both AUCs on your own scores before quoting either.

ROC-AUC says these models are within one percentage point of each other. PR-AUC says one does 2.6x the work of the other (0.31 / 0.12 = 2.6).

The random baseline is the second half of the argument, and it is where PR-AUC differs most from what people expect:

a coin flip scores:   ROC-AUC = 0.5
                      PR-AUC  = the prevalence itself = 0.003 here

So Model A’s PR-AUC of 0.31 is roughly a 100x lift over chance (0.31 / 0.003 = 103), while its ROC-AUC of 0.982 is a 2x lift over chance. The PR curve spends its resolution in the region you actually operate in.

The catch: PR-AUC is not comparable across prevalences

State this before the interviewer does. PR-AUC is not comparable across datasets with different prevalence, for exactly the reason the identity showed: precision contains pi.

Score one unchanged model twice — once on the full population, once on a set where you downsampled negatives to 1:1. Downsampling means throwing away most negative rows to build a balanced evaluation set.

                        prevalence   ROC-AUC   PR-AUC
full population            0.003      0.982     0.31
negatives downsampled      0.500      0.982     0.93

Nothing about the model changed — same weights, same scores, same ordering. ROC-AUC is identical at 0.982. PR-AUC went from 0.31 to 0.93, roughly tripling.

A PR-AUC quoted without the prevalence it was measured at is not a number.

This is also the most common way a resampled evaluation set produces a triumphant, meaningless metric — see Resampling and the calibration it breaks.

Which curve for which situation

Three terms in the table need defining first. AP, average precision, is the standard way to compute the area under a PR curve. Precision@k is precision measured only over the top k scored rows. Partial AUC integrates the ROC curve over a restricted slice of the FPR axis instead of all of it.

Find your situation in the left column and take the metric named there.

Here are the assumptions the two curves rest on.

PR-AUC means what you think it means only if the prevalence of your evaluation set equals the prevalence you will face in production. Any resampling, any per-day scoring on a day with unusual traffic, any evaluation restricted to hard cases — and the number moves without the model moving.

ROC-AUC carries the complementary assumption. It is safe across prevalences precisely because it ignores them, which also means it will never tell you whether your operating point is affordable. Models A and B above are the proof: 0.011 apart on ROC-AUC, 3.4x apart on analyst headcount.

8. Proper scoring rules: log loss and Brier

Two metrics judge a model’s numbers rather than its ordering, and neither can be gamed. Both take a predicted probability and a 0/1 outcome and score how well the number matched reality — which is what finally settles a contest where one model wins on ordering and the other wins on numbers.

Before the metrics, two words this chapter and the next one both lean on. They name two different things a probability model can be good or bad at, and they are independent — you can have either without the other.

Calibration: are the numbers right? A model is calibrated when its stated probabilities match observed frequencies. Take every row it scored 0.30. If close to 30% of them really turn out positive, it is calibrated at 0.30. If 60% of them turn out positive, it is not — the number 0.30 is a lie regardless of how the rows were ordered.

Discrimination: is the ordering right? A model discriminates when it separates the classes at all — pushing risky rows toward 1 and safe rows toward 0, rather than emitting the same number for everybody. ROC-AUC in Roc auc is a probability and here is the derivation is a pure discrimination measure: it reads only order.

The independence is easy to see in one example. A model that predicts the base rate 0.03 for every single row in a 3%-fraud population is perfectly calibrated — among the rows it scored 0.03, exactly 3% are fraud — and has zero discrimination, because it ranks nothing above anything. It is worthless and calibration cannot tell you so.

What makes a scoring rule proper

A scoring rule is proper if reporting your true belief is optimal. You cannot score better by shading your answer toward 0 or 1 to look more decisive.

That is a property you verify by differentiation, not a definition to memorize. Both derivations below follow the same shape: write the expected loss when the truth is p and you report q, differentiate with respect to q, set it to zero, and check that q = p falls out.

Log loss, also called cross-entropy, charges you -log(q) when the event happens and -log(1-q) when it does not, where q is the probability you reported. With true probability p and reported q, expected loss is -p·log q - (1-p)·log(1-q):

d/dq = -p/q + (1-p)/(1-q) = 0
   -> p(1-q) = q(1-p)  ->  p - pq = q - pq  ->  q = p

The Brier score is simply the squared error applied to probabilities: (q - outcome)^2, averaged. Expected loss is p(1-q)^2 + (1-p)q^2:

d/dq = -2p(1-q) + 2(1-p)q = 0
   -> -p(1-q) + (1-p)q = 0
   -> -p + pq + q - pq = 0
   ->            q - p = 0     ->  q = p

Both are minimized at the truth, so both reward honest probabilities.

Accuracy, F1, and precision are not proper. Each is maximized by a whole family of distorted probabilities, which is why a model tuned to maximize F1 emits numbers you must not read as probabilities.

How the two rules differ: the tail

Log loss is unbounded — as q approaches 0 on a row that turns out positive, -log(q) runs to infinity. Brier is bounded by 1, since it squares a difference between two numbers in [0,1].

So being confidently wrong is survivable under one and not the other. One row, predicted 0.999, truth 0:

predicted 0.999, truth 0:   log loss contribution = -ln(0.001) = 6.908
                            Brier contribution    =  0.999^2   = 0.998

1,000 rows, that one confident error:
   log loss:  6.908 / 1,000 = 0.0069   added to a typical base of ~0.10  (+7%)
   Brier:     0.998 / 1,000 = 0.0010   added to a typical base of ~0.05  (+2%)

Log loss is the metric to pick when a confident mistake is catastrophic. Brier is the metric to pick when you want a bounded, outlier-resistant proper score.

The Brier decomposition puts calibration in its place

Brier splits cleanly into three terms (a result due to Murphy), and the split is worth knowing because it shows exactly where calibration sits — as one of three terms, not as the whole score.

Brier = reliability - resolution + uncertainty

The base rate pbar is the overall share of rows that are positive — the same quantity called prevalence in The confusion matrix is the root everything else is a ratio of its cells.

Now run the decomposition on a base rate of 0.20, where uncertainty = 0.20(0.80) = 0.16:

constant predictor, always emits 0.20:
    reliability = 0     (it is perfectly calibrated -- 20% of its rows are positive)
    resolution  = 0     (it never moves off the base rate, so it separates nothing)
    Brier = 0.16 - 0 + 0 = 0.16

a real model:
    reliability = 0.012   (slightly miscalibrated)
    resolution  = 0.070   (it does separate the classes)
    Brier = 0.16 + 0.012 - 0.070 = 0.102

The constant predictor is perfectly calibrated and worth nothing. Its zero reliability term buys it nothing, because its zero resolution term subtracts nothing.

That is the bridge to Calibration what it means and when it matters: optimizing calibration alone is entirely compatible with a useless model.

Here are the assumptions both scoring rules rest on.

1. The model’s output is meant to be read as a probability. False for any model trained on rebalanced data or tuned against F1.
2. No prediction is ever exactly 0 or 1 (log loss only), since -log(0) is infinite. Implementations clip to something like 1e-15, and that clip value silently becomes part of your metric.
3. The outcome you score against is the outcome you care about. A proper scoring rule on a badly defined label is still a precise measurement of the wrong thing.

Model A wins AUC, model B wins log loss

This is the discrimination-versus-calibration split from the top of the section, made numeric.

Ten rows, five positive. Model A ranks perfectly but squashes every score toward 0.5 — great discrimination, terrible calibration. Model B is well calibrated but makes one ranking mistake — the reverse.

       label:    1     1     1     1     1     0     0     0     0     0
Model A:       0.56  0.55  0.54  0.53  0.52  0.48  0.47  0.46  0.45  0.44
Model B:       0.95  0.90  0.85  0.80  0.30  0.35  0.10  0.10  0.05  0.05
                                        ^^^^  ^^^^
                                        B's one inversion: a positive
                                        scored below a negative

AUC first. There are 5 × 5 = 25 positive-negative pairs.

AUC(A): every positive outranks every negative        ->  25/25 = 1.000
AUC(B): the positive at 0.30 loses to the negative at 0.35,
        and beats the other four negatives (0.10, 0.10, 0.05, 0.05).
        The other four positives beat all five negatives = 20 pairs.
        20 + 4 = 24                                  ->  24/25 = 0.960

Now the probability scores. Each row contributes -ln(q) if its label is 1 and -ln(1-q) if its label is 0, so the five negative rows enter through 1 - q:

log loss(A) = -(1/10)[ ln.56 + ln.55 + ln.54 + ln.53 + ln.52     <- the 5 positives
                     + ln.52 + ln.53 + ln.54 + ln.55 + ln.56 ]   <- 1-q for the 5 negatives
            = 6.1653 / 10 = 0.6165
log loss(B) = -(1/10)[ ln.95 + ln.90 + ln.85 + ln.80 + ln.30
                     + ln.65 + ln.90 + ln.90 + ln.95 + ln.95 ]
            = 2.4904 / 10 = 0.2490

Brier(A)    = [(1-.56)^2 + ... + (1-.52)^2 + .48^2 + ... + .44^2] / 10
            = 2.1180 / 10 = 0.2118
Brier(B)    = 0.7125 / 10 = 0.0713

Collecting the results, the two models win on different metrics:

The part that decides it is not “it depends on the use case”:

A’s defect is repairable and B’s is not.

AUC is invariant to any strictly increasing transform of the scores (Roc auc is a probability and here is the derivation). So you can stretch A’s compressed scores out toward 0 and 1 without touching its AUC of 1.000, and its log loss drops. Fit a monotone calibrator on held-out data and you get AUC 1.000 and a low log loss.

A monotone calibrator is a function that stretches scores without reordering them, fitted so the stretched numbers match observed frequencies. The three standard choices:

• Platt scaling — fit a logistic curve to the scores.
• Isotonic regression — fit any non-decreasing step function.
• Temperature scaling — divide the pre-sigmoid scores by one learned constant.

There is no function you can apply to B’s scores that unswaps 0.30 and 0.35, because a monotone map preserves order by definition. B’s 0.960 is permanent.

So the shipping decision:

• Ship A, calibrated. Ranking is the hard-won part; the numbers are a repairable defect. This is the default.
• Ship B as-is only when you have no held-out data to fit a calibrator on, or when the deployment consumes the probability today and cannot wait.
• Ship A raw when nothing downstream reads the number — a ranked queue, a top-k feed, a fixed-capacity review list. Then log loss is measuring a property nobody uses.

What interviewers probe: “Your AUC went up and your log loss got worse. What happened?”

Something changed the scale of the scores without changing their order. The usual suspects:

• a class-weight change — telling the trainer that one class’s rows count for more
• a resampled training set
• a new loss function
• a shrinkage parameter change. Shrinkage is the step size in boosting — the technique of fitting a sequence of small models, each trained on what its predecessors still get wrong, then summing them — and it pulls each round’s contribution toward zero.

Check the score histogram (the distribution of the scores the model emits) before you go looking at the model. AUC is rank-only, so it cannot see the change that log loss just reported.

9. Ranking and recommendation metrics

When the model returns an ordered list rather than a number or a label — search results, feeds, recommendation slates (a slate being the fixed set of items shown together in one placement) — the metrics change, and each of them can see only part of the list. Which part is exactly what you will be asked.

Ranking metrics answer a different question from classification metrics. Not “is this item relevant” but “did the relevant items end up near the top.”

The input is a list of items sorted by model score. The output is a number saying how good that ordering was, judged against labels marking which items were actually relevant.

One term recurs below. The corpus is the full collection the list was drawn from, so “3 relevant in the corpus” means three items exist anywhere that should have been returned — including any the model failed to retrieve.

Precision@k and recall@k

• P@k = the fraction of the top k that are relevant. It answers how much of what I showed was worth showing.
• R@k = the fraction of all relevant items that made the top k. It answers how much of what existed did I manage to surface.

One query, 10 results returned, with relevant items sitting at ranks 1, 3, and 6. Three relevant items exist in the corpus in total, so all three were found somewhere in the ten.

ranks:      1    2    3    4    5    6    7    8    9   10
relevant:   *         *              *

P@5  = (relevant in top 5) / 5  = 2/5  = 0.40
R@5  = (relevant in top 5) / 3  = 2/3  = 0.667
P@10 = 3/10 = 0.30
R@10 = 3/3  = 1.000

P@k is capped at min(k, #relevant)/k. A query with only 2 relevant documents in the whole corpus can never exceed P@10 = 2/10 = 0.2, no matter how good the ranker is.

So averaging P@10 across queries with different numbers of relevant documents averages together different ceilings, and a query’s score partly reports how many relevant documents happened to exist. That is the failure P@k hides.

R@k has no such problem: its denominator is the same for every scoring of the same query, which is exactly what makes it averageable. That is why R@k is the metric of choice for the retrieval stage of a RAG system — retrieval-augmented generation, where a search step fetches documents and a language model answers from them. Anything the search step misses is unrecoverable downstream (agents track, Diagnosing a rag agent in the right order — that is the agents chapter 08, not ml/08).

MRR: where was the first hit?

MRR is the mean reciprocal rank: the mean of 1 / (rank of the first relevant item), with a score of 0 for a query that returned nothing relevant. It answers “how far down did the user have to look before finding something useful?”

Three queries, with the first hit at rank 1, at rank 3, and nowhere at all:

query 1: first hit at rank 1     ->  1/1 = 1.0000
query 2: first hit at rank 3     ->  1/3 = 0.3333
query 3: no relevant item found  ->        0.0000

MRR = (1.0000 + 0.3333 + 0.0000) / 3 = 1.3333 / 3 = 0.4444

MRR sees only the first hit and nothing after it. A list with one relevant item at rank 1 and a list with ten relevant items starting at rank 1 score identically at 1.0.

Use MRR when there is exactly one right answer — a lookup, a navigational query, a “which document answers this” retrieval. Never for a feed.

MAP: precision that pays for early hits

MAP is the mean average precision. Two steps, and the names are confusingly similar, so keep them apart:

1. AP (average precision) is computed for one query. Look at each rank that holds a relevant item, compute P@i there, and average those values over the number of relevant items.
2. MAP is the average of AP over all queries.

Here it is on two queries. Notice that in query 1 the first relevant item is at rank 1, so it contributes a full 1.0:

query 1: relevant at ranks 1, 3, 6 (3 relevant total)
   at rank 1: P@1 = 1/1 = 1.0000     (1 relevant seen out of 1 shown)
   at rank 3: P@3 = 2/3 = 0.6667     (2 relevant seen out of 3 shown)
   at rank 6: P@6 = 3/6 = 0.5000     (3 relevant seen out of 6 shown)
   AP  = (1.0000 + 0.6667 + 0.5000) / 3 = 2.1667 / 3 = 0.7222

query 2: relevant at ranks 2, 4 (2 relevant total)
   at rank 2: P@2 = 1/2 = 0.5000
   at rank 4: P@4 = 2/4 = 0.5000
   AP  = (0.5000 + 0.5000) / 2 = 0.5000

MAP = (0.7222 + 0.5000) / 2 = 0.6111

AP is a position-weighted recall. Each relevant item contributes the precision achieved at the moment it was found, so finding it early contributes more — query 1 scores higher largely because it got a hit at rank 1.

AP handles binary relevance only: every item is either relevant or not, with no grades in between. That limitation is what the next subsection fixes.

NDCG, with the discount derived

NDCG is the normalized discounted cumulative gain, and it is the metric to reach for when relevance comes in grades rather than yes/no. It assembles from two pieces, and its logarithmic discount deserves a derivation of its own, because that choice encodes a claim about your users.

Graded relevance means each item carries a score like 0, 1, 2, or 3 rather than a yes/no. Building a metric out of that needs two pieces:

• a gain function, which converts a relevance grade into value, and
• a position discount, which shrinks that value according to where the item sat in the list.

DCG@k = sum over i=1..k of  gain(rel_i) · discount(i)

gain(rel)   = 2^rel - 1        exponential: grade 3 -> 7, grade 2 -> 3, grade 1 -> 1, grade 0 -> 0
discount(i) = 1 / log2(i + 1)  1.0 at rank 1, falling slowly after that

The exponential gain is a deliberate claim about value. A grade-3 document is worth 7 and a grade-2 is worth 3, so one excellent result outweighs two merely good ones. A linear gain would say the opposite. Which of those matches your product is a question worth asking before you accept the default.

DCG is the discounted cumulative gain — the running total of discounted gains down the list. It is not yet comparable across queries, which is what the normalization step below fixes.

Why a logarithmic discount and not 1/i

The discount encodes a user model: the probability the user is still scanning when they reach rank i. Three requirements pin down its shape.

1. It must equal 1 at rank 1.
2. It must decrease as i grows.
3. Its marginal decrease must itself shrink, so a swap deep in the list matters less than a swap at the top but never becomes exactly free.

Both 1/log2(i+1) and 1/i satisfy all three. The table below shows how differently they treat the deep end of the list — compare the two columns at rank 100.

1/i models an impatient user: rank 100 is worth 1% of rank 1, so the metric is nearly blind past the first page. 1/log2(i+1) models a patient one: rank 100 still carries 15% of rank 1’s weight. The log discount was chosen deliberately as the conservative option — it keeps the metric sensitive to the whole list, which is what you want when the list is a search result page and not a single answer.

Here is NDCG@5 worked end to end. Take a retrieved list of five items with relevance grades [3, 2, 0, 1, 2], where 3 means highly relevant and 0 means irrelevant. Note that this ordering is not ideal — the grade-2 item is stuck at rank 5 while a grade-0 item sits at rank 3.

Compute the actual DCG first. Each row is gain × discount:

rank   rel   gain = 2^rel - 1   discount = 1/log2(i+1)   contribution
  1     3           7                  1.0000               7.0000
  2     2           3                  0.6309               1.8928
  3     0           0                  0.5000               0.0000
  4     1           1                  0.4307               0.4307
  5     2           3                  0.3869               1.1606
                                              DCG@5    =   10.4841

ideal ordering by grade: [3, 2, 2, 1, 0]  ->  gains [7, 3, 3, 1, 0]
   IDCG@5 = 7(1.0) + 3(0.6309) + 3(0.5) + 1(0.4307) + 0 = 10.8235

NDCG@5 = 10.4841 / 10.8235 = 0.9686

IDCG is the ideal DCG: the score the best possible ordering of these same five items would have achieved, which is just the grades sorted descending. Dividing by it is the “normalized” in NDCG.

That division is what makes NDCG comparable across queries. A query with three highly relevant documents and a query with one both top out at 1.0, so averaging NDCG across queries does not average different ceilings the way P@k did.

Pricing the discount: where a swap hurts

Start from the ideal list [3,2,2,1,0], which scores NDCG = 1.000 by construction, and swap one adjacent pair. Do it once at the top and once four positions down, and compare the damage:

swap ranks 1 and 2:  [2,3,2,1,0]  DCG =  9.3472  ->  NDCG = 0.8636   loss 0.1364
swap ranks 4 and 5:  [3,2,2,0,1]  DCG = 10.7796  ->  NDCG = 0.9959   loss 0.0041

The same kind of swap costs 33x more at the top of the list than four positions down (0.1364 / 0.0041 = 33). That factor decomposes into two pieces:

position discount:  (d1 - d2) / (d4 - d5)
                  = (1.0000 - 0.6309) / (0.4307 - 0.3869)
                  = 0.3691 / 0.0438  =  8.4x

gain gap:           ranks 1-2 exchange gains 7 and 3  ->  a gap of 4
                    ranks 4-5 exchange gains 1 and 0  ->  a gap of 1
                                                      =  4x

8.4 × 4 = 34, which is the 33 above up to rounding.

That gradient is the entire reason learning-to-rank models optimize an NDCG surrogate — a differentiable stand-in for a metric that is itself full of flat steps — rather than a pointwise loss that scores each item on its own.

LambdaRank makes it literal. It is a learning-to-rank method that declines to write a loss function at all and specifies the gradient directly: its lambda for a pair of documents is the NDCG change that swapping them would cause. A pointwise loss treats every position equally, and no user does.

The two functions below implement exactly the arithmetic above. One line is worth reading twice: log2(i + 2). Python’s enumerate starts at 0, so the item at rank 1 arrives with i = 0. i + 2 is the 0-indexed spelling of the i + 1 in the formula, not an off-by-one bug.

from math import log2

def dcg(rels, k=None):
    """Exponential gain, log2 position discount. rels is in ranked order."""
    rels = rels[:k] if k else rels
    return sum((2 ** r - 1) / log2(i + 2) for i, r in enumerate(rels))

def ndcg(rels, k=None):
    ideal = sorted(rels, reverse=True)
    denom = dcg(ideal, k)
    return dcg(rels, k) / denom if denom else 0.0

Here are the assumptions every ranking metric rests on.

1. The relevance labels are complete for the items being scored. An unjudged item is treated as irrelevant, so a model that surfaces a genuinely good document nobody labelled is punished for it.
2. Relevance is a property of the item alone, independent of the rest of the list. This rules out redundancy: ten copies of the same perfect answer score as ten perfect results.
3. NDCG’s discount matches your users. On a mobile feed where nobody scrolls past rank 5, the log discount is measuring attention that is not being paid.

10. Why offline ranking metrics disagree with online CTR

A ranking model can improve on every offline metric and still lose money in production. CTR is the click-through rate, the fraction of shown items that get clicked, and it is the number the business usually watches.

You improve NDCG@10 by 8% offline, ship it, and CTR drops. This is the single most common surprise in recommender systems, and it has three distinct mechanisms — position bias, the feedback loop, and a label mismatch. Each gets its own subsection below.

All three mechanisms live on one cycle. Follow the arrows around it and note that the training data is produced by the very model it will train.

flowchart TD
    M["Ranking model"] --> S["Chooses which items<br/>get shown, and where"]
    S --> E["Position decides<br/>P of being examined"]
    E --> C["Clicks logged<br/>click = examined AND relevant"]
    C --> T["Next training set"]
    T --> M
    S --> U["Items never shown<br/>get zero clicks<br/>-> look irrelevant<br/>-> never shown"]
    U --> T

    style E fill:#bc6c25,color:#fff
    style U fill:#9d0208,color:#fff
    style T fill:#1d3557,color:#fff

Read it as a cycle. The ranking model chooses which items get shown and where. Position decides the probability of being examined at all. The clicks that get logged record only the cases where the item was examined AND relevant, so the log conflates the two. Those clicks become the next training set, which trains the next ranking model, closing the loop.

The red branch is the leak. Items never shown get zero clicks, therefore look irrelevant, therefore are never shown. No amount of retraining on that log can recover them, because the log contains no evidence about them either way.

Mechanism 1: position bias

The standard click model factorizes a click into two independent events — the user looked at the slot, and the item in it was worth clicking:

P(click | item d at rank i) = P(examine | i) · P(relevant | d)
                              ^^^^^^^^^^^^^   ^^^^^^^^^^^^^^^
                              about the slot   about the item

P(examine | i) is called the propensity: how likely a slot is to be looked at, independent of what sits in it.

Propensities are measured by swap experiments, where the same items are deliberately shuffled between positions so the position effect can be isolated from the item effect. Typical measured values:

rank:        1     2     3     4     5
propensity: 1.00  0.65  0.50  0.40  0.33

Now take one item whose true relevance is 0.30 and put it in two different slots:

shown at rank 1:  CTR = 1.00 × 0.30 = 0.300
shown at rank 5:  CTR = 0.33 × 0.30 = 0.099

A 3x difference in logged CTR from an identical item. Train on raw clicks and the strongest signal in the data is where the previous model chose to put each item. The logged CTR of an item is largely a measurement of the incumbent ranker.

The correction is inverse-propensity scoring, abbreviated IPS. Weight each observation by 1 / P(examine | i), so an impression in a rarely-examined slot counts for more:

click at rank 5, weighted:  0.099 × (1 / 0.33) = 0.300     <- relevance recovered

IPS is unbiased: over enough samples its estimate lands on the true value rather than sitting systematically above or below it. But its variance explodes as propensities approach zero, and the deeper down the list you go the worse it gets.

To see how far down that starts to bite you need to extrapolate, since the measured table stops at rank 5. Those five values follow a power law closely — P(examine | i) ≈ i^-0.65 reproduces every one of them to within 0.022. Continue it:

rank  40:  propensity ≈ 40^-0.65 = 0.09   ->  weight ≈ 1/0.09 =  11
a slot examined 1% of the time:  0.01      ->  weight        = 100

At weight 100, a single observation outvotes a hundred others. That is why propensities are clipped at a floor in practice — trading a little bias for a large reduction in variance.

Mechanism 2: the feedback loop closes over the unshown

IPS fixes the bias among items that were displayed. It can say nothing at all about an item that was never displayed, because there is no observation to reweight — 1/0 is not a weight.

That item accumulates zero clicks, looks irrelevant to the next training run, and is buried further. This is the red branch of the diagram.

The fix is structural, not statistical. Reserve a small randomized slot — 1-5% of impressions, an impression being one occasion on which an item was displayed to a user — whose ordering is uniform rather than model-chosen. That stream does two jobs: it estimates propensities, and it gives you an unbiased sample of the full catalog including items the ranker would never have surfaced.

Mechanism 3: the offline label and the online reward measure different things

Offline NDCG uses human relevance grades on a fixed judged pool — a frozen set of query-document pairs someone rated once.

Online CTR is measured on the live catalog and is driven by thumbnail, brand familiarity, and price as much as by relevance.

Those are not the same quantity, and here is a disagreement in the shape you will actually see it:

variant        offline NDCG@10   week-1 CTR   week-4 CTR
control              0.412          3.10%        3.08%
new ranker           0.447          2.94%        2.71%
                     ^^^^^          ^^^^^        ^^^^^
                     wins           loses        loses more

The new ranker genuinely wins on graded relevance and loses on clicks — and then loses more over four weeks.

The four-week decay is the loop from the diagram doing its work. The new ranker promoted long-tail items — the many rarely-shown items that individually get little traffic — which were unfamiliar, so they drew fewer clicks, so the next training round saw them as worse, so they got shown less.

What to do about it

The conclusion is not “offline metrics are useless.”

Offline metrics measure relevance under a fixed judged pool. The online metric measures relevance times exposure times familiarity under a distribution your own model controls. They are different measurements, and each is correct about its own quantity.

Use offline NDCG as a gate — a change that drops it is probably broken. Use an online experiment as the decision:

• An A/B test splits users into two groups and serves each group a different ranker.
• Interleaving merges two rankers’ results into a single list shown to every user, then attributes each click to whichever ranker contributed that item.

Interleaving is preferred here because merging the two lists holds position bias approximately constant between them — both rankers’ items land in the same mix of slots.

Here are the assumptions every correction above rests on.

1. The click model actually factorizes as examination times relevance. This breaks when a user’s decision to click depends on the neighbouring items.
2. The propensities are known or well estimated, and every item had some chance of being shown. Where that chance is exactly zero, no reweighting exists.
3. The judged pool covers what the new model wants to surface. This is false by construction whenever the new model is genuinely different from the one that built the pool.

11. Choosing a threshold from the cost matrix

A classifier outputs a probability. A decision needs a threshold, and the threshold is not a modelling choice — it is an arithmetic consequence of what mistakes cost. Turning a probability into an action is a calculation, not a judgement call.

Here is the derivation. The cost matrix is the pair of numbers naming what each kind of mistake costs: C_fp for a false positive and C_fn for a false negative. For a row with true probability p, compare the two expected costs:

predict positive:  cost = (1 - p) · C_fp        you are wrong when the row is negative
predict negative:  cost =      p  · C_fn        you are wrong when the row is positive

predict positive when   (1 - p)·C_fp  <  p·C_fn
                         C_fp - p·C_fp <  p·C_fn
                         C_fp          <  p·(C_fn + C_fp)

                      p  >  C_fp / (C_fp + C_fn)   =   threshold*

threshold* = C_fp / (C_fp + C_fn).

Two things to notice about that result.

Equal costs give C_fp / (2·C_fp) = 0.5. That is where the familiar default 0.5 comes from. It is a statement about costs, not a fact about probability, and it is almost never the right statement.

Prevalence is absent from the formula. That is not an oversight. The prevalence is already inside p — a rare event simply gets a low probability. The threshold only encodes the exchange rate between the two mistakes.

The diagram below is the whole procedure as a set of questions. Answer them top to bottom, but check the right-hand branch first, because it overrides everything else.

flowchart TD
    A["What does a false positive cost?<br/>C_fp"] --> T["threshold* = C_fp / C_fp + C_fn"]
    B["What does a false negative cost?<br/>C_fn"] --> T
    T --> CAL{"Is p calibrated?"}
    CAL -->|yes| USE["Apply threshold* directly.<br/>It is in probability units."]
    CAL -->|no| FIX["Calibrate first, or sweep the<br/>raw score and pick the operating<br/>point that minimizes measured cost"]
    T --> CAP{"Is capacity the<br/>real constraint?"}
    CAP -->|yes| Q["Ignore threshold*.<br/>Take the top K by score.<br/>Metric becomes precision@K."]

    style T fill:#1d3557,color:#fff
    style USE fill:#2d6a4f,color:#fff
    style FIX fill:#bc6c25,color:#fff
    style Q fill:#40916c,color:#fff

In words, the diagram is four questions:

1. What does a false positive cost? (C_fp)
2. What does a false negative cost? (C_fn) — those two give threshold* = C_fp / (C_fp + C_fn).
3. Is p calibrated — does a score of 0.30 really correspond to a 30% chance? If yes, apply threshold* directly, because it is in probability units. If no, calibrate first, or sweep the raw score and pick the operating point that minimizes measured cost.
4. Is capacity the real constraint? If reviewers can only handle a fixed number of cases, ignore threshold* entirely, take the top K by score, and the metric becomes precision@K. When capacity is not binding, this branch does not fire and the threshold stands.

Running the rule on the fraud numbers

Price the two mistakes first.

A missed fraud costs the chargeback — the amount the bank claws back from the merchant when a cardholder disputes the transaction — plus fees. Call it C_fn = $220.

A false positive costs three minutes of an analyst’s time. Call it C_fp = $4.

threshold* = C_fp / (C_fp + C_fn) = 4 / (4 + 220) = 4 / 224 = 0.0179

That is 28x lower than the default 0.5, which is the whole point.

Now score it on 100,000 transactions with 300 frauds — the same population The confusion matrix is the root everything else is a ratio of its cells and Pr auc vs roc auc under heavy imbalance and the comparison pr auc is not allowed to make used, scored here by a third model, weaker than either. Take its score distribution as a premise:

• above 0.50 it puts 120 of the 300 frauds, along with 200 false alarms
• above 0.018 it puts 285 of the 300 frauds, along with 6,000 false alarms

At its default threshold that is recall 120/300 = 0.40, against Model X’s 0.80 in §5 and Models A and B’s 0.90 in §7. Everything below is arithmetic on those two premises.

t = 0.50 (the default):
    TP = 120, FN = 180, FP =   200
    cost = 180($220) + 200($4)   = $39,600 + $  800 = $40,400
    precision = 120/320   = 0.375     recall = 120/300 = 0.400

t = 0.018 (the derived threshold):
    TP = 285, FN =  15, FP = 6,000
    cost =  15($220) + 6,000($4) = $ 3,300 + $24,000 = $27,300
    precision = 285/6,285 = 0.045     recall = 285/300 = 0.950

saving = $40,400 - $27,300 = $13,100 per 100,000 transactions

The better decision rule has a precision of 0.045.

Precision fell by a factor of 8 (0.375 / 0.045 = 8.3), F1 collapsed with it, and the rule still saves $13,100. Neither of those metrics is what anyone is paying for.

Say it out loud in the interview: the metric that got worse is not on the invoice.

Three refinements you should volunteer

Refinement 1: the full utility matrix. Benefits belong in the calculation too, not just costs. If a caught fraud recovers $220 and a correctly-approved transaction earns $1.20 in fees, the rule generalizes to “predict positive when the expected utility of doing so is higher,” and the threshold becomes:

threshold* = (C_fp + B_tn) / (C_fp + B_tn + C_fn + B_tp)

  B_tn = the benefit of correctly approving a good transaction -- the $1.20 fee,
         which is precisely what you forgo when you wrongly flag it
  B_tp = the benefit gained by catching a bad one

Same derivation, four cells instead of two.

Refinement 2: the threshold is in probability units, so it only means anything if p is calibrated. Two common ways that fails:

A boosted tree may never reach the threshold. A boosted tree is an ensemble that sums many small decision trees fitted one after another, each new tree trained on what its predecessors still got wrong. (It is not an average of independently grown trees — that is a random forest.) It builds its score up from the base rate in small shrunken steps, so a finite number of rounds rarely carries it out to an extreme probability. A model that never emits a score above 0.96 will simply never cross a threshold of 0.98.

A model trained on rebalanced data is off by a constant. Rebalancing shifts every prediction by a fixed amount in log-odds — log(p / (1-p)), the scale on which a probability becomes an unbounded number. The threshold you derived is then wrong by that same constant.

Both are treated in Resampling and the calibration it breaks and Calibration what it means and when it matters. If you cannot calibrate, sweep the raw score on validation data and pick the operating point that minimizes measured cost — the same decision, made empirically.

Refinement 3: capacity often overrides cost. If analysts can review 500 cases a day out of 100,000, then:

• the threshold is the 99.5th percentile of the score distribution, whatever probability that turns out to be
• the metric is precision@500
• calibration is irrelevant, because only the order matters

Ask which constraint is binding before deriving anything.

Here are the assumptions the threshold rule rests on.

1. The two costs are known, constant across rows, and additive. A false positive on a $12 transaction costs the same three analyst-minutes as one on a $12,000 transaction — often worth challenging.
2. p is a calibrated probability, since the threshold is compared against it directly.
3. The capacity to act on every row above the threshold exists. Where it does not, the capacity constraint replaces the cost calculation entirely.

The threshold you chose by accident

You never escape choosing a cost ratio. If you tune the threshold to maximize F1, you chose one anyway — you just did not write it down.

There is a clean result behind this: the F1-optimal threshold equals half the maximum F1 achieved.

Why the F1-optimal threshold is F1/2

Here is the one-line reason, since this would otherwise be the only claim in the chapter you had to take on trust.

Write F1 in its four-cell form and name the denominator:

F1 = 2·TP / (2·TP + FP + FN)         call the denominator D

Now flag one more row, whose true probability of being positive is q. In expectation:

TP rises by q                   FP rises by (1 - q)                FN falls by q

numerator 2·TP  rises by  2q
denominator D   rises by  2q + (1 - q) - q  =  1

Flagging that row is worth doing exactly when the new F1 beats the old one:

(2·TP + 2q) / (D + 1)  >  2·TP / D
        D(2·TP + 2q)   >  2·TP(D + 1)
   2·TP·D + 2qD        >  2·TP·D + 2·TP
              2qD      >  2·TP
                q      >  TP / D  =  F1 / 2

So at the optimum you are including precisely the rows that score above half the F1 you achieved. That is what “the threshold is half the F1” means. (Full statement and proof: Lipton, Elkan & Naryanaswamy, Thresholding Classifiers to Maximize F1 Score, arXiv:1402.1892.)

The cost ratio you just declared

Chain that result to the threshold formula from earlier in this section and the hidden assumption falls out.

Suppose tuning gets you F1 = 0.40. Then the threshold that achieved it is t = 0.40 / 2 = 0.20. And threshold* = C_fp / (C_fp + C_fn) rearranges to:

C_fn / C_fp = (1 - t) / t = (1 - 0.20) / 0.20 = 0.80 / 0.20 = 4

You just declared that a missed case costs four times a false alarm. In the fraud problem above, the true ratio is 220 / 4 = 55.

“We optimized F1” is a cost assumption nobody wrote down and nobody checked — and here it was wrong by more than a factor of 13.

The two functions

optimal_threshold is the closed-form rule. It is only valid on a calibrated probability.

sweep_cost is the empirical fallback: it makes the same decision without trusting the probability scale, by trying every threshold on a grid and keeping the one with the lowest measured cost. Two caveats go with it. It minimizes cost on the data you hand it, so it will overfit a small validation set and should be checked on a second split. And it fixes the boundary convention that p >= t counts as a positive prediction.

def optimal_threshold(c_fp, c_fn):
    """Cost-optimal cutoff on a calibrated probability."""
    return c_fp / (c_fp + c_fn)


def sweep_cost(probs, labels, c_fp, c_fn, grid=None):
    """Empirical operating point: minimize measured cost over a threshold grid.

    Use this when probabilities are not calibrated -- it makes the same
    decision without trusting the probability scale.
    """
    grid = grid or [i / 1000 for i in range(1, 1000)]
    best = None
    for t in grid:
        fp = sum(1 for p, y in zip(probs, labels) if p >= t and y == 0)
        fn = sum(1 for p, y in zip(probs, labels) if p < t and y == 1)
        cost = fp * c_fp + fn * c_fn
        if best is None or cost < best[1]:
            best = (t, cost, fp, fn)
    return {"threshold": best[0], "cost": best[1], "fp": best[2], "fn": best[3]}

12. Cheat sheet

This table is the chapter compressed into a lookup. Use it in three steps: find the shape of your problem in the left column, take the metric next to it, then read the last column.

That last column is the one that matters in an interview. The failure a metric hides is what you will be asked about, and it is what will eventually surprise you in production.

The one-line version of the whole chapter: every metric is a loss whose minimizer is a specific functional of p(y|x) — mean, median, quantile, or the probability itself; accuracy dies under imbalance because it is prevalence-weighted, ROC-AUC survives it because both its axes are within-class, PR-AUC is sensitive to it because precision contains the prevalence, AUC and log loss disagree because one reads ranks and the other reads numbers, and the threshold that turns any of it into a decision is C_fp / (C_fp + C_fn).

Next: 07 — Imbalanced Data, Calibration & Drift — what to do when the positives are rare, the probabilities are wrong, and the distribution moves underneath you.