09 — Model Debugging Playbook

A model is underperforming and time is limited. What do you do first?

This chapter gives a seven-step procedure. Step 0 reproduces the number you are unhappy about. Step 6 trains a better model. Do not reach step 6 until the six cheaper explanations in between have been ruled out.

The aim is to turn a complaint as vague as “the model is bad” into three concrete things: a named broken assumption, a number that proves the assumption is broken, and the one experiment that would confirm or refute it.

The organizing idea is that a bug is an assumption you did not know you were making. Every step below opens by naming the assumption it puts on trial.

Why the usual answer fails

The common interview answer is an unordered list: “check the data, tune hyperparameters, try a bigger model, do error analysis.” (A hyperparameter is a setting you choose before training rather than learn from data — how large a step the training algorithm takes each time it adjusts the model, for instance.)

A list has no order, and the order is what matters. Each step below either eliminates a whole class of explanation or produces the number the next step needs. Run them out of order and you can spend days tuning a step size on a pipeline whose labels are shuffled.

What goes in, and what comes out

Two separate input/output pairs are in play, and confusing them is a common source of wasted time. One pair belongs to the model. The other belongs to the playbook.

Both pairs are defined below, along with the measurement words used elsewhere in the chapter.

What goes into the model, and what comes out

A supervised model is a function fitted to past examples.

• Feature vector — the input. One row of measurable properties describing one thing: a user, a transaction, an image. A churn model’s row might be purchases_7d = 3 (purchases in the last seven days) and tenure_days = 88.
• Prediction — the output. A class (“churned” or “retained”), a number, or most often a score between 0 and 1 that is meant to behave like a probability.
• Label — the correct answer for that row, recorded by a human annotator or emitted by a business process.

Rows get divided into three disjoint sets, and telling them apart matters in every step below.

Four more words describe how training itself runs.

• Batch — the group of rows processed together in one update.
• Loss — a single number measuring how wrong the model’s outputs are on a batch.
• Gradients — the per-parameter directions that would reduce that loss fastest.
• Generalization — how well the model does on rows it never saw.

Two words say where a number was measured. Offline means measured on stored rows. Online means measured on live traffic in production. A large part of this chapter is about the two disagreeing.

What goes into the playbook, and what comes out

Its input is a symptom plus four artifacts.

The symptom is one of these: an offline number that looks wrong, an online number that disagrees with the offline one, or a metric that improved while the business did not.

The four artifacts are the training code, the data and its splits, the serving code path, and the logs of what production actually scored. Most steps below are unrunnable without all four, so if one is missing, say so before you start.

Its output is a single sentence in this shape:

Assumption X is false. Here is the number that proves it. Here is the fix. Here is the check that would have failed if I were wrong.

A debugging session that ends in “we tried some things and it got a bit better” has produced no such sentence and will not survive its next repetition.

The measurement words

Six words for scoring a model. AUC gets its own explanation below.

AUC is short for area under the receiver operating characteristic (ROC) curve. The name is misleading; read it as a probability rather than as an area:

AUC is the chance that the model gives a randomly chosen positive row a higher score than a randomly chosen negative row.

So 0.5 is a coin flip and 1.0 is perfect ranking. Roc auc is a probability and here is the derivation derives that equivalence.

Where this chapter sits

This is the applied capstone of the machine-learning track.

Diagnosing a training run diagnoses a training run from its loss curve — the picture of the model’s error falling as optimization proceeds. This chapter diagnoses a model from its metrics, and the surface it searches is much larger.

The procedure

The whole playbook fits in one flowchart. Read it before any individual step. Read it top to bottom: diamonds are questions you answer with a measurement, boxes are conclusions.

flowchart TD
    S0["STEP 0<br/>Reproduce · seed everything · baseline"] --> G0{"Beats the<br/>baseline?"}
    G0 -->|no| STOP["The model adds nothing.<br/>Nothing below matters yet."]
    G0 -->|yes| S1{"STEP 1a<br/>Can it overfit<br/>8 samples to ~0 loss?"}

    S1 -->|no| WIRE["Wiring bug: label alignment,<br/>loss axis, frozen params"]
    S1 -->|yes| S1A{"STEP 1b<br/>Blind relabel of 100 rows<br/>agrees with the stored label?"}
    S1A -->|no| NOISE["LABEL NOISE<br/>caps every metric below"]
    S1A -->|yes| S1B{"Offline number<br/>implausibly high?"}
    S1B -->|yes| LEAK["LEAKAGE<br/>single-feature AUC scan<br/>then ablate"]
    S1B -->|no| S1C{"Offline good,<br/>online bad?"}
    S1C -->|yes| SKEW["TRAIN/SERVE SKEW<br/>replay serving logs<br/>through the offline scorer"]
    S1C -->|no| S2{"STEP 2<br/>Learning curve shape?"}

    S2 -->|"converged, high error"| BIAS["BIAS-limited<br/>too simple for the pattern"]
    S2 -->|"gap, val still falling in n"| VAR["VARIANCE-limited<br/>more data or more regularization"]
    S2 -->|"converged, low error"| S3["STEP 3<br/>Error analysis by slice"]
    VAR --> S3
    BIAS --> S3

    S3 --> G3{"One slice<br/>catastrophic?"}
    G3 -->|yes| FIXS["Fix the slice:<br/>feature availability, sampling,<br/>slice-specific data"]
    G3 -->|no| S4{"STEP 4<br/>Does the metric<br/>match the decision?"}
    S4 -->|no| METRIC["Fix the metric<br/>or the operating point"]
    S4 -->|yes| S5{"STEP 5<br/>Are the labels right?"}
    NOISE --> S5
    S5 -->|no| LBL["Estimate the noise ceiling.<br/>Relabel TEST first."]
    S5 -->|yes| S6["STEP 6<br/>Capacity first, then features,<br/>then data — in that order"]

    style STOP fill:#9d0208,color:#fff
    style WIRE fill:#9d0208,color:#fff
    style LEAK fill:#9d0208,color:#fff
    style SKEW fill:#9d0208,color:#fff
    style NOISE fill:#bc6c25,color:#fff
    style S3 fill:#2d6a4f,color:#fff
    style S6 fill:#2d6a4f,color:#fff

The colors encode one thing: red is a terminal state reached because something is broken, orange is a real limit that is nobody’s bug, and green is a step you reach only once the red ones are excluded.

The same tree as a sequence of questions:

1. Step 0 — reproduce, seed, build a baseline. Then the first gate: beats the baseline? If no, the model adds nothing and nothing below matters yet.
2. Step 1a — can it overfit 8 samples to near-zero loss? A “no” here is a wiring bug: a mistake in how data, loss, and parameters are connected, not a modelling problem.
3. Step 1b — the label-noise audit. Relabel 100 rows blind and compare them to what is stored. Labels that are 12% wrong cap every number the rest of the tree produces, so this branch does not terminate — it jumps straight to step 5.
4. The two disagreement checks. An offline number that is implausibly high points at leakage: information about the answer has crept into the inputs. Offline good but online bad points at train/serve skew: the features computed in production differ from the ones computed during training.
5. The remaining gates, in order. Learning curve shape, then one slice catastrophic, then does the metric match the decision, then are the labels right.
6. Step 6. Only a clean answer at every gate above gets you here, and this is the first point at which you are allowed to change the model.

Note the shape of the tree: every branch that terminates early terminates on a bug, and the only path that reaches “train a better model” is the one where six other explanations have been ruled out. In applied work, most underperformance is a data, label, metric, or skew problem, and model improvements are the residual.

Step 0 — Reproduce, seed, baseline

A measurement means something only after three things are done: every source of randomness is fixed, the metric’s variation when nothing changes is known, and the number is anchored against a trivial model.

Assumption on trial: the number I just measured is the number I would measure again tomorrow, and it means something on its own. Both halves are usually false. Randomness moves the number, and without a floor to compare against, the number has no scale.

Seed everything, including the split

A seed is the starting value of a pseudorandom number generator. Fixing it makes every “random” choice repeat identically on the next run.

The helper below fixes every source of randomness that is known to move a metric. Read the comments as a checklist: the last four, in the trailing comment block, are the ones the function cannot fix for you because they live in your own data code.

import os, random
import numpy as np
import torch

def seed_everything(seed: int = 0) -> None:
    """Every source of randomness that can move a metric."""
    os.environ["PYTHONHASHSEED"] = str(seed)      # dict/set iteration order
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)                       # covers CUDA too in recent versions
    torch.backends.cudnn.deterministic = True     # pick deterministic kernels
    torch.backends.cudnn.benchmark = False        # autotuner picks by timing -> nondeterministic
    # And the ones people forget:
    #   - the train/val/test split seed
    #   - the DataLoader worker_init_fn (each worker reseeds independently)
    #   - augmentation RNG
    #   - any hashing-based feature (hash bucket assignment)

Several of those lines need a translation.

• CUDA is the interface through which code runs on an NVIDIA graphics processor. cuDNN is its neural-network library.
• A kernel here is one routine in that library. cuDNN ships several interchangeable implementations of the same operation, and by default it picks between them by timing them at startup. That choice depends on what else the machine was doing, so it changes between runs — which is what benchmark = False switches off.
• RNG is the random number generator.
• Augmentation is the practice of randomly perturbing training inputs — cropping, flipping, adding noise — to enlarge the effective dataset. It is a second generator, and it needs its own seed.
• A DataLoader worker is a background process that prepares batches. Each one reseeds itself independently unless told otherwise.
• A hashing-based feature maps a category name to one of a fixed number of buckets by hashing it. Change the hash seed and every category silently lands in a different bucket.

Why the split seed is the one that matters most

The standard error of a measurement is the typical amount it moves purely from which rows happened to be sampled — no code change, no retraining, just a different draw.

For a proportion such as accuracy, the standard error is sqrt( p * (1 - p) / n ), where p is the accuracy and n is the number of rows scored.

Substitute a test set of n = 2,000 at a true accuracy of p = 0.90:

sqrt( 0.90 * 0.10 / 2000 )
  = sqrt( 0.09 / 2000 )
  = sqrt( 0.000045 )
  = 0.0067                =  0.67 percentage points

So two runs that are identical in every respect except which rows landed in the test set will typically land about two thirds of a point apart.

An unseeded split means every experiment scores a different test set, so a “0.4 point improvement” is smaller than the noise introduced by re-splitting. You are not measuring your change; you are measuring which rows landed where.

Establish the noise floor before establishing anything else

Before comparing two configurations you need to know how far apart two identical configurations land, and that requires running the same one several times.

Run the same config five times, changing only the seed. Nothing else differs, so every bit of spread you see is noise:

seed   val AUC
  0     0.912
  1     0.907
  2     0.918
  3     0.903
  4     0.914
        -----
mean    0.9108     sd 0.0059

In that listing and everywhere below, val is short for validation, and sd is the standard deviation of those five numbers — the usual measure of spread.

sd is not the number you gate on

sd = 0.0059 is the spread of one measurement. But you never make a decision about one measurement. You make it about a difference: config A’s score minus config B’s score.

Subtracting two independent draws adds their variances. Variance is the square of the spread, so:

var of a difference = var(A) + var(B) = 2 * var(one run)
sd  of a difference = sqrt(2) * sd(one run)
                    = 0.0059 * sqrt(2)  =  0.0083

Consider what happens when you apply the familiar “two standard deviations” rule using the wrong spread. The naive band is 2 * 0.0059 = 0.0118. Measured against the spread that actually governs a difference, that band sits at

0.0118 / 0.0083  =  1.41  =  sqrt(2) standard deviations out

and a threshold only sqrt(2) standard deviations out lets through

2 * (1 - Phi(sqrt(2)))  =  2 * (1 - 0.921)  =  0.157   =  15.7%

of pure noise. Phi is the standard normal cumulative distribution function: Phi(x) is the probability that a standard bell curve lands below x, so 2 * (1 - Phi(z)) is the chance of exceeding z standard deviations in either direction. You believed you had built a 5% filter. You built a 15.7% one.

That sqrt(2) is not a coincidence of these particular numbers. A band of 2*sd measured against a spread of sd*sqrt(2) is 2/sqrt(2) = sqrt(2) standard deviations out no matter what sd is — the sd cancels. So this mistake always lets through 15.7%, on every dataset, whatever sd turns out to be.

It is the same one-variance-where-the-situation-needs-two mistake that two-sample testing exists to avoid (Ab testing), and it costs you roughly 3x here: a rule you believed was a 1-in-20 filter is a 1-in-6 filter.

The two bands you are allowed to use

Which band applies depends on how many runs you paid for. Say which comparison you actually made.

The 1.96 is the number of standard deviations that actually contains 95% of a normal distribution. The sqrt(2/5) in the second row is the same sqrt(2) as before, divided by sqrt(5), because averaging 5 runs shrinks the spread of each side by sqrt(5).

The single-run band is 0.016, not 0.012, and the five-seed band is 0.007 — and the only way to get the small number is to pay for it with ten training runs instead of two.

Fix whichever band you earned and hold every subsequent experiment to it. Many apparent weekly “improvements” fall inside it. The 0.0059 itself is only an input: it never appears in a decision rule on its own.

The baseline, and why skipping it poisons everything after

A baseline gives your metric a scale, and its absence has a cost you can put numbers on.

You need three baselines, and it is worth building them cheapest first.

Three entries in that table need unpacking.

• Prevalence is how common the positive class is. A majority-class baseline always predicts whichever class is more common, so beating it proves your metric is measuring something other than that imbalance.
• A linear model predicts by adding up the features with one weight each. Logistic regression is the version that squashes that sum into a probability. A regularized one is trained with an extra penalty on large weights, which discourages it from memorizing noise.
• A GBDT — gradient-boosted decision tree — is an ensemble of small trees added one at a time, each correcting the previous ones’ errors. It is still the strongest cheap model on tabular data (Why gbdts still beat neural nets on tabular data).

Here is the failure all three prevent, on a real task. Compare the first two lines, then the last two — they say different things.

task: churn prediction, 30 days
class balance:  94.2% retained, 5.8% churned

deep model accuracy .............. 0.940
"predict retained" accuracy ...... 0.942     <- the constant function wins
logistic regression, 6 features ... AUC 0.911
deep model ....................... AUC 0.926

Churn is a customer leaving; this task predicts who will leave within 30 days. sklearn is scikit-learn, the standard Python machine-learning library.

Two facts fall out of those four lines.

1. Accuracy was the wrong metric. The deep model scores 0.940 and a constant function that always says “retained” scores 0.942, because 94.2% of customers are retained. That is step 4’s territory.
2. The model’s real contribution is 0.926 - 0.911 = 0.015 AUC over six features and a sklearn one-liner. That may or may not be worth its serving cost — but it is now a decision instead of an assumption.

Without a baseline, every number you produce is unanchored: you cannot tell “good” from “the problem was easy” from “the metric is degenerate.”

The baseline also serves as a control. A baseline trained on the same pipeline moves when the pipeline moves, so if it shifts after a change nobody thought was risky, the pipeline broke.

Step 1 — Data bug or model bug?

Separating a broken pipeline from a weak model takes four checks, all cheap enough to run in an afternoon, arranged so that each one is cheaper than the next and each rules out more.

Assumption on trial: the data the model sees is the data I think it sees. Every check below is a different way for that to be false — the inputs are wired to the wrong labels, the labels are wrong, the production inputs differ from the training inputs, or the inputs secretly contain the answer.

1a. Overfit a single batch

This check asks whether the training machinery works at all, independent of whether the model is any good.

First the word, because this check inverts its usual sense and step 2 depends on the usual one.

Overfitting is fitting the training rows so closely that the model has absorbed their noise as well as their pattern, so training error keeps falling while validation error stops falling or rises.

Everywhere else in this chapter that is a failure — step 2’s variance-limited branch is exactly this.

Here it is the goal. With only 8 rows there is no generalization to lose, so a model that fails to overfit them is not being well behaved. It is telling you the training loop is broken.

The procedure is four lines long.

1. Take 8 samples.
2. Turn off shuffling, augmentation, dropout, and weight decay. The last two are regularizers: deliberate handicaps that trade training accuracy for generalization. Dropout randomly switches off part of the network on each step; weight decay penalizes large parameter values. Turn them off, because on this test you want memorization.
3. Train on those same 8 rows over and over.
4. Watch whether the loss goes to approximately zero.

The full treatment is in Step zero overfit a single batch. The reason it belongs here is the elimination argument, which is worth stating slowly.

A model that cannot memorize 8 examples has no capacity, data-volume, or generalization explanation available. Capacity means how much structure the model is able to represent at all, and eight rows demand almost none of it. Data volume cannot be the problem when the task is eight rows. Generalization cannot be the problem when nothing is being generalized to.

So a model that can overfit 8 samples to near-zero loss has a working training loop, and one that cannot has a broken one. What remains is a short list:

• labels misaligned with inputs;
• loss applied to the wrong axis of the output array;
• frozen parameters that never receive updates;
• a missing zero_grad call, so gradients from old batches accumulate;
• a learning rate of lr = 0, which means the optimizer takes steps of size zero.

Ten minutes of work converts an open-ended investigation into a five-item checklist.

1b. Label-noise audit

This check estimates what fraction of your stored answers are simply wrong, which caps every metric you will ever compute.

Sample 100 training rows at random and label them yourself, blind to the stored label. Then compute your agreement rate, the fraction on which you and the stored label match. There are two outcomes worth distinguishing.

• Agreement 0.97+: labels are fine, move on.
• Agreement 0.88: your labels are 12% wrong, which caps everything downstream. Go to step 5 before touching the model.

Do this on 100 rows, not 10, because the estimate itself has a standard error. Use the same proportion formula from step 0, with p = 0.88 and n = 100:

sqrt( 0.88 * 0.12 / 100 ) = sqrt( 0.001056 ) = 0.032

So your agreement estimate is good to about plus or minus 3 points. That is enough to tell “clean” (0.97) from “12% noise” (0.88), and not enough to tell 8% noise from 12%. Do not argue about the second decimal place.

1c. Train/serve skew

Train/serve skew is a difference between the feature values computed during training and those computed at serving time for the same entity, and this check finds it by computing both and subtracting.

It is the highest-yield check when offline and online numbers disagree. Compute the same feature, for the same entity — the thing a row is about, such as one user or one account — at the same timestamp, through both code paths, and diff.

In the table below, p50 is the 50th percentile, that is, the median. The column to read is the last one: the share of rows where the two paths produced different values at all. Everything else is context for it.

feature              offline mean   online mean   offline p50   online p50   % rows differing
purchases_7d              0.42          0.19          0.0          0.0            31%
session_len_s            184.2         181.9        142.0        141.0             4%
country_code_hash        (uniform)     (uniform)      --           --              0%
days_since_signup         88.4          88.6         61.0         61.0             1%

One feature is wildly off — purchases_7d differs on 31% of rows, and its offline mean is more than double its online mean — while the other three differ on 4%, 1%, and 0%. One bad feature against a clean background is the signature of skew. If everything were off by a little, you would be looking at a different problem, such as a sampling difference between the two log sources.

Three mechanisms produce it, and Feature stores and trainingserving skew has the full taxonomy.

1. Different code. The offline aggregation is written in SQL, the query language of the data warehouse, the online one is written in Python, and they disagree on < versus <= at the window boundary — that is, on whether an event exactly at the cutoff counts.
2. Different data availability. Offline you can see the whole day; online, at 09:00, you cannot.
3. Different time semantics. Offline joins use the row’s label date; online uses “now.”

Mechanism 3 is the dangerous one, because it is simultaneously a skew bug and a leakage bug, which is the subject of 1d.

1d. Leakage

Leakage is information about the label reaching the model through its inputs. Recognizing it and proving it are two separate jobs.

The signal is that the number is too good. Not “surprisingly good” — implausibly good, relative to what a domain expert achieves or what the problem could possibly contain.

The comparison you need is your number against the two reference points below it:

offline val AUC .................. 0.987
domain expert, same information .. ~0.75
prior production model ........... 0.81

An 0.987 on a problem humans solve at 0.75 is not a breakthrough. Two diagnostics settle it, and the order matters: the scan finds a suspect, the ablation convicts it.

Diagnostic 1: the single-feature AUC scan

Fit each feature alone — one feature, nothing else — and rank the results. Any single raw feature above about 0.95 is a leak until proven otherwise, because no honest individual measurement separates the classes that cleanly.

from sklearn.metrics import roc_auc_score


def single_feature_auc(X, y, feature_names):
    """Rank features by their standalone AUC. A single feature near 1.0
    is almost always leakage, not signal."""
    scores = []
    for j, name in enumerate(feature_names):
        col = X[:, j]
        auc = roc_auc_score(y, col)
        scores.append((name, max(auc, 1.0 - auc)))   # direction-agnostic
    return sorted(scores, key=lambda kv: -kv[1])

The max(auc, 1.0 - auc) line is the only subtle thing in that function, and it makes the scan indifferent to direction.

A feature that predicts the answer perfectly backwards — high value always means negative — has a raw AUC of 0.0. Run it through max(auc, 1.0 - auc) and you get max(0.0, 1.0) = 1.0, the same number a perfectly forwards feature gets. That is the point: a raw AUC of 0.0 and a raw AUC of 1.0 are equally suspicious, and collapsing them onto one scale lets a single threshold catch both.

Here is what the ranked output looks like. Notice the gap between the first row and the second — that gap, not the absolute value, is what makes the top feature stand out:

feature                 standalone AUC
account_status_code          0.981      <- set to 'closed' when churn is recorded
tenure_days                  0.688
support_tickets_30d          0.640
purchases_7d                 0.612

Diagnostic 2: the confirming ablation

An ablation is retraining with something deliberately removed to measure what it contributed. Drop the suspect feature, retrain, and check that the metric falls to a plausible level rather than merely falling. Always ablate an ordinary feature too, as a control:

all features ....................... AUC 0.987
minus account_status_code .......... AUC 0.792     <- plausible; the leak is confirmed
minus tenure_days (control) ........ AUC 0.981     <- ordinary feature, small effect

Removing the leak drops AUC from 0.987 to 0.792, which sits right next to the prior production model’s 0.81 and the expert’s 0.75. Removing an ordinary feature drops it by 0.006. The contrast between those two lines is the evidence.

The confirmation is not “the metric dropped” — it is “the metric dropped to the range the problem actually supports.” A leak removal that leaves you at 0.96 means there is a second leak.

The four leakage patterns

Each has a different mechanism and therefore a different check.

Several terms in that table need glosses.

• ETL stands for extract-transform-load: the batch job that moves and reshapes data between systems. Reading it is how you find out which process wrote a column.
• t - 1 means “as of one time unit before the decision”. Recomputing every feature at t - 1 and diffing against the stored value shows which features were quietly reading the future.
• A CV fold is one train/validation partition inside cross-validation, the practice of rotating which slice is held out.
• The last row names four transforms. A scaler rescales a feature to a standard range. An imputer fills in missing values. A target encoder replaces a category with the average label seen for that category. A feature selector keeps a subset of columns.

Fit any of those four on all the data before splitting and the held-out rows have influenced the transform, which is a quiet leak. Target encoding and the exact leakage mechanism shows the worst case, target encoding, where the held-out row’s own label ends up baked into its own feature.

Group leakage, in numbers

Group leakage deserves its own arithmetic, because the intuitive estimate badly understates it.

Set up: 40,000 rows over 8,000 users, so 5 rows per user, split 80/20 at random by row.

First question: how many users land on both sides? A user is entirely in train if all 5 of their rows go there, with probability 0.8^5 = 0.328. Entirely in test with probability 0.2^5 = 0.0003. Everyone else straddles the split:

1 - 0.8^5 - 0.2^5  =  1 - 0.328 - 0.0003  =  0.672    ->  67% of users

Second question — and this is the one that matters — how many test rows have their user in train? Pick any test row. Its user has 4 other rows, each landing in train with probability 0.8. The user is absent from train only if all four other rows also went to test:

1 - 0.2^4  =  1 - 0.0016  =  0.998    ->  99.8% of test rows

A per-user memorizable feature then makes essentially the entire test set memorized, and no amount of regularization tuning will reveal it.

Step 2 — Bias or variance?

One of the most expensive planning questions in applied machine learning — would more data help? — can be measured rather than guessed.

Assumption on trial: more data would help. It is roughly a coin flip whether that is true, the two answers point at completely different quarters of work, and the curve below tells you which without spending the quarter.

Two different plots get called “learning curves” and they answer different questions. Get the wrong one and you will answer the wrong question confidently.

An epoch is one full pass through the training set. Everything below is about the second plot: you retrain the same model on 5,000 rows, then 10,000, then 20,000, and so on, and plot validation error against the size you trained on.

The chart below turns the resulting shape into a diagnosis.

flowchart TD
    LC["Plot val error vs training-set size<br/>at 5-6 sizes, several seeds each"] --> Q1{"Train and validation<br/>curves converged?"}
    Q1 -->|"yes, at HIGH error"| B["BIAS-limited<br/>more data will not help<br/>-> capacity or features"]
    Q1 -->|"yes, at LOW error"| D["Done for this model class<br/>-> slice analysis, metric, labels"]
    Q1 -->|"no, large gap"| Q2{"Is validation error still<br/>falling with n?"}
    Q2 -->|yes| V["VARIANCE-limited<br/>extrapolate the curve<br/>to price the data"]
    Q2 -->|"no, flat"| N["Noise ceiling or<br/>distribution mismatch<br/>-> STEP 5"]
    Q1 -->|"validation BELOW train"| A["Accounting or leakage<br/>re-measure train in eval mode"]

    style B fill:#bc6c25,color:#fff
    style V fill:#bc6c25,color:#fff
    style N fill:#bc6c25,color:#fff
    style D fill:#2d6a4f,color:#fff
    style A fill:#9d0208,color:#fff

The colors carry the same meanings as the main flowchart: red is a bug (validation below train is an accounting error or a leak), orange is a real limitation that is nobody’s fault (bias, variance, a noise ceiling), and green is the clean outcome.

The chart branches on one question — have the train and validation curves converged? — and then on a follow-up. Five outcomes:

• Converged at HIGH error — bias limited. The model is too simple to represent the pattern. More data will not help. Go to capacity or features.
• Converged at LOW error — you are done with this model class. Go to slice analysis, the metric, or the labels.
• Large gap, validation error still falling as n grows — variance limited. Extrapolate the curve and price the data, which the next subsection does.
• Large gap, validation error flat in n — neither. Go to step 5.
• Validation error BELOW training error — an accounting artifact or a leak. Separate them by re-measuring training error in evaluation mode, which turns off dropout.

Reading it properly, with numbers

The shape of the curve turns into a dollar figure, which is the entire reason to plot it.

Measured validation error at four training-set sizes:

n        val error
 5,000     0.290
10,000     0.240
20,000     0.205
40,000     0.180

The error falls, but by less each time you double the data: 5.0 points, then 3.5, then 2.5. That decelerating shape is what you are fitting.

Guess that the curve is heading for a floor of 0.12 and subtract it out. The excess over that floor behaves much more simply:

n        val error   excess over 0.12   ratio to previous excess
 5,000     0.290          0.170                --
10,000     0.240          0.120               0.71
20,000     0.205          0.085               0.71
40,000     0.180          0.060               0.71

Each doubling of n multiplies the excess by a constant 0.71, and 0.71 is 1/sqrt(2) — exactly what you get when a quantity falls as 1 / sqrt(n) and n doubles. That is the fingerprint of a power law: a relationship in which the quantity falls as a fixed power of the input rather than by a fixed amount.

So fit this form:

err(n) = e_inf + a / sqrt(n)

read as “error at sample size n equals e-infinity plus a divided by the square root of n”. e_inf is the asymptote: the error the curve approaches but never crosses, no matter how much data you add. a sets how fast you approach it.

Fitting gives e_inf = 0.12 and a = 12. Substitute each n back in and check:

err( 5,000) = 0.12 + 12 / 70.7  = 0.12 + 0.170 = 0.290
err(10,000) = 0.12 + 12 / 100.0 = 0.12 + 0.120 = 0.240
err(20,000) = 0.12 + 12 / 141.4 = 0.12 + 0.085 = 0.205
err(40,000) = 0.12 + 12 / 200.0 = 0.12 + 0.060 = 0.180

All four match the measured column. Now invert the formula to ask the question you care about. To hit a target error e you need a / sqrt(n) = e - e_inf, so

n = ( a / (e - e_inf) )^2

For a target of 0.150 that is (12 / 0.030)^2 = 400^2 = 160,000 rows — 4x what you have. Do that for a few targets and the curve becomes a price list:

“Get more data” stops being advice and becomes a budget line: the next 3 points of error cost 4x the labeling spend, and the 2 points after that cost 36x.

The e_inf = 0.12 asymptote is equally load-bearing. It is the part of the error that more of the same kind of data cannot touch — the last row of the table says you would need infinite rows to reach it. That is what sends you to features in step 6 rather than to labeling.

The five curve shapes

These five shapes exhaust the cases. The third column is the useful one: each shape forbids a remedy, and the remedy it forbids is usually the one someone is about to propose.

Three rows need a note.

The hypothesis class in the first row is the set of functions your model family can express at all. A straight line cannot bend, no matter how much data it sees.

The third row — converged, low error — is the good case. Its point is that further tuning of this model family buys nothing.

The fourth row is the one that gets misread. A big gap looks like variance and gets treated with data. But if validation error is flat while n quadruples, the extra data is not reaching the failure, and buying four times more of it will not either. Go to step 5.

Step 3 — Error analysis by slice

Taking a respectable overall number apart by subgroup is where most real defects are found.

Assumption on trial: the average describes every user. It never does, and the arithmetic guarantees it: an overall metric is a traffic-weighted average, so a subgroup must be small for the average to hide its failure, and small subgroups are exactly where new markets, new devices, and new languages live.

This is among the highest-value activities in applied machine learning, and the one most often cut for time. The mechanism is arithmetic: an average cannot surface a catastrophic minority, because the minority’s weight in the average is its share of traffic.

A worked breakdown

The table below shows an unremarkable headline number concealing a slice at chance level.

Overall accuracy 0.922 on 20,000 rows is perfectly respectable, comfortably above the baseline, and gives no reason to look further.

Read the bolded row against the total row. Then compare its “share of traffic” column against its “share of all errors” column — those two numbers are the finding.

Four percent of traffic produces twenty-five percent of all errors, at an accuracy of 0.510 — a coin flip.

Now see how little that costs the headline number. If the slice performed like the rest of traffic (about 0.93 instead of 0.51), the overall metric would move by

0.04 * (0.93 - 0.51) = 0.04 * 0.42 = 0.017

that is, the slice’s 4% share of traffic times its 0.42 accuracy gap. Seventeen thousandths — well inside the range people casually attribute to hyperparameters. The average is arithmetically incapable of making this slice visible.

Two consequences make this worth saying out loud in an interview.

• If mobile · es is the segment the company is expanding into, the model is unusable in the market it is being built for, and the headline metric says it is fine.
• Fixing that one slice from 0.51 to 0.93 cuts its errors from 392 to 56, so total errors go 1,560 to 1,224 and overall accuracy goes 0.922 to 0.939. That is the same 1.7 points that a quarter of model work would not buy.

Finding the cause

Once a slice is identified, the next move is to compare its inputs against everyone else’s rather than to assume the slice is intrinsically harder.

The slice is not “hard.” Slice the features, not just the metric — that is, compare the model’s inputs inside the slice against everywhere else:

                        mobile·es    everything else
page_text non-null         22%             97%
page_text mean length      41 chars      1,180 chars
model p(class) mean       0.08            0.47

Read those three rows in order.

The first two say the model is not getting the same information: page_text is present for 97% of other traffic and only 22% of this slice, and when it is present it is 41 characters instead of 1,180.

The third — “the model’s mean predicted probability for the class” — is 0.08 in the slice against 0.47 elsewhere. The model is confidently saying “no” to almost the entire slice.

The cause is upstream of the model entirely. The mobile SDK — software development kit, the library the mobile app uses to send events — truncates page_text at 512 characters and emits null below a length threshold. The imputer then fills those nulls with 0, and the model reads 0 as a confident negative rather than as “unknown.”

The failure is a feature-availability difference: invisible in aggregate, total within the slice. The fix is a missingness indicator — an extra binary feature that says “this value was absent” — plus a slice-specific fallback. Not a better architecture (Missing values three mechanisms three different correct answers).

Doing this systematically

Slice analysis becomes routine only when it is a function you call rather than a spreadsheet you build, and the sort order is the part that carries the insight.

The function below groups rows by whatever columns you name, computes each group’s accuracy and error count, and sorts by error share rather than by accuracy. The long docstring exists because of one subtlety: slices smaller than min_n are dropped, and if you divide by the total error count while dropping slices, the shares silently fail to sum to 1. The assertion at the bottom is what enforces that they do.

import numpy as np
import pandas as pd


def slice_report(df, y_true, y_pred, by, min_n=50):
    """Per-slice accuracy AND error mass. Sorting by error share is the
    point: a bad slice matters in proportion to n * (1 - acc).

    Two passes, and the reason is the denominator. `err_share` is the sort
    key this entire section is built on, so it has to be a share OF SOMETHING
    STATED. Dividing by the error count over all rows while skipping the
    sub-min_n slices makes the column sum to less than 1 by however much
    error the skipped slices held -- silently, and worst exactly when there
    are many small slices, which is when you most need the ranking.
    """
    y_true, y_pred = np.asarray(y_true), np.asarray(y_pred)
    groups = [(key, g) for key, g in df.groupby(by) if len(g) >= min_n]
    reported_err = sum(int((y_true[g.index] != y_pred[g.index]).sum())
                       for _, g in groups)
    dropped_err = int((y_true != y_pred).sum()) - reported_err

    rows = []
    for key, g in groups:
        err = int((y_true[g.index] != y_pred[g.index]).sum())
        rows.append({
            "slice": key,
            "n": len(g),
            "acc": 1.0 - err / len(g),
            "errors": err,
            # share of the errors this report COVERS, so the column sums to 1
            "err_share": err / reported_err if reported_err else 0.0,
        })
    rows.sort(key=lambda r: -r["err_share"])
    # Report what was excluded rather than absorbing it into the shares.
    rows.append({"slice": f"<{min_n} rows (excluded)", "n": None, "acc": None,
                 "errors": dropped_err, "err_share": None})
    return rows


# The column has to sum to 1, or the sort key is measuring an unstated
# denominator. This assert is the whole fix.
_df = pd.DataFrame({"g": ["a"] * 100 + ["b"] * 100 + ["c"] * 10})
_yt = np.zeros(210, dtype=int)
_yp = np.array([1] * 20 + [0] * 80 + [1] * 40 + [0] * 60 + [1] * 10)
_rep = slice_report(_df, _yt, _yp, "g", min_n=50)
_shares = [r["err_share"] for r in _rep if r["err_share"] is not None]
assert abs(sum(_shares) - 1.0) < 1e-12, sum(_shares)
assert _rep[-1]["errors"] == 10          # slice "c" is excluded, and it says so

Why sort by error share and not by accuracy? A 30-row slice with two errors has an accuracy of 0.933 and contributes 2 errors out of 1,560. Sorting by accuracy would float it near the top of your worry list; sorting by err_share — the share of all covered errors that this slice contributes — ranks slices by how much fixing them would actually move the headline number. A slice matters in proportion to n * (1 - acc), which is just its error count.

Three practices make this routine rather than heroic.

1. Slice on everything cheap: device, locale, tenure bucket, traffic source, time of day, input-length decile — one tenth of the range, so the shortest 10% of inputs, then the next 10% — and the label itself, which gives you per-class recall. Then slice on pairs of the top offenders.
2. Automate the search. Fit a shallow decision tree whose target is the error indicator, a column that is 1 when the model got the row wrong and 0 when it did not. Its high-error leaves with non-trivial support are your slices, discovered rather than guessed.
3. Report worst-slice metrics as a standing number, not just the mean. A model whose mean rose and whose worst slice fell is usually a regression.

Step 4 — Is the metric wrong?

Sometimes the model behaves correctly and the number describing it is wrong, in which case fixing the model would not have helped.

Assumption on trial: the metric goes up exactly when the product gets better. Five distinct mechanisms break that link — imbalance, the wrong unit of aggregation, an inherited threshold, missing calibration, and proxy drift — and each one has a different repair.

Accuracy under imbalance

A metric can be dominated by class prevalence rather than by skill, and the common alternative has the same blind spot.

You already saw one form of it in step 0, where 0.940 accuracy loses to a constant function at 0.942. The subtler version is ROC-AUC.

Set up the numbers. You have 100,000 negatives and 1,000 positives — a 100:1 imbalance. Your operating point, meaning the specific threshold at which you convert scores into decisions, gives recall 0.80 at a false-positive rate (FPR) of 0.05, so you catch 80% of the positives and flag 5% of the negatives.

Count what lands on an analyst’s desk:

TP = 0.80 * 1,000   =    800     <- positives caught
FP = 0.05 * 100,000 =  5,000     <- negatives flagged by mistake
precision = TP / (TP + FP) = 800 / 5,800 = 0.138

TP is true positives and FP is false positives. Precision of 0.138 means 6.25 false alarms for every real one — 5,000 divided by 800 — and an FPR of 0.05 looks perfectly respectable while producing it.

The mechanism is normalization. ROC-AUC’s x-axis is the false-positive rate, which divides by the negative count, so a 100:1 imbalance cancels out of it entirely. Precision does not divide by the negative count, which is why PR-AUC — the area under the precision-recall curve — and precision-at-fixed-recall are the metrics that show you this and ROC-AUC is not.

That normalization is not a defect, and the trade runs both ways:

• ROC-AUC is stable across prevalence, so it is the only one of the two you can compare across datasets or time periods.
• PR-AUC is meaningless unless you state the prevalence it was measured at — but it is the one that shows the workload at your operating point.

Report both, for those two reasons (Pr auc vs roc auc under heavy imbalance and the comparison pr auc is not allowed to make).

The aggregation unit is wrong

A metric can be correct per row and meaningless per user, because the product’s unit of success is not the row.

Per-row accuracy 0.970 sounds strong. But suppose the product only counts a session as successful when all 20 of its rows are right. Assuming the errors are independent, the session succeeds only if all 20 rows succeed:

P(session correct) = 0.97 * 0.97 * ... (20 times)
                   = 0.97^20
                   = 0.544

Only fifty-four percent of sessions come out fully correct while the model is 97% accurate, and both numbers are true.

Always report the metric at the unit the user experiences: per session, per document, per invoice — not per token or per row.

The threshold is inherited, not chosen

The number that converts a score into an action is almost never chosen deliberately.

0.5 is a default, not a decision.

Say a false negative — a miss — costs 20x what a false positive — a false alarm — costs. Write C_FN for the cost of a miss and C_FP for the cost of a false alarm, so C_FN = 20 * C_FP.

For a row the model scores at probability p, you should predict positive when the expected cost of staying silent exceeds the expected cost of raising an alarm. The break-even point is where they are equal:

p * C_FN  =  (1 - p) * C_FP

read as “p times the cost of a miss equals one minus p times the cost of a false alarm”. Solve for p:

p * C_FN + p * C_FP = C_FP
p * (C_FN + C_FP)   = C_FP
p*                  = C_FP / (C_FP + C_FN)

Substitute C_FN = 20 * C_FP and the units cancel:

p* = C_FP / (C_FP + 20*C_FP) = 1 / 21 = 0.048

So p*, read “p-star”, the cost-optimal threshold, is about 0.048 — not 0.5. Every row the model scores between 0.048 and 0.5 should be alarmed and, on the default threshold, is not.

def cost_optimal_threshold(c_fp: float, c_fn: float) -> float:
    """Predict positive when expected cost of a miss exceeds a false alarm.
    p*C_fn > (1-p)*C_fp  ->  p > c_fp / (c_fp + c_fn)"""
    return c_fp / (c_fp + c_fn)

The trap this produces is comparing two models at an inherited threshold, which is worked through in mini-case D and derived at length in Choosing a threshold from the cost matrix.

Calibration, when a probability is consumed as a number

Calibration means that among rows the model scores 0.30, about 30% really are positive — a property that is sometimes irrelevant and sometimes the whole game.

Whether calibration matters depends on what happens to the score after the model emits it.

• The score is only thresholded (flag / do not flag). Then only its ranking matters, and AUC is the right metric.
• The score is multiplied by something — expected revenue, expected loss, a bid. Then calibration is the metric, and AUC cannot see it.

AUC cannot see it for a specific reason. Any monotone transform of the scores — any relabelling that preserves their order, such as squaring every score — leaves AUC completely unchanged, because AUC only ever asks which of two scores is larger. Squaring turns 0.30 into 0.09, so it destroys calibration while AUC reports no change at all.

To measure calibration, build a reliability table. Bucket the rows by predicted score, then compare each bucket’s mean prediction against the fraction of those rows that actually turned out positive. The two right-hand columns should match; where they do not, the model is lying about its confidence.

bucket        n      mean predicted p     observed rate
0.0 - 0.1   4,120        0.041               0.038
0.1 - 0.3   2,880        0.198               0.212
0.3 - 0.6   1,540        0.442               0.590        <- underconfident here
0.6 - 0.9     980        0.744               0.881
0.9 - 1.0     480        0.951               0.972

ECE = SUM (n_b/n) * |pred_b - obs_b| = 0.042

The middle bucket is the worst: the model says 0.442 and reality says 0.590, a gap of 0.148.

ECE is the expected calibration error: the average of those gaps, weighted by how many rows each bucket holds. Read the formula as “the sum over buckets b of the bucket’s share of rows, n_b/n, times the absolute gap between its mean prediction and its observed rate”. On 10,000 rows total, that weighted average comes to 0.042.

An ECE of 0.042 means an expected-value calculation built on these scores is off by four points of probability on average, and every AUC-based report will say the model is fine.

Two standard fixes, both fitted on a held-out calibration split — rows not used to fit the model:

• Isotonic regression fits a monotone step function from raw score to calibrated probability.
• Platt scaling fits a one-dimensional logistic curve instead, which is smoother and needs less data.

Re-fit on every deploy. Calibration is the first thing distribution shift breaks — live traffic drifting away from what the training window contained (Platt scaling vs isotonic regression).

The metric is a proxy for something else

The last case is the one no diagnostic catches, because the metric is computed correctly and still points the wrong way.

Every offline metric stands in for something the business actually wants, and optimizing hard against the stand-in eventually produces the stand-in without the thing.

• Click-through rate (CTR) stands in for “the user found this useful”. Optimize it and you get clickbait.
• Session length stands in for engagement. Optimize it and you get confusion.
• Per-token accuracy stands in for a good answer. Optimize it and you get hedging.

When the offline metric improves and the business metric does not, the burden of proof is on the offline metric.

A proxy that has been optimized against for two quarters is no longer a proxy. That is Goodhart’s law — a measure ceases to be a good measure once it becomes a target — and it is the same mechanism as reward hacking in Rlhf the pipeline this curriculum actually runs on. Why offline ranking metrics disagree with online ctr works through why offline ranking metrics and online click-through rate routinely disagree.

Step 5 — Is the label wrong?

Stored answers are a measurement with their own error rate. That error rate imposes a ceiling — derivable in two lines — and the wrong labels can be found cheaply.

Assumption on trial: the test label is the truth. When it is not, the test set stops being a ruler and becomes a second noisy model that yours is being scored against — and above a certain point the scoring can rank a better model below a worse one.

The noise ceiling, derived

How high can a perfect model score against imperfect labels? Two lines of probability answer it.

Two symbols carry the whole derivation:

• a — the model’s accuracy against the true label. Nobody can observe this directly, which is the point.
• eta — the Greek letter η, used throughout this chapter as the label error rate: the probability that any given stored label was flipped away from the truth, independently of every other label.

The binary case

You score the model against the stored label, not the true one. So the model gets credit for a row in two disjoint ways:

1. The prediction was right and the label was not flipped: a * (1 - eta).
2. The prediction was wrong and the label was flipped to match the wrong answer: (1 - a) * eta.

Add them:

observed = P(pred right) * P(no flip) + P(pred wrong) * P(flip)
         = a*(1 - eta) + (1 - a)*eta                      (BINARY: C = 2)

invert:    a = (observed - eta) / (1 - 2*eta)

Read the first line as “the observed accuracy equals the chance the prediction is right times the chance the label was not flipped, plus the chance the prediction is wrong times the chance the label was flipped”. Read the inversion as “true accuracy equals observed minus eta, all divided by one minus two eta”. The inversion is just the first line rearranged for a.

Now set a = 1 — a perfect model — and the second term disappears:

at a = 1 (a PERFECT model):  observed = 1*(1 - eta) + 0*eta = 1 - eta

A perfect model cannot score above 1 - eta. That is the ceiling, and it is the number that matters most in this chapter.

What the ceiling costs you, in a table

The second column below is 1 - eta. The third runs the inversion on a measured 0.85 — for example at eta = 0.12 that is (0.85 - 0.12) / (1 - 0.24) = 0.73 / 0.76 = 0.961.

With 12% label noise, a measured 0.880 is what a perfect model scores, and a measured 0.850 corresponds to a true accuracy of 0.961. Your model is far better than the report says, and no amount of further training will move the reported number.

The last row is the useful one. At eta = 0.20 a perfect model tops out at 0.800, so an observed 0.85 implies a true accuracy above 1.0, which is impossible. An observed score above 1 - eta is therefore proof that your noise estimate is wrong, your test set is leaking, or the noise is not symmetric.

The derivation above is binary only

Reason 2 in the derivation — “the prediction was wrong and the label flipped to match it” — is a two-class statement. With two classes there is only one other class, so a wrong prediction and a flipped label always coincide.

With C classes they do not. A flip sends the label to one of the other C - 1 classes, so a wrong prediction matches the flipped label only 1/(C - 1) of the time:

observed = a*(1 - eta) + (1 - a) * eta/(C - 1)

invert:    a = (observed - eta/(C-1)) / (1 - eta - eta/(C-1))

The ceiling 1 - eta is unchanged for every C: set a = 1 and the second term vanishes regardless. That is why the “max observed accuracy” column of the table above survives the generalization and the “true accuracy implied” column does not.

Read that last column as binary only, and use the C-class inversion when your task is not. The 3-class example later in this step is not binary; there, an observed 0.85 at eta = 0.12 gives

(0.85 - 0.12/2) / (1 - 0.12 - 0.12/2) = 0.79 / 0.82 = 0.963

rather than the binary answer of 0.961.

Estimating eta

Estimating the flip rate takes a small labelled sample and four steps, the last of which is the one that settles arguments.

1. Draw a random sample of 200 test rows.
2. Have three independent annotators — people who assign labels — label them, blind to the stored label and to each other.
3. Take the majority as the reference; eta is the disagreement rate against the stored label.
4. Report inter-annotator agreement, meaning how often the annotators agree with each other, measured as Cohen’s kappa between annotator pairs.

Step 4 is the one people skip, and it is also the one people get wrong when they do it. A kappa is not an accuracy, and quoting it directly as a ceiling understates the ceiling badly.

Step one: undo the chance correction

Cohen’s kappa (the Greek letter κ) is chance-corrected. It starts from the raw agreement rate, subtracts off the agreement two annotators would have reached by guessing independently, and rescales so 1.0 is perfect and 0.0 is chance. Two symbols:

• p_o, read “p-observed” — the raw fraction of items on which two annotators actually agreed.
• p_e, read “p-expected” — the agreement they would have reached by chance given the class frequencies. For a balanced two-class task, p_e = 0.5^2 + 0.5^2 = 0.50. For a 90/10 split, p_e = 0.9^2 + 0.1^2 = 0.82.

That subtraction has to be undone before kappa says anything about how often the annotators actually agreed:

kappa = (p_o - p_e) / (1 - p_e)      ->    p_o = kappa(1 - p_e) + p_e

kappa = 0.62, raw agreement p_o by task shape:
   balanced binary    p_e = 0.50  ->  p_o = 0.62*0.500 + 0.500 = 0.810
   balanced 3-class   p_e = 0.333 ->  p_o = 0.62*0.667 + 0.333 = 0.747
   90 / 10 binary     p_e = 0.820 ->  p_o = 0.62*0.180 + 0.820 = 0.932

Read the rearrangement as “p-observed equals kappa times one minus p-expected, plus p-expected”.

The conversion depends on the class balance, so there is no single number to memorise. The same kappa of 0.62 means 0.75 agreement on one task and 0.93 on another. Compute p_e from your own label distribution before you quote anything.

Step two: turn agreement into a ceiling

This is a second calculation and it produces a different number. Do not stop at p_o.

Suppose two annotators label independently, each with the same error rate eta. They agree in two cases: both are right, or both are wrong in the same way. In the binary case that is

(1 - eta)^2 + eta^2 = p_o

read as “one minus eta, squared, plus eta squared, equals p-observed”. Substitute the balanced-binary p_o = 0.81 and solve the quadratic:

1 - 2*eta + 2*eta^2 = 0.81
2*eta^2 - 2*eta + 0.19 = 0
eta = 0.106      (the root below 0.5)

So the label error rate is about 10.6%, and by the ceiling result above a perfect model scores

1 - eta = 0.894    ->    about 0.89

The ceiling is 0.89, not 0.81. The annotators disagree with each other more often than either disagrees with the truth, because two independent 10.6% error rates compound into a 19% disagreement rate. A model reported above 0.89 is fitting one annotator’s idiosyncrasies and will not transfer.

Finding the mislabeled rows cheaply

Rather than relabelling everything, you can let the model nominate the suspects, because the rows it is most confidently “wrong” about are disproportionately the rows where the model is right and the stored label is wrong.

Rank the test set by per-row loss — highest loss first — and read the top 100 by hand.

Expect a large fraction of them to be label errors rather than model errors. But the fraction is a property of your dataset, not a constant, so measure it instead of quoting one. The worked example below found 38 in 100 on this dataset; published audits of standard benchmarks land in the same general region. The only number that should ever appear in your write-up is the one your own hand-review produced.

The confident-learning version formalizes the same idea. Get out-of-fold predicted probabilities — predictions for each row made by a model that never saw that row in training, so they are honest — and flag rows where the model assigns high probability to a class other than the stored one.

In the listing below, look at the gap between the two probability columns. A row where the model gives the stored label 0.002 and some other class 0.981 is not a row the model found hard; it is a row where the model and the annotator flatly disagree.

rank   stored label   p(stored)   p(argmax)   argmax class   verdict on review
  1    "billing"        0.002       0.981      "shipping"     label wrong
  2    "billing"        0.004       0.955      "shipping"     label wrong
  3    "technical"      0.006       0.943      "billing"      genuinely ambiguous
  4    "shipping"       0.008       0.902      "billing"      label wrong
...
      of the top 100:  38 label errors, 22 ambiguous, 40 real model errors

p(stored) is the probability the model assigned to the label on file. p(argmax) is the probability it assigned to its own top choice — argmax meaning whichever class scores highest.

The last line is the payoff: only 40 of the 100 worst-scoring rows are actually model errors. The other 60 are problems with the test set.

Train noise and test noise are not the same problem

Noise in training labels and noise in test labels have different effects, different self-correction, different costs, and therefore different priorities. The bolded cells are the ones that decide the order of work.

Clean the test set first, for two reasons.

The first is cost, and the cost ratio is just the size ratio — so state both sizes or the “cheaper” claim means nothing. On the running example the training set is 200,000 rows and the test set is 2,000:

200,000 / 2,000 = 100    ->  relabelling the test set is 100x cheaper
                             at the same price per row

Recompute that for your own split. Against a 20,000-row test set the ratio is only 200,000 / 20,000 = 10, and the argument gets weaker in proportion.

The second reason does not depend on size at all: the test set is what you steer by. Until it is clean, you cannot tell whether cleaning the training set helped.

Step 6 — Capacity, then features, then data

This is the branch where you improve the model, and there is a spending order that yields the most information per hour.

Assumption on trial: none — this is the branch you reach when no assumption turned out to be broken, so the model really is the bottleneck. What remains is a budgeting question, and the order below is set by information gained per hour spent, not by preference.

flowchart LR
    C["1. CAPACITY<br/>hours · no new data<br/>DIAGNOSTIC as well as a fix"] --> F["2. FEATURES<br/>days-weeks · adds information<br/>needs offline/online parity"]
    F --> D["3. DATA<br/>weeks-quarters · price known<br/>from the STEP 2 curve"]

    style C fill:#2d6a4f,color:#fff
    style F fill:#40916c,color:#fff
    style D fill:#bc6c25,color:#fff

Here the fills encode cost only — dark green is hours, mid green is days to weeks, orange is weeks to quarters — a different scheme from the two diagrams above, because this chain has no bugs and no clean outcome to reach.

The chain runs left to right in increasing cost. Capacity work — making the model bigger — costs hours and needs no new data, and is a diagnostic as well as a fix. Features cost days to weeks, are the only step that adds information, and require offline/online parity, meaning the same value computed both places. Data costs weeks to quarters, with its price known from the step 2 curve.

1. Capacity first, because it is the only one of the three that is also a diagnostic.

Capacity is, roughly, how many parameters the model has and how flexibly they combine. Double the width or depth, remove regularization, and watch training error rather than validation error. Training error is the right column here because you are asking what the model is able to fit, not what it generalizes to.

config             train err   val err     conclusion
base                 0.171       0.180     --
2x width             0.166       0.179     not capacity-limited: train barely moved
2x width, no wd      0.088       0.181     capacity was there; regularization was binding
4x width, no wd      0.021       0.186     now variance-limited -> data or regularization

wd is weight decay, the penalty on large parameter values, so “no wd” means the handicap is off. Read those four rows in order.

• 2x width moves training error from 0.171 to 0.166 — barely. The model was not straining against its size.
• 2x width with weight decay removed drops training error to 0.088. So the capacity was there all along; regularization was what held it back.
• 4x width, no weight decay drops training error to 0.021 while validation error rises to 0.186. That is variance: the model is now fitting sample-specific structure.

Reading that table top to bottom is the bias-variance diagnosis, and it costs an afternoon. The decisive case is the one this example does not show: training error that will not fall even with the handicap removed means the model is not the limitation. You have a data-representation problem, and going straight to data collection would have burned a quarter to learn the same thing.

2. Features second, because they are the only lever that adds information.

Capacity re-uses the information already in the inputs. A feature brings information that was not there at all. When step 2’s curve shows a high e_inf asymptote, that asymptote is precisely the part more data cannot reach and a new feature can.

The cost is real, though. Every new feature needs three things: a point-in-time-correct offline computation — one that can only see what was knowable at the decision moment — an online implementation, and a parity test between them. That is why features are days and not hours, and why the skew check in step 1c is a permanent tax on this step.

3. Data last, because it is the slowest, the most expensive, and — crucially — the one whose value you can already price.

Step 2’s fitted curve already said it: 4x the data for 3 points of error, 36x for 5. Bringing a number like that to a planning meeting is a completely different conversation from “we need more data.”

The one legitimate reorder

A starved slice jumps the queue. If step 3 found one, targeted collection for that slice goes first.

The reason is that the two purchases are not comparable. 5,000 mobile · es rows is a week of work and worth 1.7 points of overall accuracy. 5,000 more rows of the dominant slice is worth roughly nothing, because the dominant slice is already far out on the flat part of its own curve.

Order by information gained per hour. The step 2 curve and the step 3 table are what make that computable.

Symptom to broken assumption to diagnostic to fix

This table is the playbook in lookup form: find your symptom in the first column and read across. Read the second column first. Naming the assumption turns a symptom into a search with an end, because an assumption can be tested and a symptom cannot.

Several shorthands in that table need expanding.

• ln(C) is the natural logarithm of the number of classes. It is exactly the loss of a model that outputs a uniform guess — for ten classes, ln(10) = 2.3026. A loss pinned at that value means the model is guessing and knows it.
• requires_grad is the PyTorch flag that decides whether a parameter accumulates gradients at all. Set to False, the parameter never moves.
• eval() is the call that switches dropout and other train-only behaviour off. Measure training loss without it and that loss is inflated, which can make validation look better than train for no real reason.
• EV is expected value: the score multiplied by an amount of money.
• PSI is the population stability index. It compares this week’s distribution of a feature against the training window’s and returns a single number that grows as they diverge (Psi and kl computed derives it alongside KL divergence and a chi-square noise floor).
• An A/B test splits live traffic between two variants and compares outcomes.

The last row’s three remedies all do the same thing by different means — make the rare class count for more.

• Class weights multiply the rare class’s contribution to the loss.
• Resampling changes how often its rows appear.
• Focal loss down-weights examples the model already gets right, so the rare, hard ones dominate the gradient (Focal loss rebalancing by difficulty instead of by class).

That row’s two diagnoses also need names. A collapsed head is a final output layer that has settled on emitting one class regardless of input. A degenerate optimum is the situation that rewards it: under enough imbalance, always guessing the majority genuinely does minimize the loss you wrote down, so the model is not malfunctioning — the objective is.

Worked mini-cases

Each of the four cases below follows the same shape: signal, first diagnostic, the assumption that turned out to be false, the mechanism, the fix, and a verification that would have failed if the diagnosis were wrong. Each is chosen because its correct resolution looks wrong at first glance.

A. “AUC 0.94 offline, 0.71 online”

Signal. Offline validation AUC is 0.940 on a time-based split. Two weeks live, online AUC is 0.712 on the same population.

Broken assumption. The feature computed during training is the same quantity as the feature computed at serving time.

First diagnostic. Replay one day of serving logs through the offline scorer and diff the feature vectors per entity. Do not touch the model.

feature            offline mean   online mean   importance share
purchases_7d            0.42          0.19            0.61          <- 61% of the model
tenure_days            88.4          88.6            0.11
support_tickets_30d     0.31          0.30            0.08

The third column, importance share, carries the whole argument, so it needs a definition.

For a tree ensemble, a feature’s gain is the total reduction in loss contributed by every split made on that feature, summed over every tree. Its importance share is that gain divided by the sum of the gains of all features, so the column sums to 1 over the whole feature vector.

A share of 0.61 therefore means this one feature accounts for 61% of the model’s total loss reduction — and it is the one whose offline mean (0.42) is more than double its online mean (0.19). The model’s dominant input is not the same quantity in the two environments.

One caution on reading that column: it says “how much of the model this feature is”, not “how much accuracy you would lose by dropping it”. Those two differ whenever features are correlated, and the second is what the ablation in step 1d measures instead.

Mechanism. The purchases_7d aggregation was computed offline with BETWEEN label_date - 7 AND label_date, so it included purchases made on the label day — after the event the model is supposed to predict. Online at scoring time the window can only reach t - 1. This is a lookahead leak and a skew bug simultaneously: the model’s dominant feature was partly a copy of the label offline, and a genuinely weaker feature online.

Fix. Use point-in-time-correct (as-of) joins, so the offline feature can only see what was visible at the decision timestamp, and then move to one shared implementation for both paths.

Verification — and this is the counterintuitive part. Ignore the first two rows for a moment and read the third:

                 before      after
offline AUC       0.940      0.791
online AUC        0.712      0.784
gap               0.228      0.007

Offline AUC fell by 0.149. Online AUC rose by 0.072. The gap between them collapsed from 0.228 to 0.007.

The correct outcome of fixing leakage is that your offline number gets worse. A team that treats the offline drop as a regression will revert the fix and re-break production. Verify on the gap, never on the offline metric alone.

B. “Accuracy 0.922, and the model is unusable”

Signal. Aggregate accuracy sits comfortably above baseline, while support tickets concentrate in one market.

Broken assumption. The overall accuracy describes what any given user experiences.

First diagnostic. Run slice_report on (device, locale), sorted by error share — the table in step 3. mobile · es is 4.0% of traffic, 25.1% of errors, and 0.510 accuracy.

Mechanism. page_text is non-null for 97% of other traffic and 22% of this slice, because the mobile SDK drops it below a length threshold. The imputer fills nulls with 0, which the model cannot distinguish from a genuine zero, so it emits a confident low score for almost the entire slice (mean p = 0.08 versus 0.47 elsewhere).

Fix, in order. Add a missingness indicator so that “absent” and “zero” are different inputs; add a slice-appropriate fallback feature that the mobile SDK does emit; then collect 5,000 labeled rows from that slice.

Verification. Slice accuracy goes 0.510 -> 0.930 and overall goes 0.922 -> 0.939; the worst-slice metric then goes on the standing dashboard, so the next occurrence is caught by an alarm rather than by tickets.

C. “The model will not learn at all”

Signal. Training loss is pinned at 2.303 for 4,000 steps on a 10-class problem. Since ln(10) = 2.3026, the model is outputting a uniform distribution over the ten classes and never leaves it.

Broken assumption. Each input is paired with its own label.

First diagnostic. Overfit 8 samples. It fails: the loss stays at 2.3025 for 400 steps on a batch that a linear model could memorize.

The two columns below are step number and loss. Compare them row by row — the healthy run falls three orders of magnitude while this run does not move at all:

healthy (8 samples, 10 classes)     this run
   0   2.3026                          0   2.3026
  50   1.9034                         50   2.3025
 200   0.1107                        200   2.3026
 400   0.0009                        400   2.3025

Mechanism. Once 8 samples cannot be memorized, learning rate, initialization, and capacity are all excluded by construction, so the answer is in the data.

Printing one batch and inspecting the pairs finds it: the dataset builder shuffled the image list and the label list with two separate random.shuffle calls, so each image ended up next to some other image’s label.

Every label was valid, every image was valid, and the pairing was uniformly random. Given a random pairing, the best possible prediction — the Bayes-optimal output, meaning the best any model could do with the information available — is exactly the uniform distribution the model settled on. The model was right. The data was a permutation.

Fix. Shuffle indices, not lists, so that images and labels move together. Then add a permanent assertion that the loaded pair’s filename stem matches the label record’s key.

Verification. The 8-sample test reaches a loss below 1e-3 — one thousandth — in 400 steps, and full training reaches 0.34 validation loss. The single-batch test then becomes a continuous-integration gate, a check that runs automatically on every commit, because this class of bug costs days and is caught in ten minutes.

D. “F1 went from 0.62 to 0.68 and alerts dropped 38%”

Signal. The offline report says the new model is better on F1. After deploy, alerting volume fell by 38% and analysts started finding missed cases.

Broken assumption. A threshold tuned for one model means the same thing for another.

First diagnostic. Evaluate both models at the deployed threshold, not at each model’s own tuned threshold.

Three rows, and the trick is that the second and third are the same model:

                       threshold   precision   recall    F1
model A (deployed)        0.50       0.55       0.71     0.620
model B (offline report)  0.28       0.62       0.76     0.683   <- what the report measured
model B (as deployed)     0.50       0.83       0.44     0.575   <- what production got

At 0.28, B beats A on F1 (0.683 against 0.620). At 0.50, the same B loses to A (0.575 against 0.620), because its recall collapses from 0.76 to 0.44 — it stops flagging things. The 38% drop in alert volume is that recall collapse.

Mechanism. Model B is genuinely better: it dominates A on the precision-recall curve, meaning at any recall you pick, B has the higher precision.

But B was trained with different regularization, and its score distribution is shifted left — B’s scores are systematically lower for the same underlying risk. So the same numeric threshold sits at a completely different operating point on B than on A.

A threshold-dependent metric compares two models at a point, and a model that moves its score distribution has moved the point. The offline report tuned B’s threshold to 0.28. The deploy inherited A’s 0.50.

Fix. Ship the threshold as part of the model artifact, chosen on a held-out set by the actual cost ratio (step 4), and never as a constant in the serving code. Report threshold-free curves such as PR-AUC and the metric at the exact operating point you will deploy.

Verification. Re-deploy B at 0.28: recall goes 0.44 -> 0.76, alert volume returns above A’s, and precision remains better than A’s 0.55. Then add a deploy-time check that fails if the new model’s predicted-positive rate at the configured threshold differs from the incumbent’s by more than a set tolerance — that single guard catches this entire class of failure.

Cheat sheet

One row per thing an interviewer can ask. The right-hand column always carries a mechanism or a number, because “run error analysis” is not an answer and “4% of traffic, 25% of errors” is.

Next: the math track, starting with 01 — Probability — the estimators, intervals, and hypothesis tests that every number in this chapter relies on.