21 — Design A Metrics Monitoring And Alerting System

Design a monitoring system: metrics from every service, dashboards, and alerting.

The problem is usually answered with a pipeline diagram — agents, a queue, a store, Grafana. The quantity that actually sizes the system is the number of distinct label combinations that are actively receiving samples. Get that wrong by a factor of a thousand, which is one careless label, and the fleet is a thousand times larger.

Five terms first:

• A metric is a named quantity a process reports about itself: http_requests_total, process_memory_bytes.
• A label is a key/value pair attached to a metric so you can slice it: service="api", status="500". Labels are how you ask “errors, but only on checkout.”
• A series is one metric name plus one complete set of label values. http_requests_total{service="api", endpoint="/checkout", status="500"} is one series. Change any single label value and you have a different series.
• A sample is one (timestamp, value) pair appended to a series. A scrape collects one sample for every series a process currently exposes.
• A series is active if it is still receiving samples. Active series are the ones the system must keep resident in memory. A series nobody writes to any more has been flushed to disk and costs almost nothing.

Cardinality is the count of active series, and it is a product. Three services x five endpoints x four status codes is 60 series. Add one more label carrying a thousand distinct values and the whole product is multiplied by a thousand. That multiplication is the entire chapter.

Five numbers carry the rest, each settling a claim an interviewer will otherwise argue about:

Estimation templates and the latency table are ch 02; LSM write amplification is Lsm trees vs b trees; the log that the ingest tier’s write-ahead buffer is built from is Why a log beats a queue 328 devices or one.

1. Framing: what decision, and what breaks

The decision is what you are willing to let a service developer put in a label.

Each label name is a dimension. Each of its distinct values is a coordinate along that dimension. The set of series is the Cartesian product — every combination of one value from each dimension — and that product is what you buy machines for.

Everything else you might argue about in this interview — push or pull, which compression, how many retention tiers — moves the answer by a factor of two or three. Cardinality moves it by a factor of ten thousand. Spend your time there.

What breaks is the ingest tier, and it breaks by falling behind rather than by failing.

A monitoring system that returns errors is annoying. A monitoring system that silently drops the 4% of samples belonging to the service that is currently on fire is worse than having no monitoring at all, because people trust it.

So two properties are non-negotiable. Ingest availability sits strictly above query availability — you can lose a dashboard for a minute and recover, but a dropped sample never comes back. And load shedding is loud, attributed, and bounded: when the system refuses data it says so, names who and what it refused, and refuses a known amount (The write path fan in and why the door says 429).

Requirements

Functional

Three metric types cover everything the fleet reports, and they cost wildly different amounts of storage:

• A counter only ever goes up: total requests served, total bytes sent. You never read a counter directly; you ask the server for its rate().
• A gauge goes up and down: memory in use, queue depth, temperature.
• A histogram is a set of counters, one per latency bucket (“requests that took under 10 ms”, “under 25 ms”, …), plus a running _sum and _count. One histogram is therefore many series, which is why it dominates the estimate below.

With those defined:

• Ingest counters, gauges, and histograms from every process in the fleet, tagged with a label set.
• Query by label selector over a time range, with aggregation pushed down into the store rather than done in the dashboard. The three aggregations that matter: sum adds series together, rate() converts a counter into per-second change, and histogram_quantile() reads a percentile back out of histogram buckets.
• Alert rules evaluated on a schedule, with a for duration (the condition must hold continuously for that long before the alert fires) and a routing/deduplication layer.
• Retention tiers: full resolution for recent data, downsampled for long-term.
• Multi-tenancy with per-tenant series limits. A tenant is one team or environment sharing the cluster, and the per-tenant limit is the only real defence against cardinality (Cardinality is the whole problem).

Non-functional

Two shorthands are used from here to the end. p99 is the 99th-percentile latency: one query in a hundred is slower than the stated bound. HA is high availability — running more machines than capacity requires so that losing one changes nothing a user sees.

None of these targets was picked because it looked round; each carries the arithmetic or argument that produced it.

Back-of-envelope: cardinality first, everything else second

Assume a fleet of 500 hosts running 20 services, each service exposing 50 endpoints, each endpoint reporting 10 status codes, scraped every 30 seconds. Every service exports the same metrics, so host, service, endpoint, and status are the four label dimensions.

The obvious move is to multiply all four. Do it, then check it. The first line below is the naive answer, and the three lines after it show why it is wrong:

naive product:        500 x 20 x 50 x 10                         =  5,000,000
hosts per service:    500 / 20                                   =  25
independent labels:   500 x 50 x 10                              =  250,000
overcount:            5,000,000 / 250,000                        =  20

A host runs exactly one service. So once you know the host, you know the service — service carries no information the host label did not already carry. Multiplying by it invents 19 combinations per host that can never exist, and counts each real series 20 times.

That is a functional dependency between labels, and the overcount it causes is always exactly the cardinality of the dependent label, here 20x.

Cardinality is the product of the labels that vary independently. Spotting one functional dependency is worth an order of magnitude before you have designed anything.

Now the real total. Three groups of series, added at the end. A counter contributes one series per label tuple. A histogram contributes one series per bucket plus one for _sum and one for _count, so 12 buckets becomes 14 series — that is where the largest group comes from:

request counters:     500 x 50 x 10                              =  250,000
latency histograms, 12 buckets + sum + count:
                      500 x 50 x 14                              =  350,000
process + host + runtime, ~300 per host:
                      500 x 300                                  =  150,000
active series:        250,000 + 350,000 + 150,000                =  750,000
samples/s:            750,000 / 30                               =  25,000
samples/day:          25,000 x 86,400                            =  2,160,000,000

The samples/s line divides by 30 because every one of the 750,000 series gets exactly one new sample per 30-second scrape.

A note on the seconds-per-day constant. While you are doing arithmetic in your head, round 86,400 s/day up to 1e5 — it is one digit and it errs high, which is the safe direction for a capacity estimate. Every number reported below as a result is recomputed with the real 86,400 (ch 02 explains the convention; it is for estimating, not for reporting).

25,000 samples/s is a small number. One machine handles it easily, which is the point: monitoring is not a throughput problem, it is a cardinality problem. The two are easy to confuse because both get described as “how much data,” but the fleet you buy is set by how many series you must keep resident, not by how many samples per second arrive.

API sketch

Three methods — and no delete, no update, no unbounded write buffer. write raises rather than queues, and query_range takes a resolution; both are load-bearing design decisions defended later in the chapter.

class MetricsStore:
    def write(self, tenant: str, samples: list[tuple[dict, int, float]]) -> int:
        """samples = [(labels, timestamp_ms, value)]. Returns accepted count.
        Raises TooManySeries (HTTP 429) rather than queueing. See section 2.8."""

    def query_range(self, tenant: str, selector: str, start: int, end: int,
                    step_s: int) -> dict:
        """selector is a label matcher + aggregation. step_s lets the store pick
        a retention tier instead of decoding samples the caller will discard."""

    def series(self, tenant: str, matchers: list[str]) -> list[dict]:
        """Label sets only. This is the endpoint that answers 'what exploded'."""

step_s on the query is not a convenience. It is the parameter that lets the store answer a 30-day panel from the 5-minute tier instead of decoding 8.6 million raw samples to render 1,000 pixels (Retention tiers and the surprise that downsampling does not save storage).

Data model: series, chunk, block

Four nested units, smallest to largest: samples group into chunks, chunks group into blocks, and the index is the lookup table that finds series inside them.

series      = metric name + sorted label set  ->  a 64-bit series ref
sample      = (series ref, timestamp ms, float64)
chunk       = 120 consecutive samples of ONE series, compressed together
block       = 2 hours of chunks + an inverted index, immutable once written
index       = label=value  ->  sorted list of series refs (a postings list)

Two lines in that block deserve unpacking. The series ref is a 64-bit integer the store assigns to each distinct label set, so that a sample on the wire carries 8 bytes instead of a repeated 300-byte label set. A postings list is the classic inverted-index structure: for each label=value pair, the sorted list of series refs carrying it. Looking up {status="500"} means fetching one such list rather than scanning every series.

The unit that matters is the chunk. Samples of one series are stored adjacently and compressed against each other. That adjacency is what makes Delta of delta and xor 16 bytes down to 14’s compression possible and what turns a range query into a handful of reads instead of thousands of lookups.

A design that stores samples keyed by timestamp instead of by series has thrown away the only locality this workload has. Every other decision in the chapter follows from keeping that locality.

High-level architecture

Trace two paths through the diagram. The write path runs down the left: targets are scraped, samples land in a local write-ahead log, a distributor hashes each series to three ingesters, ingesters flush two-hour blocks to object storage, and a compactor merges and downsamples them. The read path runs up the right: a query frontend splits a request by time and answers recent data from the ingesters and older data from a store gateway reading object storage.

The one box that is not on either path is the red one. That is the shed door — the point where the system refuses data rather than buffering it, and the reason it exists is The write path fan in and why the door says 429.

flowchart TB
    T1["targets · 2,500 /metrics endpoints"] -->|"pull, 30 s"| SC["scrapers<br/>sharded by hash(target)"]
    SD["service discovery"] -.-> SC
    SC --> WAL["local WAL + head block<br/>2 h in memory"]
    WAL -->|"remote write"| DIST["distributor<br/>hash(series ref) -> ingester"]

    DIST -->|"429 on limit breach"| SHED(["shed at the door<br/>never queue"])
    DIST --> I1["ingester 1 · RF 3"]
    DIST --> I2["ingester 2"]
    DIST --> I3["ingester 3"]

    I1 --> BLK["2 h blocks -> object storage"]
    BLK --> CMP["compactor<br/>merge + downsample 5 m, 1 h"]
    CMP --> OBJ[("object storage<br/>194 GB per replica")]

    Q["query frontend<br/>split by time, cache"] --> I1
    Q --> SG["store gateway<br/>reads blocks + index"]
    SG --> OBJ

    RUL["ruler<br/>evaluates every 15 s"] --> Q
    RUL --> AM["alertmanager<br/>group, dedupe, silence, route"]

    style DIST fill:#495057,color:#fff
    style I1 fill:#1d3557,color:#fff
    style I2 fill:#1d3557,color:#fff
    style I3 fill:#1d3557,color:#fff
    style OBJ fill:#1d3557,color:#fff
    style SHED fill:#9d0208,color:#fff
    style CMP fill:#40916c,color:#fff
    style Q fill:#2d6a4f,color:#fff
    style SG fill:#2d6a4f,color:#fff
    style AM fill:#bc6c25,color:#fff

Two labels in that diagram are shorthand. WAL is a write-ahead log: an append-only file a process writes and flushes before it acknowledges, so a crash loses nothing already accepted. RF 3 is a replication factor of three — every series is held on three ingesters, so losing one loses no data.

Colours follow ch 01’s key: blue is the authoritative copy of the data — here the ingesters holding the head block and the object storage holding everything older; green is read capacity; light green is work taken off the request path that answers no read; orange is a box forced by something other than processor time; red is the step you cannot undo; grey is the plane that routes and watches but stores nothing.

2. Deep dive

1. Cardinality is the whole problem

What one active series costs

Every active series costs memory whether or not anyone ever queries it. To accept the next sample for a series, the process must already hold three things in RAM: the label set, the index entry pointing at it, and the half-filled chunk the sample will be appended to. None of that can live on disk, because a sample arrives every 30 seconds.

Build the per-series cost from parts. Each line is one resident structure, in bytes:

label set, 10 labels x 30 B                                      =  300
series ref -> label set map entry                                =  48
inverted index postings, 10 postings x 8 B x 2                   =  160
closed head chunk, 120 samples x 1.4 B + 40 B header             =  208
open head chunk                                                  =  208
WAL / checkpoint share                                           =  100
bytes per series: 300 + 48 + 160 + 208 + 208 + 100               =  1,024
live heap to RSS at a 2x GC target: 1,024 x 2                    =  2,048

Three of those lines need unpacking.

The 10 postings x 8 B x 2 line: each of the 10 labels puts this series’ 8-byte ref into one postings list, and the x 2 is slack — postings lists are grown by doubling, so on average half the allocated array is empty.

The 120 samples x 1.4 B line: 1.4 bytes per compressed sample is Delta of delta and xor 16 bytes down to 14’s result, used here rather than re-derived. There are two chunks resident because the just-filled one is still being held while the next one fills.

GC is garbage collection. RSS is resident set size — the memory the operating system is actually holding for the process. RSS is what a 64 GB box has 64 GB of.

The last line is the one that trips people. A garbage collector configured to run when the heap reaches twice the live data leaves roughly half of RSS as garbage awaiting collection. So the live heap cost of a series is 1,024 B, but its RSS cost is 2,048 B.

Divide an RSS budget by an RSS figure. Dividing 48 GB of RSS by 1,024 B of live heap gives 46,875,000 series and overstates the box by exactly 2x. 2 KB per active series is the number to carry.

The baseline: 3% of one machine

Now size the fleet. The 75% figure below is the usable fraction of a box’s RAM after the operating system, page cache, and query working memory take their share:

baseline memory:      750,000 x 2,048                            =  1,536,000,000
usable RSS per 64 GB box, 75% of 64 GB:                             48,000,000,000
series per box:       48,000,000,000 / 2,048                     =  23,437,500
boxes for baseline:   750,000 / 23,437,500                       =  0.032

1.5 GB. The entire monitoring system for a 500-host fleet fits in 3% of one machine. You run two, and the second one is purely for availability, not capacity.

Add one label: 2,134 machines

A developer wants to break the request counter down by customer, so they add a user_id label. It has 200,000 distinct values per day.

It multiplies only the 250,000 request counters, not the histograms or the process metrics. That is the conservative reading, and it is still catastrophic. ceil() in the block below means “round up to the next whole machine” — you cannot buy 0.032 of a server, and applying the same rounding to both sides is the whole point of the paragraph after it:

with user_id:         250,000 x 200,000                          =  50,000,000,000
explosion factor:     50,000,000,000 / 750,000                   =  66,667
boxes of CAPACITY, baseline:      ceil(0.032)                    =  1
capacity needed, with user_id:  50,000,000,000 / 23,437,500      =  2,133.33
boxes of CAPACITY, with user_id:  ceil(2,133.33)                 =  2,134
ratio:                2,134 / 1                                  =  2,134

cost/year at $1.00/box-hour, both sides sized the same way
  baseline:           1 x 8,760 x 1.00                           =  8,760
  with user_id:       2,134 x 8,760 x 1.00                       =  18,693,840
  ratio:              18,693,840 / 8,760                         =  2,134

Both sides of that ratio are bare capacity — no redundancy on either, and both rounded up to a whole machine, because capacity is bought in whole machines and rounding the two sides by different rules is how a ratio starts lying. Redundancy does not cancel out of the comparison by itself: the exploded fleet needs its own replicas on top of 2,134, so adding HA to both sides moves both numbers and leaves the ratio where it is.

One label took an $8,760/year capacity bill to an $18.7 M/year one — 2,134x — and the pull request that did it was one line long. That is why the defence against it is a hard per-tenant limit, not a code review guideline.

State whether a rate is offered load or service capacity at the moment you derive it, and never divide one by the other without saying which is which. The same discipline applies to a fleet count: say whether the number is capacity-required or machines-provisioned, and never put one on each side of a ratio.

The request_id case: 731x more index than data

user_id at least has a ceiling — you only have so many customers. request_id has none: a new value is minted for every request forever. Work that case through, because it shows what kind of mistake an unbounded label is.

Assume each of the 500 hosts serves 100 requests/s. Every request creates a series that receives exactly one sample and is never written to again:

fleet request rate:   500 x 100                                  =  50,000
requests/day:         50,000 x 86,400                            =  4,320,000,000
series/day:                                                         4,320,000,000
samples per series:                                                 1
index bytes:          4,320,000,000 x 1,024                      =  4,423,680,000,000
sample bytes:         4,320,000,000 x 1.4                        =  6,048,000,000
index to sample:      4,423,680,000,000 / 6,048,000,000          =  731

4.4 TB of index describing 6.0 GB of data — 731x more metadata than data, with exactly one sample per series.

Every optimization in this chapter assumes many samples share one label set. A label with one distinct value per event breaks that assumption completely: it turns a time-series database into a log store, with none of a log store’s optimizations.

The rule follows directly: a label is legal only if its value set is bounded and does not grow with traffic.

status, region, version, instance are fine — new values appear rarely and the set has a ceiling. user_id, request_id, url, error_message are not, and no amount of tuning makes them fine. Those belong in logs or traces, where the storage engine indexes high-cardinality fields on purpose and charges per event instead of per series.

The arithmetic, as code

The functions below are the two rules above made executable: series() refuses to multiply dependent labels, and boxes_of_capacity() applies one rounding rule to both sides of a comparison. The asserts reproduce every figure in this section.

from math import ceil, prod


def series(labels: dict, dependent: tuple = ()) -> int:
    """Product of INDEPENDENT label cardinalities. `dependent` names labels that
    are a function of one already counted: a host runs one service, so `service`
    must not multiply the product."""
    return prod(v for k, v in labels.items() if k not in dependent)


def boxes(n_series: int, bytes_per_series: int = 2_048,
          rss_per_box: int = 48_000_000_000) -> float:
    """FRACTIONAL machines of capacity. `rss_per_box` is a resident-memory
    budget, so `bytes_per_series` must be the post-GC 2,048 and not the 1,024
    of live heap."""
    if n_series < 0 or bytes_per_series <= 0 or rss_per_box <= 0:
        raise ValueError("a series count and a memory budget are non-negative")
    return n_series * bytes_per_series / rss_per_box


def boxes_of_capacity(n_series: int, **kw) -> int:
    """Capacity is bought in whole machines, so ceil BOTH sides or the ratio lies."""
    return ceil(boxes(n_series, **kw))


BASE = {"host": 500, "service": 20, "endpoint": 50, "status": 10}
assert series(BASE) == 5_000_000                       # naive product
assert series(BASE, dependent=("service",)) == 250_000  # service depends on host
assert series({**BASE, "user_id": 200_000}, dependent=("service",)) == 50_000_000_000
assert round(boxes(750_000), 3) == 0.032
assert round(boxes(50_000_000_000), 2) == 2133.33
# One rounding rule, applied to both sides. `round` would send the baseline to
# 0 and make the ratio undefined; `round` on one side and "round up" on the
# other is what produced 2,133.
assert boxes_of_capacity(750_000) == 1
assert boxes_of_capacity(50_000_000_000) == 2134
assert boxes_of_capacity(50_000_000_000) // boxes_of_capacity(750_000) == 2134
try:
    boxes(-1)                     # -4.27e-08 machines is not a small fleet
except ValueError:
    pass
else:
    raise AssertionError("a negative series count must be refused, not returned")

Two defences, and they are not interchangeable

A limit on total active series per tenant bounds the damage. It stops the ingester dying. It tells you nothing about the cause, so you still get paged and still have to go looking.

A limit on distinct values per label name, enforced at ingest and reported with the offending label name in the rejection body, is what turns a 3 a.m. page into a one-line fix. The rejection says user_id and the engineer knows exactly which pull request to revert.

Run both. Then alert on the rate of new series creation, d(series)/dt, rather than on memory. An explosion is visible in the creation rate within a minute or two; it is only visible in memory once the ingester is already most of the way to dying.

2. Push vs pull collection, and what each costs

Neither model wins on bandwidth

The usual interview answer compares bytes on the wire. Do that comparison once, get it out of the way, and then never use it again.

Pull: the monitoring server holds a list of targets from service discovery — the registry that says which processes currently exist and where — and scrapes GET /metrics from each one on a schedule. Assume 5 processes per host:

targets:              500 x 5                                    =  2,500
scrapes/s:            2,500 / 30                                 =  83.3
series per target:    750,000 / 2,500                            =  300
exposition bytes:     300 x 100                                  =  30,000
bandwidth:            83.3 x 30,000                              =  2,499,000
fraction of a 1 Gbps NIC:  2,499,000 / 125,000,000               =  0.020

The 100 in the exposition line is bytes of text per series in the /metrics response — a metric name, its labels, and a number. The 125,000,000 is 1 Gbps converted to bytes per second (1e9 / 8).

2% of one NIC — one network interface card, the 1 Gbps port connecting a machine to the network.

Push: the process sends samples to a collector on its own schedule, with no target list involved. Labels are re-sent with every sample in a naive protocol, which is where the 200 bytes comes from:

samples/s:            750,000 / 30                               =  25,000
wire bytes/sample, labels re-sent each time:  200 + 16           =  216
push bandwidth:       25,000 x 216                               =  5,400,000
ratio to pull:        5,400,000 / 2,499,000                      =  2.16

2.16x, and even that is an artifact. Label interning — sending each label set once and referring to it by an integer afterwards — erases the gap entirely. Do not argue push vs pull on bytes.

The four properties that actually decide it

Argue it on these instead. One term needs naming first: backpressure is a receiver’s ability to make a sender slow down, as opposed to absorbing the excess in a buffer or dropping it silently. Pull has backpressure built in; push does not.

The blast-radius row is the one to quantify, because it is the same cardinality failure from Cardinality is the whole problem seen from the collection side.

sample_limit is a pull-side setting: if a target’s /metrics response contains more than N series, the scraper discards the entire response and records the scrape as failed. Watch what that does to a bad deploy on 100 of the 2,500 targets:

one bad deploy, 100 targets, user_id added to one metric:
new series per target: 50 x 10 x 200,000                         =  100,000,000
across the deploy:    100 x 100,000,000                          =  10,000,000,000
with sample_limit = 10,000, admitted per target:                    0

With a pull-side sample limit the scrape fails wholesale, admits nothing, and fires up == 0 on 100 targets. With push there is no such door, and 10 billion series arrive. Pull converts an unbounded global failure into a bounded local one plus an alert — which is the entire argument, and it has nothing to do with bandwidth.

What pull costs you

Pull’s two real costs, and both are honest:

• Service discovery is now a hard dependency. No SD, no targets, no metrics.
• The interval quantizes everything. Sampling any signal at interval T can only resolve features longer than 2T — the Nyquist limit, and the reason is intuitive: to see a bump you need at least one sample inside it and one outside it. At a 30 s scrape that floor is 30 x 2 = 60 seconds, so a 5-second latency spike is invisible no matter how good the storage is. A counter still captures the spike’s total, because a counter accumulates between scrapes and the next scrape reports the accumulated value. That is why you export counters and let the server compute rate(), rather than exporting a pre-computed rate that gets sampled and loses the spike entirely.

The practical answer is both: pull for anything long-lived that you can reach, and a push gateway for batch jobs and functions that will not exist by the next scrape.

3. Why a general key-value store is the wrong shape

The obvious cheap answer to “where do the samples go” is a general key-value store — Cassandra, RocksDB, DynamoDB. It loses on writes by two to three orders of magnitude and on reads by two, and both losses are structural.

Writes: the payload is the smallest part

An LSM tree (log-structured merge tree) is the storage engine most modern key-value stores use. It buffers writes in memory, flushes them to a sorted file on disk, and then repeatedly compacts — merges small files into larger ones to keep read performance from degrading. Every merge rewrites data that was already on disk. That rewriting is the cost.

A sample’s payload is three fields: (series ref 8 B, timestamp 8 B, value 8 B) = 24 bytes. Put that in an LSM key-value store and the payload becomes the smallest part of the cost:

sample payload:       8 + 8 + 8                                  =  24
LSM row after compaction: key 16 + value 8 + header 20 + index share  =  50

leveled compaction, write amp 42x (sql 03 sec 6):  50 x 42       =  2,100
KV disk write bytes/s:  25,000 x 2,100                           =  52,500,000
TSDB disk write bytes/s: 25,000 x 1.4                            =  35,000
ratio:                52,500,000 / 35,000                        =  1,500

size-tiered instead, write amp 4.5x:               50 x 4.5      =  225
KV disk write bytes/s:  25,000 x 225                             =  5,625,000
ratio:                 5,625,000 / 35,000                        =  161

Between 161x and 1,500x more disk written per sample, depending on which compaction strategy the key-value store runs.

Unpack the two amplification factors, both taken from Lsm trees vs b trees.

Write amplification is the number of bytes the storage engine actually writes to the device for every byte the application handed it. A 42x amplification means the application’s 50-byte row costs 2,100 bytes of device writes.

Leveled compaction gives 1 (write-ahead log) + 1 (flush to level 0) + T x L = 42, where T = 10 is the size ratio between levels and L = 4 is the number of levels. Each byte is rewritten once per level as it is merged downward.

Size-tiered compaction gives roughly 4-5x instead. It merges files of similar size rather than pushing every byte down a ladder, so it writes far less — but it leaves more files that a read has to search through. It is a write-cost-for-read-cost trade, not a free win.

State which one you mean. Quoting one amplification figure for “an LSM store” is the mistake: the two strategies differ by an order of magnitude, and the choice is a config knob, not a property of the engine.

Two mechanisms produce the gap, and both are structural:

1. Per-row overhead dwarfs a 24-byte payload. A key-value store is built for values orders of magnitude larger than its own bookkeeping. Here the bookkeeping is the record.
2. Nothing is compressed against its neighbours. Each sample is an independent row. A time-series store compresses 120 samples of one series together, which is the only reason Delta of delta and xor 16 bytes down to 14 reaches 1.4 bytes.

Reads: 14,400 lookups against 120

The read path is the same story from the other end. Take a one-hour dashboard panel over 120 series, scraped every 30 s:

samples:              120 x 3,600 / 30                           =  14,400
KV point lookups:                                                   14,400
TSDB chunk reads:     14,400 / 120                               =  120

The TSDB line divides by 120 because a chunk holds 120 samples of one series, so one read returns an hour of one series in a single I/O.

14,400 lookups against 120 reads — and that already assumes the key-value store’s keys sort as (series, timestamp), which means you have hand-built a time-series layout inside it.

Turning I/O counts into time, carefully

It is tempting to multiply 14,400 by an SSD latency and declare the key-value read path too slow. Do not — that argument is wrong, and an interviewer will catch it.

Ch 02’s 100 us SSD figure is a queue-depth-1 latency: how long one read takes when nothing else is in flight. It is not a device throughput ceiling. A modern NVMe drive sustains 500,000 to 1,000,000 random 4 KB IOPS — input/output operations per second — by servicing hundreds of requests concurrently.

So compute both numbers, because they answer different questions:

issued serially, at 100 us each:  14,400 x 0.0001                =  1.44 s
issued at depth, 500,000 IOPS:    14,400 / 500,000               =  0.0288 s  =  28.8 ms
panels/s one device sustains, KV:  500,000 / 14,400              =  34.7
panels/s one device sustains, TSDB: 500,000 / 120                =  4,167

The argument here is throughput, not latency. A store that keeps its queue deep answers a single panel in 28.8 ms, comfortably inside the 1 s target, so do not claim the key-value read path is too slow for one panel — it is not. The throughput claim is the one that decides it: the same device serves 34.7 KV-shaped panels a second against 4,167 TSDB-shaped ones, a 120x gap that is exactly the I/O count above. A dashboard is not one panel on one device; it is thousands of panels against a shared fleet, and 120x fewer device operations per panel is 120x fewer devices.

The index: postings lists, and the query that kills the box

The index is the other half of the store, and it is an inverted index, not a B-tree over rows. It maps label=value -> sorted postings list of series refs, exactly like a search engine mapping a word to the documents containing it.

A selector with two matchers fetches two postings lists and intersects them. Because both lists are sorted 64-bit integers, the intersection can use galloping search — jump ahead exponentially in the longer list to find the next candidate from the shorter one, rather than walking every element. The work is driven by the shorter list, not by the total number of series:

{service="api"} matches:   750,000 / 20                          =  37,500
{status="500"} matches:    750,000 / 10                          =  75,000
intersection cost, galloping search over sorted uint64:             37,500
full scan of all series:                                            750,000
speedup:              750,000 / 37,500                           =  20

The structure also explains the one query that reliably kills the box: a regex matcher cannot use a postings list directly.

A postings list is keyed by an exact label=value pair. endpoint=~".*checkout.*" is not a pair — the store has to enumerate every distinct value of endpoint, test each one against the pattern, fetch a postings list for each match, and union them all. That is 50 lists on this schema. On the exploded schema from Cardinality is the whole problem, user_id=~"..." would be 200,000 lists, in one query, from one dashboard panel.

“Never regex a high-cardinality label” is not a style rule. It is the difference between one postings lookup and 200,000.

4. Delta-of-delta and XOR: 16 bytes down to 1.4

A raw sample on the wire is 8 + 8 = 16 bytes: an 8-byte millisecond timestamp and an 8-byte float64 value. Gorilla-style encoding gets that to 1.4 bytes by exploiting two facts about how metrics are actually produced: scrapes happen on a fixed schedule, and most values barely move between scrapes.

Timestamps and values get different tricks, so take them separately.

Timestamps: delta-of-delta

Do not store the timestamp. Store how much the gap changed:

D = (t[n] - t[n-1]) - (t[n-1] - t[n-2])

On a perfect 30 s schedule, every gap is 30,000 ms, so every gap-of-gaps D is zero. Zero is cheap to encode.

The encoding is a variable-length prefix code: a few leading tag bits say how wide the payload is, so small values cost few bits and rare large values cost more. Each row answers “how far off schedule was this scrape, and what does that cost” — and the Bits column already includes the tag bits:

Real scrapes jitter by a few milliseconds, so D is small but not always zero. Assume 96% of samples land exactly on schedule, 3% land inside the 7-bit bucket, and 1% inside the 9-bit bucket. Those three fractions sum to 1.00, which matters — they are a probability distribution, and the line below is its expected value:

timestamp bits:       0.96 x 1 + 0.03 x 9 + 0.01 x 12            =  1.35

That distribution is an assumption about your scrapers, not a law. A fleet with badly jittering scrapers pays more.

Values: XOR against the previous value

A float64 stores a number as a sign bit, an 11-bit exponent, and a 52-bit mantissa — the significant digits. A CPU gauge going 0.4100 -> 0.4103 keeps the same sign and exponent and changes only the low mantissa bits.

So XOR the two 64-bit patterns together. Identical bits cancel to zero, which leaves a long run of leading zeros, a short window of bits that actually differ, and a long run of trailing zeros. Store only that middle window, and when the next XOR fits inside the same window, re-use it instead of re-describing it:

The third row’s 5 and 6 bits are the window description itself: 5 bits to say how many leading zeros, 6 bits to say how long the meaningful window is.

Now the same expected-value calculation, with a value profile of 50% unchanged, 37% fitting the previous window, and 13% needing a new one:

value bits:           0.50 x 1 + 0.37 x 14 + 0.13 x 32           =  9.84
sample bits:          1.35 + 9.84                                =  11.19
bytes per sample:     11.19 / 8                                  =  1.40
compression ratio:    16 / 1.40                                  =  11.43

16 bytes to 1.40 — 11.4x — and the entire gain comes from storing samples of one series adjacently.

That is the structural reason Why a general key value store is the wrong shape’s key-value layout cannot be rescued with a better compressor. Delta-of-delta needs the previous two samples of the same series; XOR needs the previous one. A layout that never puts two samples of a series next to each other has nothing to compress against.

The encoding, as code

dod_bits walks a run of delta-of-deltas and totals the bits. mean_bits computes the expected value of a profile — and refuses a profile whose fractions do not sum to 1, which is the single easiest way to manufacture a flattering compression number.

def dod_bits(deltas_of_deltas) -> int:
    """Gorilla timestamp encoding: total bits for a run of delta-of-deltas."""
    total = 0
    for d in deltas_of_deltas:
        if isinstance(d, bool) or not isinstance(d, int):
            # A float NaN compares False against every bound and silently takes
            # the 36-bit branch; a bool compares == 0 and takes the 1-bit one.
            # Both are milliseconds that are not milliseconds.
            raise TypeError(f"delta-of-delta must be an int, got {d!r}")
        if d == 0:
            total += 1
        elif -63 <= d <= 64:
            total += 2 + 7
        elif -255 <= d <= 256:
            total += 3 + 9
        elif -2047 <= d <= 2048:
            total += 4 + 12
        else:
            total += 4 + 32
    return total


def mean_bits(profile) -> float:
    """profile = [(fraction, bits)] -> expected bits per value.

    The fractions are a probability distribution over encodings. A profile that
    does not sum to 1 is not a distribution, and its expectation is not bits per
    sample -- which is how a headline of 1.40 B/sample gets computed from a
    profile covering half the samples.
    """
    total = sum(f for f, _ in profile)
    assert abs(total - 1.0) < 1e-9, f"profile fractions sum to {total}, not 1"
    return sum(f * b for f, b in profile)


assert dod_bits([0] * 96 + [5] * 3 + [100]) == 135          # 1.35 bits/sample
VALUE_PROFILE = [(0.50, 1), (0.37, 14), (0.13, 32)]
assert abs(mean_bits(VALUE_PROFILE) - 9.84) < 1e-9
assert abs((1.35 + 9.84) / 8 - 1.399) < 1e-3

for bad in ([(0.50, 1)], VALUE_PROFILE + [(9.0, 0)]):       # 0.5, and 9.99
    try:
        mean_bits(bad)
    except AssertionError:
        pass
    else:
        raise AssertionError(f"{bad} is not a distribution and must be refused")
for bad_d in (float("nan"), False, 30.0):
    try:
        dod_bits([bad_d])
    except TypeError:
        pass
    else:
        raise AssertionError(f"dod_bits accepted {bad_d!r}")

Two design constraints the encoding forces

The compression is not free. It buys those bytes by giving up two things, and both show up in operations:

• Chunks are immutable and append-only. A bit-packed XOR stream cannot have a sample inserted in the middle. Out-of-order writes must go to a separate head or be rejected — which is why “my backfill was silently dropped” is such a common operational surprise.
• Counters compress far better than gauges. A monotonically increasing counter sampled at a steady rate has near-constant deltas, so its XOR window is stable; a noisy gauge takes the 32-bit path more often. Preferring counters plus server-side rate() is a storage decision as much as a semantics decision.

5. Retention tiers, and the surprise that downsampling does not save storage

Downsampling means replacing N raw samples with one summary point covering the same span. A 5-minute tier replaces the ten 30-second samples in each 5-minute window with a single point.

That summary point must carry four aggregates — min, max, sum, count — not one. Keeping only the mean makes it impossible to answer “what was the peak,” and keeping only the sum makes it impossible to divide by the right denominator for a mean or a rate. So one downsampled point costs four values, not one.

Three tiers, at 1.40 bytes/sample. The retention windows are converted to seconds first, so the arithmetic is samples/s x bytes/sample x seconds:

15 days in seconds:   15 x 86,400                                =  1,296,000
90 days in seconds:   90 x 86,400                                =  7,776,000
400 days in seconds:  400 x 86,400                               =  34,560,000

raw, 30 s, 15 days:   25,000 x 1.4 x 1,296,000                   =  45,360,000,000
5-min tier samples/s:  750,000 / 300 x 4                         =  10,000
5-min tier, 90 days:   10,000 x 1.4 x 7,776,000                  =  108,864,000,000
1-hour tier samples/s: 750,000 / 3,600 x 4                       =  833
1-hour tier, 400 days: 833 x 1.4 x 34,560,000                    =  40,303,872,000
total: 45,360,000,000 + 108,864,000,000 + 40,303,872,000         =  194,527,872,000
replicated x2:        194,527,872,000 x 2                        =  389,055,744,000

The two x 4 factors are the four aggregates. The / 300 and / 3,600 are the tier intervals in seconds: 750,000 series produce one summary point each per 5 minutes, and one each per hour.

Total: 194.5 GB per replica, 389 GB replicated — the two figures on the object-storage box in the architecture diagram. Now compare two of those lines against each other:

5-min tier vs raw tier:  108,864,000,000 / 45,360,000,000        =  2.4

The 5-minute tier costs 2.4x more than the raw tier it summarizes.

Both factors are visible in the numbers above. Ten samples become four values, so the rate drops only 10 / 4 = 2.5x per unit time — not 10x, because of the four aggregates. And the tier is kept for 90 days instead of 15, which is 90 / 15 = 6x longer. 6 / 2.5 = 2.4.

Downsampling does not save storage. It buys query speed. Say that in the interview rather than repeating the folk claim, because the folk claim is checkable in two lines of arithmetic and the interviewer may check it.

Where downsampling actually pays

Query cost, not storage cost. Take a 30-day dashboard panel over 100 series (30 x 86,400 = 2,592,000 seconds):

30-day panel, 100 series, raw:   100 x 2,592,000 / 30            =  8,640,000
same panel from the 5-min tier:  100 x 2,592,000 / 300           =  864,000
panel width in pixels:                                              1,000
raw samples per pixel:  8,640,000 / 1,000                        =  8,640

8,640 samples decoded per rendered pixel. The 5-minute tier cuts the decode 10x and changes nothing a human can see, which is why step_s is on the query API — the store, not the dashboard, should choose the tier.

The other reason to keep raw data short has nothing to do with bytes. Raw retention sets how far back an incident review can go at full resolution. 15 days is chosen against “how long until nobody re-opens this postmortem,” not against a disk quota.

Then compare the whole thing with the logging half of observability. 389 GB of metrics in total, against a log pipeline that ch 01 prices at 4 TB per day once the call-graph fan-out is counted.

That gap is why metrics and logs are separate products with separate retention, and the asymmetry is structural: metrics cost per series and stay flat as traffic grows, logs cost per event and grow linearly with it.

6. Alerting: for does not fix a bad alert

Per-series thresholds page 57,600 times a day

Alert on a per-series threshold and the false-alarm rate scales with the fleet, not with the incident rate. That is the whole failure, and it takes three lines of arithmetic to show.

Take a threshold that a perfectly healthy series exceeds with probability 0.001 on any single evaluation. That is not a badly chosen threshold — it is roughly a 3-sigma line, three standard deviations above the mean of a normal distribution, the textbook “this basically never happens by chance” cutoff. Evaluate it every 15 seconds across 10,000 series:

evaluations/s:        10,000 / 15                                =  667
false trips/s:        667 x 0.001                                =  0.667
false trips/day:      0.667 x 86,400                             =  57,600

57,600 pages a day. That is not “noisy,” it is structurally unusable — roughly 40 pages a minute, forever, with nothing wrong.

Why for cannot rescue it

The reflex fix is a for clause: require the condition to hold continuously for some duration before firing. Count how many evaluations for: 5m covers (300 seconds at one evaluation every 15):

evaluations in the window:  300 / 15                             =  20

If those 20 trips were independent — each one a fresh coin flip — the probability of all 20 firing is 0.001 ^ 20 = 1e-60 and the problem is solved forever.

They are not independent, and assuming they are is the mistake. Real metric noise is autocorrelated: a series does not bounce randomly around its mean, it drifts into a bad regime and stays there for a while. If it was above the threshold 15 seconds ago it is very likely above it now.

Model regime lengths as an exponential distribution with a 120-second mean. Under that model, for: 5m survives whenever a regime happens to outlast 300 seconds, and the exponential’s survival probability is e^(-t/mean):

t / mean:             300 / 120                                  =  2.5
P(regime > 300 s):    2.718281828 ^ -2.5                         =  0.0821
false pages/day:      57,600 x 0.0821                            =  4,729
suppression factor:   57,600 / 4,729                             =  12.2

for bought a factor of 12. You needed a factor of 57,600.

That is the lesson: for suppresses flapping within one alert. It cannot rescue an alert whose false-positive rate scales with fleet size, because it does not touch the fleet size term.

The fix: change what is measured

Do not tune the threshold. Change the signal.

An SLO is a service level objective — a stated target for a user-visible quantity, such as “99.9% of checkout requests succeed over 30 days.” Alert on that, which means aggregating the 10,000 series into 20 SLO signals first and thresholding the aggregate:

SLO alerts:           20
evaluations/s:        20 / 15                                    =  1.33
series per group:     10,000 / 20                                =  500
standard error shrinks by:  500 ^ 0.5                            =  22.4
threshold now sits at:  3.1 x 22.4                               =  69.4

The 500 ^ 0.5 line is the standard-error rule: average n independent noisy measurements and the noise of the average shrinks by sqrt(n). Averaging 500 series shrinks it by sqrt(500) = 22.4.

Two independent improvements, and they multiply. 500x fewer evaluations, and 22.4x less noise in what is being evaluated — so a threshold sitting 3.1 standard deviations out on a single series sits 69.4 standard deviations out on the group.

At 69 sigma, random excursions never reach it. Only a real shift in the underlying error rate does. That is what “the alert fires when something is actually wrong” means mechanically.

Symptoms, not causes

The arithmetic above is the justification for a rule usually stated as folklore. Compare the two alert shapes directly:

A cause alert answers a question nobody asked at 3 a.m. Keep the cause metrics — they are how you diagnose after the symptom alert fires — but do not page on them. That is what dashboards and, at most, tickets are for.

7. Burn-rate alerting, derived from the budget

Start from the SLO and let every threshold fall out of it. 99.9% availability over 30 days, and 30 days is exactly 720 hours:

error budget:         1 - 0.999                                  =  0.001
SLO period, hours:    30 x 24                                    =  720
page at 2% of budget over 1 h:   0.02 x 720                      =  14.4

The error budget is what the SLO permits you to fail: at 99.9%, one request in a thousand may fail before you have broken your promise.

Burn rate is the observed error rate divided by the budgeted error rate. Burn rate 1 means you are failing at exactly 0.1% and will consume the whole budget in exactly 30 days. Burn rate 10 means 1% and consumes it in 3 days. Burn rate 1,000 means everything is failing.

The budget consumed by sustaining burn rate B for W hours is B x W / 720.

Then invert it. Choose how much budget you are willing to lose before someone tells you, and the threshold is forced. Willing to lose 2% in an hour? 0.02 x 720 / 1 = 14.4. That is where the famous number comes from — it is not tuned, it is a consequence of one policy decision.

Every row of the table is that same equation with different inputs; the last column is how long the budget would last if the burn continued at exactly the threshold.

Run B x W / 720 on each row to confirm the fifth column really is the policy, not a rounded guess:

1 h  at burn 14.4:    14.4 x 1 / 720                             =  0.02
6 h  at burn 6:       6 x 6 / 720                                =  0.05
24 h at burn 3:       3 x 24 / 720                               =  0.1
72 h at burn 1:       1 x 72 / 720                               =  0.1

The same equation in both directions — budget_burned forward, burn_for inverted — with the bounds that stop you deriving a threshold from a 0-hour window or a 200% budget:

def budget_burned(burn_rate: float, window_hours: float,
                  slo_period_hours: float = 720.0) -> float:
    """Fraction of a 30-day error budget consumed by sustaining `burn_rate`."""
    if burn_rate < 0 or window_hours < 0 or slo_period_hours <= 0:
        raise ValueError("a burn rate and a window are non-negative")
    return burn_rate * window_hours / slo_period_hours


def burn_for(budget_fraction: float, window_hours: float,
             slo_period_hours: float = 720.0) -> float:
    """The threshold that fires after losing `budget_fraction` in the window.

    Both arguments are bounded: you cannot spend more than the whole budget,
    and a zero-length window has no threshold -- it has a ZeroDivisionError.
    """
    if not 0 < budget_fraction <= 1:
        raise ValueError("budget_fraction is a share of the budget, in (0, 1]")
    if not 0 < window_hours <= slo_period_hours:
        raise ValueError("window_hours must be positive and inside the period")
    return budget_fraction * slo_period_hours / window_hours


for args in ((0.02, 0), (2.0, 1), (0.0, 1)):                # 0 h, 200%, 0%
    try:
        burn_for(*args)
    except ValueError:
        pass
    else:
        raise AssertionError(f"burn_for{args} must be refused")
try:
    budget_burned(-14.4, 1)                                 # -2% of a budget
except ValueError:
    pass
else:
    raise AssertionError("a negative burn rate must be refused")

assert abs(burn_for(0.02, 1) - 14.4) < 1e-9
assert abs(budget_burned(14.4, 1) - 0.02) < 1e-12
assert abs(budget_burned(6, 6) - 0.05) < 1e-12
assert abs(budget_burned(3, 24) - 0.10) < 1e-12
assert abs(budget_burned(1, 72) - 0.10) < 1e-12

Why four rows instead of one

A single fast rule misses slow burns. A service failing 0.5% of requests is burning at 0.005 / 0.001 = 5, which never trips the 14.4 threshold — but 720 / 5 = 144 hours means it exhausts the budget in six days while nothing pages.

A single slow rule has the opposite problem: it is blind to a total outage for hours, because a 72-hour average takes hours to move.

And the severities differ because the response differs. 2% of the budget gone in an hour needs a human awake now. 10% gone over three days needs a ticket in the morning.

Detection time falls out of the threshold

Work out how fast the 1-hour rule notices a total outage, where every request fails:

outage burn rate:     1 / 0.001                                  =  1,000
detection, 1 h rule, hours:  1 x 14.4 / 1,000                    =  0.0144
in seconds:           0.0144 x 3,600                             =  51.8

51.8 seconds, which is where the sub-minute requirement in Requirements came from. It was not chosen; it is what the 14.4 threshold implies.

The outage burn rate is 1 / 0.001 because 100% of requests are failing against a 0.1% budget. The rule fires once the 1-hour window’s average burn crosses 14.4, and at burn 1,000 that takes 14.4 / 1,000 of the window.

Why every row has a second, shorter window

The long window has memory, and memory is a problem on the way down.

After a 10-minute total outage ends, the 1-hour window still contains those ten bad minutes. Its average stays above 14.4 until the outage occupies less than 14.4 / 1,000 of the window:

alert lingers, minutes:  60 x (1 - 14.4 / 1,000)                 =  59.1

Almost an hour of firing after the incident is over — which is how on-call engineers learn to ignore the alert.

Requiring the short window to also be above the threshold clears it within 5 minutes, because a 5-minute window forgets a finished outage in 5 minutes. The long window controls precision; the short window controls reset time. Neither one alone gives you both.

The diagram below is the full rule set. Follow one error ratio into four branches: the two AND-gated pairs that page, and the two single-window rules that only file a ticket.

flowchart LR
    E["error ratio<br/>rate(errors) / rate(total)"] --> L1["1 h burn > 14.4"]
    E --> S1["5 m burn > 14.4"]
    L1 --> A1{"AND"} -->|"2% budget · 51.8 s on a full outage"| PAGE(["page"])
    S1 --> A1
    E --> L2["6 h burn > 6 AND 30 m burn > 6"] -->|"5% budget · slower burns"| PAGE
    E --> L3["24 h burn > 3"] --> TIC(["ticket"])
    E --> L4["72 h burn > 1"] --> TIC

    style PAGE fill:#9d0208,color:#fff

Same ch 01 key. Red marks the step you cannot undo, and in this diagram that is the page itself: the AND gate is a condition you can retune tomorrow, but a human woken at 3 a.m. stays woken.

8. The write path: fan-in, and why the door says 429

Sizing for the recovery burst, not the steady state

The write path is a fan-in: thousands of independent senders, one logical sink. The defining property of a fan-in is that the arrival rate is set by the senders, and no amount of provisioning changes who controls it.

Steady state is undramatic — 25,000 samples/s, one machine’s work. Recovery is the case that sizes the tier.

During a network partition every sender keeps scraping and buffers to its local write-ahead log. When connectivity returns, all of them replay their backlogs simultaneously. That is a thundering herd: many clients acting at once because they were all released by the same event.

Take a 10-minute partition, and a sender configured to drain its backlog within 30 seconds:

10-minute partition backlog:  25,000 x 600                       =  15,000,000
senders drain a backlog within, seconds:                            30
replay rate:          15,000,000 / 30                            =  500,000
replay burst:         500,000 / 25,000                           =  20

A 10-minute network blip produces a 20x burst on recovery, and that burst — not the 25,000/s steady state — is what the ingest tier is sized against.

The 20x is not a folk constant. It is 600 s of backlog / 30 s of catch-up deadline. State that assumption out loud in the interview, because halving the deadline to 15 s doubles the herd to 40x.

Why queueing the burst collapses the system

Now suppose the tier is provisioned for 100,000 samples/s — four times steady state, but a fifth of the burst — and it responds to the excess by queueing rather than rejecting:

queue growth/s:       500,000 - 100,000                          =  400,000
depth after 10 s:     400,000 x 10                               =  4,000,000
wait at the head:     4,000,000 / 100,000                        =  40
sender timeout, seconds:                                            30

The wait at the head line is Little’s law in its simplest form: a queue 4,000,000 items deep, drained at 100,000 items/s, makes the item at the head wait 40 seconds.

The queue’s own latency, 40 s, exceeds the sender’s 30 s timeout.

Follow what that does. The sender gives up waiting, marks the batch un-acked, and retries it. So the same samples arrive twice, and queueing has increased the arrival rate. Arrivals go up, the queue grows faster, the head wait gets longer, more senders time out, more retries arrive.

That loop is congestion collapse: throughput falls as offered load rises. It is caused by the buffer, not relieved by it.

Reject at the door instead

An immediate HTTP 429 Too Many Requests does three things a queue cannot:

• costs the ingest tier one cheap rejection instead of a buffered sample plus a doomed write;
• pushes the backlog into the sender’s bounded, disk-backed WAL, which is designed for exactly this and holds hours (Why a log beats a queue 328 devices or one is why that WAL is cheap);
• gives the sender an explicit signal to back off exponentially, so the herd de-synchronizes instead of retrying in lockstep.

“Queue it” moves an overload from a place with backpressure to a place without one. Rejecting keeps the overload where the buffer already exists.

Shed selectively, and in a specific order

Shedding first-come-first-served means the loudest tenant wins, because it simply sends more. Give each tenant a limit instead — fair share, doubled, so a tenant with genuinely more series than average is not punished for it:

tenants:              20
fair share:           750,000 / 20                               =  37,500
per-tenant limit at 2x:  37,500 x 2                              =  75,000

Within a tenant, the order of rejection matters too: reject samples for new series before samples for existing ones.

A series created ten seconds ago is on no dashboard and in no alert rule, by construction — nobody has had time to reference it. Dropping it costs nothing today. An existing series is something a rule is evaluating right now, and dropping it makes an alert go blind.

That one ordering rule is what makes a cardinality explosion (Cardinality is the whole problem) degrade only the tenant causing it, while every existing alert keeps evaluating right through the incident.

Finally, the 429 body must name the tenant, the metric, and the label that breached. The difference between a four-minute fix and a four-hour one is whether the rejection said which label exploded.

3. Bottlenecks and scaling

Six things bind before anything else does, and each one has a named fix.

The scaling axis to note: this system scales on series count, not on sample rate. Doubling the scrape frequency doubles samples and costs 2 x 1.4 bytes per series per interval — nothing. Doubling the label cardinality doubles memory, index size, query cost, and box count simultaneously. When someone asks “can we scrape every 10 seconds instead of 30,” the answer is usually yes and it is cheap. When they ask “can we add a label,” the answer requires arithmetic.

4. Failure modes

The first two rows are the ones that end careers: the ingester dying from cardinality, and the monitoring system being down at the exact moment you need it. The rest are ordinary, but each has a mitigation worth naming out loud.

Volunteer the dead-man’s switch. Every alert in this system fires on the presence of bad data. A pipeline that stops delivering data fires nothing at all, which looks exactly like health. One rule that always fires, routed to a receiver that pages when it stops arriving, is the only thing that covers that case.

5. Alternatives rejected

Every row is a design an interviewer might propose, the mechanism that sinks it, and the figure to quote when they do.

6. Interviewer pushback

Six common questions, each answered with arithmetic rather than a principle.

“Why do you keep talking about cardinality instead of throughput?”

Because throughput here is 25,000 samples/s, which is one machine’s work, and cardinality is what actually decides the fleet. 750,000 active series at 2 KB each is 1.5 GB and fits in 3% of a box. Add user_id with 200,000 values to a single counter and the series count goes to 50 billion — 2,134 boxes of capacity against one, so about $18.7 M a year against $8,760. I would say explicitly that both of those are bare capacity: if I quoted the exploded fleet against the two boxes I actually run for availability I would report 1,067x, and I would be halving my own answer by putting redundancy on one side of the ratio only. The bytes never changed much; the number of things I have to keep resident changed by 66,667x. Throughput is a factor I can buy my way out of. Cardinality is a factor of ten thousand that no amount of hardware fixes.

“A team needs per-customer error rates. What do you tell them?”

Not “no,” because that answer gets routed around. Three options in order. First, bound the label: top-50 customers by name and everything else as other, which is 51 values instead of 200,000 and answers the question they actually have. Second, if they need arbitrary customer lookup, that is a traces or logs query, not a metrics query — those engines index high-cardinality fields on purpose and charge per event instead of per series. Third, if they truly need per-customer time series, that is a separate tenant with its own limit and its own budget line, so the cost lands on the team that asked for it. What I would not do is let it into the shared metrics tenant, because the failure is not gradual — the ingester OOMs and everyone’s alerting stops.

“Your alert threshold looks reasonable. Why does it page 57,600 times a day?”

Because it is per-series and the fleet is large. A 1-in-1,000 excursion is fine for one series; across 10,000 series evaluated every 15 seconds that is 667 evaluations a second, so 0.667 false trips a second, so 57,600 a day. Adding for: 5m helps by about 12x under a realistic autocorrelated noise model, not by the 0.001 ^ 20 that independence would predict — which is the trap. The fix is not a better threshold, it is a different measurement: aggregate 500 series into one SLO signal. That is 500x fewer evaluations and the aggregate’s noise is 22.4x smaller, so the same threshold sits 69.4 standard deviations out instead of 3.1. Then I page on burn rate against the error budget, not on the metric.

“Where does 14.4 come from? It looks like a magic number.”

It is 0.02 x 720. The SLO period is 30 days, which is 720 hours, and I have decided I want to be paged after losing 2% of the error budget. Sustaining burn rate B for W hours consumes B x W / 720 of the budget, so for a 1-hour window, 14.4. The rest of the table is the same equation: 5% over 6 hours is 6, 10% over 24 hours is 3, 10% over 72 hours is 1. And the consequence I would check is detection time — a total outage burns at 1 / 0.001 = 1,000, so the 1-hour rule trips after 1 x 14.4 / 1,000 hours, 51.8 seconds. That number is where my sub-minute detection requirement came from; I did not pick it separately.

“Downsampling saves storage, right?”

No, and the arithmetic is worth showing. A 5-minute point has to carry min, max, sum, and count or you cannot compute a max or a rate from it, so the rate drops only 2.5x — 25,000 samples/s to 10,000 — while the retention stretches from 15 days to 90. The 5-minute tier ends up costing 2.4x the raw tier. What downsampling buys is query cost: a 30-day panel over 100 series is 8.6 million raw samples rendered into 1,000 pixels, 8,640 samples per pixel, and the 5-minute tier cuts that 10x with no visible difference. I keep the tiers, I just do not claim they are a storage optimization.

“The ingest tier is overloaded. Why not add a queue?”

Because the queue makes it worse, and I can show it. If arrivals are 500,000/s against 100,000/s of service, the queue grows 400,000/s and after ten seconds a sample entering it waits 40 seconds. The sender’s remote-write timeout is 30, so it gives up and retries — arrivals go up, the queue grows faster, more senders time out. Congestion collapse, caused by the buffer. A 429 at the door costs one cheap rejection and pushes the backlog into the sender’s own disk-backed WAL, which is bounded, holds hours, and already implements exponential backoff. And I shed selectively: new series before existing ones, because a new series is on no dashboard and in no alert rule yet, while an existing one is being evaluated right now.

Cheat sheet

Every number in this table is derived somewhere above. If you cannot reconstruct a row’s arithmetic, that is the section to re-read.

Related: 02 — Back-Of-The-Envelope is the estimation drill these numbers follow; sql 03 is the LSM write amplification that makes a general store lose by two to three orders of magnitude; 20 — Distributed Message Queue is the same shed-versus-queue argument on the other side of the pipe; 01 — Scale From Zero To Millions prices the logging half of observability.