10 — Design A Notification System

A notification system is the service every other service calls when it needs to reach a person: an order ships, a login code is issued, someone likes a post. Each of those has to become a message on a device.

Two facts drive the whole design:

1. Guaranteed once-and-only-once delivery is not achievable here.
2. The three delivery channels differ so much in unit cost that choosing between them is a financial decision before it is a technical one.

Everything else — the queue layout, the consent gate, the retry rules, the tracking pipeline — follows from those two facts.

What goes in, and what comes out. The input is a request from an internal service, in the shape (user, category, template, params) — for instance “user 8891, category order_shipped, template ship_v3, with the tracking number filled in.”

The output is a message on a device: a push notification, an SMS text, or an email. It is in the user’s language and subject to that user’s consent. Alongside it goes a trail of events recording what the system attempted and what came back.

Everything in the middle is fan-out, filtering, and queueing.

Two terms up front, because everything below uses them:

• Fan-out is the step that turns one logical request into many concrete deliveries. One “your team posted” becomes 10 million individual pushes. One notification becomes an attempt on each of a user’s four devices.
• A channel is one delivery road: push, SMS, or email.

The two sentences that decide the round

The architecture is a queue and some workers, which is quick to draw. What matters more is being precise about two things: delivery semantics and cost.

Exactly-once delivery is not available here.

Exactly-once delivery means every notification arrives, and arrives exactly once — never zero times, never twice.

Here is why you cannot have it. Every hop that matters crosses into a third party you do not own:

• APNs (Apple Push Notification service) and FCM (Firebase Cloud Messaging) deliver to phones.
• Twilio or a similar aggregator delivers SMS.
• An SMTP relay delivers email. SMTP is Simple Mail Transfer Protocol, the protocol mail servers speak to each other.

No transaction spans your database and theirs. A transaction is an all-or-nothing bundle of operations: either every step lands or none does. Without one, you cannot make “the message went out” and “I recorded that it went out” succeed or fail together — and that gap is the whole problem, worked through in Deep dive 3 exactly once is not available.

What you build instead is at-least-once delivery: keep trying until the other side acknowledges, and accept that some messages arrive twice. You then add a dedup key — an identifier carried inside the payload so the receiver can recognize a repeat and discard it. That is the honest design; claiming exactly-once is not.

SMS is about 5% of volume and 96% of the bill.

The three channels differ by four orders of magnitude in unit cost, so routing between them is a financial decision. 3a what each channel costs does that arithmetic from scratch.

Background this chapter borrows

Three other chapters supply ideas this one uses rather than re-derives. None is a prerequisite — each idea is restated here in the sentence that needs it.

• Estimation discipline: chapter 02.
• Retries, backoff, and jitter: chapter 04.
• The queue’s concurrency arithmetic: chapter 01.

1. Framing: what decision, and what breaks

A notification system is a fan-out engine whose output crosses an ownership boundary you do not control. That single structural property generates every hard problem in the chapter — all four difficulties, and everything that breaks in production as a consequence.

Two terms in the table below need defining first. P&L is profit and loss, the accounting statement a business is run against; “a routing policy with a P&L attached” means the routing rule shows up directly in the company’s costs. An opt-out is a user switching a channel or category off — replying STOP to an SMS, clicking unsubscribe, revoking notification permission. It is irreversible because the channel you would use to ask them back is the one they just closed.

Cause is on the left, forced consequence on the right — and every right-hand cell is a constraint you did not choose and cannot design away.

What actually breaks in production

Four failures, ordered from most frequent to least. Each has a section below.

• The duplicate storm. A fan-out job re-runs after a deploy and 4 million people get the same push twice (Deep dive 3 exactly once is not available).
• The urgent message stuck behind the trivial one. A marketing campaign and a transactional OTP share a queue, and the OTP arrives after it has expired. An OTP is a one-time passcode — the short login code that is valid for about a minute (Deep dive 2 fanout and what the queue is actually for).
• The metric that measures nothing. The “delivered” dashboard is quietly reporting a number that is not delivery (Tracking and why delivered is a lie).
• The one-character bill increase. Somebody ships an emoji into an SMS template and the bill goes up by $37,500 a day (Templates localization and the emoji that costs 137 m).

2. Requirements

First, what the system must do, what it will not do, and — in 2a the assumptions this design rests on — the assumptions every number in the chapter stands on.

Functional

• Accept a notification request from any internal service: (user, category, template, params).
• Deliver over push (iOS/Android), SMS, and email, with a fallback ladder between them.
• Per-user preferences: channel opt-in, category opt-in, quiet hours, locale.
• Templates with localization and versioning.
• Scheduled and immediate sends; a bulk send to a segment.
• Track what happened to each notification, and expose it.

Three of those bullets use terms that need glossing.

• A fallback ladder is an ordered escalation: try the cheapest channel that can reach the user, and move to a costlier one only on evidence that the first did not work.
• Quiet hours are a per-user window, in the user’s own timezone, during which nothing non-urgent may be sent.
• An audience segment is a saved set of users a campaign targets, such as “everyone who abandoned a cart this week.” Say “audience segment”, not “segment” — Templates localization and the emoji that costs 137 m uses the same word for a completely different thing.

Out of scope, said explicitly: rendering the in-app inbox, computing who is in a segment, and the campaign authoring interface.

Non-functional — the ones that settle the architecture

The table below is not a wish list. Each row is a number that forces a structural choice later, and the third column names where. The two rows to read hardest are the two latency rows, because their targets differ by a factor of about 1,000.

Two rows use shorthand worth unpacking.

p99 under 30 s is the 99th percentile of end-to-end latency. Sort every notification by how long it took; the one 99% of the way up that sorted list took under 30 seconds. Stating a percentile rather than an average is how you make a promise about the slow tail instead of about the typical case, where the tail is what users notice.

Transactional versus marketing is the category split that runs through the whole chapter. Transactional traffic is something the user’s own action asked for — a login code, a receipt. Marketing traffic is something you decided to send them.

The latency split is the requirement that does the most architectural work. Two classes with a 1,000x difference in tolerance cannot share a queue, because the marketing backlog will be sitting in front of the OTP. Deep dive 2 fanout and what the queue is actually for turns that into a queue layout.

2a. The assumptions this design rests on

Every number below is downstream of a short list of assumptions. Say them out loud in the interview. An interviewer who changes a load-bearing one changes the design; an interviewer who changes the rest only changes the bill.

The last column is the one to read first. “Yes” means the assumption picks the architecture, so if the interviewer overturns it, you must redraw something. “No” means it only scales a number.

The five to say out loud are B1 through B5. Each one picks a different part of the design:

• B1 makes exactly-once impossible.
• B2 and B3 make cost the routing criterion.
• B4 splits the queues.
• B5 makes over-sending the most expensive failure in the system.

B6 is the one to flag as an estimate you would measure rather than assume. A basis point is one hundredth of a percent, so the 5 basis points in B6 is 0.0005 — five extra opt-outs per ten thousand users per extra notification.

The two assumptions carrying the biggest numbers

B11 and B12 are the two to name before you use them. Between them they carry the chapter’s two largest dollar figures, and neither is measured anywhere in this design.

The $13.7 M deflection saving (Deep dive 1 cost drives routing) is 0.20 multiplied by the SMS bill and nothing else. So it is exactly linear in B11: a ladder that deflects 5% instead of 20% is worth $3.4 M, and the whole routing argument shrinks by a factor of four.

The $73 M of fatigue (Rate limiting fatigue and the opt out path) is the product 0.40 x 2 x 0.0005 x $5 — four soft numbers multiplied in a row. By the sensitivity rule of ch 02, the uncertainty in a product belongs to whichever factor has the widest plausible range. Here that is B6, the opt-out probability, not B12.

So make the claim that survives both: the ordering holds and the magnitudes do not. Fatigue beats the SMS optimization across essentially every plausible setting of the four factors. That ordering, rather than either dollar figure, is what changes the design.

3. Back of the envelope

One product number — 100 million daily active users — yields the request rate, the channel mix, the bill, and the size of the tracking pipeline. The bill is the part that decides the architecture, so it gets its own subsection.

One convention first, because every division below uses it. Rounding discipline is chapter 02, and the substitution it licenses is 86,400 -> 1e5. A day holds 86,400 seconds; calling it 100,000 turns every “per day to per second” division into a decimal shift. The cost is 16% (100,000 / 86,400 = 1.16), which is well inside the error bars on everything else here. One place in the chapter that shortcut is not safe enough, and Rate limiting fatigue and the opt out path redoes the division in exact seconds when it matters.

DAU below is daily active users: the count of distinct people who use the product on a given day.

The first block turns the product number into the send rate.

DAU                                          100,000,000
notifications per user per day                         5
notifications per day
  100,000,000 x 5                        =  500,000,000
notifications per second
  500,000,000 / 100,000                  =  5,000
peak, at 3x
  5,000 x 3                              =  15,000

Two numbers to carry forward: 5,000 sends per second average, 15,000 at peak. Everything in the chapter is sized against one or the other.

Next, split that daily volume across the three channels using the 80/15/5 mix from assumption B3. These three lines are what makes the cost section possible.

push, at 80% of volume
  500,000,000 x 0.80                     =  400,000,000
email, at 15%
  500,000,000 x 0.15                     =  75,000,000
SMS, at 5%
  500,000,000 x 0.05                     =  25,000,000

3a. What each channel costs

The whole design turns on this arithmetic: price one million messages on each channel, and the spread is four orders of magnitude.

Push has no provider fee at all — APNs and FCM charge nothing — so its cost is entirely the fleet of servers you run to talk to them. That makes push the only channel whose unit cost you have to derive rather than look up.

HTTP/2 below is the version of HTTP that multiplexes many requests over one connection, which is why a single box can hold thousands of concurrent pushes in flight. The block works from the peak push rate down to a dollar figure per million messages; the two lines that carry judgement rather than arithmetic are the 2,000 per box and the x5.

peak push rate
  15,000 x 0.80                          =  12,000
pushes per second one sender box sustains over HTTP/2   2,000
sender boxes at peak
  12,000 / 2,000                         =  6
x5 for the token store, retry queue, and receipt pipeline
  6 x 5                                  =  30
box cost per hour, dollars                          0.20
fleet cost per day
  30 x 0.20 x 24                         =  144
cost per million pushes
  144 / 400                              =  0.36

The 400 in the last line is 400 million daily pushes expressed in millions, so 144 / 400 is dollars per day divided by millions of pushes per day: $0.36 per million pushes.

What the x5 means, and what it does not

The x5 is the multiplier for everything around the senders themselves — the store of device tokens, the retry queue, and the pipeline that ingests receipts. Multiplying the boxes doing the visible work by a small constant is the standard way to get from “the boxes that send” to “the boxes you actually pay for.”

Write down what it does and does not mean, because it gets misread two sections later: 6 of those 30 boxes send pushes, and the other 24 do not.

So there are two different numbers hiding in one block:

• The fleet’s push capacity is 6 x 2,000 = 12,000/s. That stays 12,000/s however large the support tier grows.
• The 30 is a cost number. It is never a throughput number.

Deep dive 2 fanout and what the queue is actually for is where dividing by the wrong one of those two changes an answer by 5x.

The headroom this fleet does not have

While the number is on the page, say the uncomfortable thing about it. The 6 boxes were solved for exactly the 12,000/s peak. So capacity equals offered load, utilization is 1.00, and the queue never drains at the peak the fleet was designed for. The push fleet has zero headroom by construction.

That is the failure 3a volume bandwidth storage and ch 01 both warn about, and the fix is the same in all three chapters. Size in this order: offered load, then a chosen utilization, then the capacity that implies, then the box count.

Run it: offered load is 12,000/s, choose 80% utilization, so you need 12,000 / 0.8 = 15,000/s of capacity, which at 2,000/s per box is 15,000 / 2,000 = 7.5, rounded up to 8 sender boxes, not 6.

Everything below keeps the 6-box fleet and its 12,000/s, because that is the fleet this chapter costs and fans out against. But a tier running at 1.00 utilization is a choice nobody made deliberately, and the 8-box correction is the first thing to offer when an interviewer asks where the headroom is.

Email and SMS: the numbers you look up

Unlike push, email and SMS carry a per-message fee from the provider. Both are quoted here per million messages so all three channels can be compared on one scale. Watch the units in each line: the email price is quoted per thousand and the SMS price per message, so the two conversions to “per million” differ.

email at $0.10 per 1,000, per million
  0.10 x 1,000                           =  100
SMS at $0.0075 per message, per million
  0.0075 x 1,000,000                     =  7,500

Now multiply each channel’s per-million price by its daily volume in millions. The / 1,000,000 on each line converts a message count into millions-of-messages, which is the unit the price is in.

push, per day
  400,000,000 / 1,000,000 x 0.36         =  144
email, per day
  75,000,000 / 1,000,000 x 100           =  7,500
SMS, per day
  25,000,000 / 1,000,000 x 7,500         =  187,500
total, per day
  144 + 7,500 + 187,500                  =  195,144
per year
  195,144 x 365                          =  71,227,560

$195,144 a day, $71.2 M a year, and one channel dominates it. The next block compares SMS’s share of the messages against its share of the money, then prices each channel against the others.

SMS share of the volume
  25,000,000 / 500,000,000               =  0.05
SMS share of the bill
  187,500 / 195,144                      =  0.961
SMS against push, per message
  7,500 / 0.36                           =  20,833
SMS against email, per message
  7,500 / 100                            =  75
email against push, per message
  100 / 0.36                             =  278

Five percent of the messages are 96% of the bill.

State the ratios carefully, because there is a trap in them. Count push as literally free and the ratio is infinite and useless. Count only the provider fee and push and email look comparable, since both are zero at the provider.

The honest comparison prices push loaded with the fleet that sends it — the fleet cost divided over the messages it carries. On that basis push is $0.36 per million against SMS at $7,500, which is 20,833x, roughly two orders of magnitude per tier as you climb push -> email -> SMS.

The number that changes a design is none of those three ratios. It is the 96%.

3b. Storage and the tracking load

The system keeps two data sets — the device tokens it sends to, and the event trail it writes about every send — and the second turns out to be a system in its own right.

First the token store. A device token is the opaque string APNs or FCM gives an app installation, and it is the only address a push can be sent to. Assume two devices per user.

registered devices
  100,000,000 x 2                        =  200,000,000
bytes per token row: token, platform, locale, tz, timestamps    200
token store, in GB
  200,000,000 x 200 / 1,000,000,000      =  40

40 GB. That fits in the memory of a single box, so device lookup is never the bottleneck in this system and nobody should spend interview time on it.

Now the event trail, which is a different story. Every notification produces three events on its way through: queued when it enters a queue, sent when a provider accepts it, and client receipt when the device says it rendered it.

events per notification: queued, sent, client receipt              3
event rows per day
  500,000,000 x 3                        =  1,500,000,000
bytes per row
  50
bytes per day
  1,500,000,000 x 50                     =  75,000,000,000
one year, in TB
  75,000,000,000 x 365 / 1,000,000,000,000  =  27.4

Convert those 1.5 billion daily rows into a write rate, then compare it against the send rate the whole system was sized for.

event writes per second, all three events per notification
  1,500,000,000 / 100,000                =  15,000
against the outbound send rate of 5,000/s
  15,000 / 5,000                         =  3

The tracking pipeline writes at 3x the rate of the thing it tracks. Three events per notification means 5,000 sends a second generate 15,000 event writes a second, and 27.4 TB a year of rows.

Telemetry here is not a side-channel hanging off the send path. It is a larger system than the send path, and it must not share a database with it (Alternatives rejected).

One substream, and the ratio not to quote

The client-receipt events are worth sizing on their own, because Tracking and why delivered is a lie builds the delivery metric out of them. Only about 80% of pushes produce a receipt (assumption B10).

push receipts per day, at 80% acknowledged
  400,000,000 x 0.80                     =  320,000,000
receipt writes per second, the client-receipt stream alone
  320,000,000 / 100,000                  =  3,200
that stream against ALL-CHANNEL sends of 5,000/s
  3,200 / 5,000                          =  0.64

Keep those two ratios apart. They are computed over different populations, and only one of them is the headline.

The 0.64 divides push client receipts (3,200/s, a subset of one channel’s events) by all-channel sends (5,000/s). Numerator and denominator do not describe the same thing, which is why it understates the true tracking load by a factor of nearly five. Quote the 3x.

Do not pair the 3,200/s with the 27.4 TB/year either, because they are two different streams. The receipt substream is 320 M rows a day, which is 320,000,000 x 50 x 365 = 5.8 TB/year. The 27.4 TB belongs to all 1.5 billion daily rows, not to receipts alone.

4. API sketch

The interface internal services call carries four small choices that are the whole design in miniature: what the response promises, who supplies the duplicate key, why every message must carry an expiry, and why a category is not the same thing as a template.

The sketch below is one endpoint in full plus four in outline — the first five lines are the request body, the three after them the responses it can return.

POST /v1/notifications
  {"user_id": ..., "category": "order_shipped", "template_id": "ship_v3",
   "params": {...}, "channels": ["push","email"], "ttl_s": 86400,
   "idempotency_key": "order-8891-shipped"}
  202 {"notification_id": "..."}      <- accepted, not delivered
  429 over the caller's quota
  422 unknown template, or params fail the template's schema

POST /v1/notifications/bulk        {"segment_id", ...}  -> job handle
GET  /v1/notifications/{id}        -> per-channel attempt history
GET/PUT /v1/users/{id}/preferences -> channels, categories, quiet hours, locale
POST /v1/receipts                  {"notification_id","event":"rendered"}

The three-digit numbers are HTTP status codes: 202 means “accepted for later processing,” 429 means “you are over your quota,” and 422 means “the request was understood and is invalid.”

Four choices in that sketch are deliberate, and each is worth defending out loud.

202, never 200. 200 means “done.” 202 means “durably queued,” which is the strongest true statement available at that moment, because nothing has been handed to a provider yet. Returning a status that implies delivery is the API-level version of the exactly-once lie, and it will be quoted back at you in an incident review.

idempotency_key is supplied by the caller, and is scoped to that caller. An idempotency key is a caller-chosen identifier for one logical intent, such as order-8891-shipped. The server records it, and on seeing it again returns the original outcome instead of acting twice. That makes the operation idempotent: repeating it has the same effect as doing it once.

This is the only class of duplicate you can actually eliminate — the same upstream event submitted twice. It does nothing about duplicates the provider creates (Deep dive 3 exactly once is not available), and you should say so before you are asked.

ttl_s is required, not optional. ttl_s is a time to live in seconds: how long this notification remains worth delivering. A notification is a perishable good. “Your ride is outside” delivered 40 minutes later is worse than nothing, so the consumer must be able to drop it (Deep dive 2 fanout and what the queue is actually for).

category, not just template_id. Rate limits, quiet hours, and opt-outs all key on category. A system whose only unit is a template cannot express “never suppress an OTP.”

5. Data model

Five tables hold everything the system knows, and two decisions inside them are the ones a reviewer should push on: the shape of the identifier, and the shape of the preferences row.

Some vocabulary before the schema, because three terms in it carry decisions.

The outbox is the table of work in progress — one row per delivery attempt the system still owes. The name is the standard one for “durable record of something that must go out.”

BIGINT, TEXT, JSONB and the rest are SQL column types: a large whole number, text, and a JSON document stored in a queryable binary form.

To shard is to split a table across several machines that each hold a disjoint slice. The shard key is the column whose value decides which machine a row lands on (ch 05).

That shard key choice appears on the very first line and is worth stating as a decision. device_tokens shards by user_id, not by device_id, so that all of one user’s devices sit on one machine. Fan-out reads every device a user owns, so with a user_id key that is one query on one hop. With a device_id key it becomes a scatter-gather across the whole fleet — on the hottest read in the system.

Five tables follow. The comment at the end of each header line gives its size or its shape; those are the numbers from 3b storage and the tracking load.

device_tokens                             -- 40 GB, shard by user_id
  user_id BIGINT, device_id TEXT, platform SMALLINT, token TEXT
  locale TEXT, tz TEXT, app_version TEXT
  last_seen_at TIMESTAMP, invalid_at TIMESTAMP    -- reaping, section 13
  PRIMARY KEY (user_id, device_id)

preferences
  user_id BIGINT, channel SMALLINT, category TEXT
  allowed BOOL, quiet_start SMALLINT, quiet_end SMALLINT
  PRIMARY KEY (user_id, channel, category)

templates
  template_id TEXT, version INT, locale TEXT, channel SMALLINT
  body TEXT, params_schema JSONB
  PRIMARY KEY (template_id, version, locale, channel)

outbox                                    -- one row per (notification, channel)
  notification_id BIGINT, channel SMALLINT, user_id BIGINT
  template_id TEXT, params JSONB, ttl_at TIMESTAMP
  state SMALLINT, attempts SMALLINT, last_provider_status TEXT
  PRIMARY KEY (notification_id, channel)

events                                    -- append-only, columnar, 27.4 TB/yr
  notification_id, channel, event, at, provider_status

Three decisions in that schema are worth defending, because a reviewer will push on all three.

notification_id is a Snowflake ID (ch 07), not a UUID. A Snowflake ID packs a timestamp, a machine number, and a counter into 64 bits — 8 bytes — and therefore sorts by time. A UUID, a universally unique identifier, is 16 random-ish bytes that sort by nothing.

The reason the size matters: this id is also the client’s dedup key, so it travels inside every single payload, and the whole push payload is capped at 4 KB. Eight bytes beats sixteen, and Deep dive 3 exactly once is not available spends that saving.

preferences is keyed on (user, channel, category) rather than carrying one JSON blob per user. The opt-out check is a point lookup — fetch exactly one row by its full primary key — and it runs on the hottest path in the system, immediately before every provider call. A JSON blob would force the worker to fetch, parse, and re-serialize a document to answer a yes/no question.

The outbox row is per (notification, channel), not per notification. A fallback ladder produces several attempts across different channels for one logical notification, and each of those attempts needs its own state, TTL, and attempt count.

6. High-level architecture

With the tables defined, follow one notification through the whole pipeline, so the three claims the picture makes have somewhere to land.

The diagram reads top to bottom: request in at the top, providers at the bottom, telemetry down the right. The two things to look at are the three parallel queue boxes in the middle, and the red diamond below them, which is where consent is checked.

flowchart TD
    SVC["Calling services<br/>orders, social, auth"] --> API["Notification API<br/>validate, idempotency key"]
    API --> FAN["Fanout service<br/>resolve users and devices"]
    FAN --> PREF[("Preferences<br/>+ device tokens")]
    FAN --> QT[["Transactional queue"]]
    FAN --> QS[["Social queue"]]
    FAN --> QM[["Marketing queue"]]
    QT & QS & QM --> W["Channel workers"]
    W --> GATE{"Opt-out, quiet hours,<br/>per-user budget<br/>checked HERE"}
    GATE -->|"suppressed"| DROP["Drop and record"]
    GATE -->|"allowed"| TPL["Render template<br/>locale, ICU plurals"]
    TPL --> APNS["APNs / FCM"] & SMS["SMS aggregator"] & MAIL["SMTP / ESP"]
    APNS & SMS & MAIL --> EV[["Event stream"]]
    CLIENT["Device"] -->|"rendered receipt"| EV
    EV --> ANA[("Analytics store<br/>columnar")]

    style GATE fill:#9d0208,color:#fff
    style QT fill:#2d6a4f,color:#fff
    style QS fill:#1d3557,color:#fff
    style QM fill:#bc6c25,color:#fff

Following one notification through

Into the API. The calling services — orders, social, auth — post to the notification API. It validates the request and records the idempotency key.

Into a queue. The fanout service expands that one request into concrete deliveries by resolving the user’s devices and preferences. It drops each delivery into the transactional queue, the social queue, or the marketing queue, according to how urgent the category is. Channel workers drain those queues.

Through the gate. Each worker’s first act is the gate: opt-out, quiet hours, and per-user budget, checked here and nowhere earlier. A notification that fails is marked suppressed, then dropped and recorded. The record matters, because “we chose not to send this” and “we failed to send this” must never look the same on a dashboard.

Rendered. A notification that passes the gate is rendered from its template in the user’s locale, with correct plural forms — the ICU plurals the diagram names, unpacked in Templates localization and the emoji that costs 137 m.

Handed to a provider. APNs or FCM for push, an SMS aggregator for text messages, SMTP or an ESP for email. An ESP is an email service provider, such as SendGrid, that operates mail infrastructure on your behalf.

Recorded. Every provider response, plus every rendered receipt the device itself posts back, lands on the event stream and settles in an analytics store. That store is columnar: it keeps each field’s values together rather than each row’s values together, which is what makes counting over billions of rows cheap.

The colour key, which changes between diagrams

The key does not mean what chapter 01’s key means, and it does not mean the same thing in both of this chapter’s own diagrams.

In the diagram above:

• Red is the gate that must not be stale — the one irreversible mistake in the system.
• Green, blue and orange are the three queues. They are coloured only so that three sibling boxes read as three distinct partitions: transactional, social, marketing, one per latency class. All three are filled deliberately, because an uncoloured sibling would read as “this one is different,” and it is not.

Those fills carry no ordering and no cost claim.

Now the amendment, which applies to the ladder diagram in Deep dive 1 cost drives routing: there, green and red mark unit cost — cheapest and dearest — not latency class. Green there is push at $0.36 per million; red is SMS at $7,500. Neither has anything to do with the green transactional queue or the red consent gate here. Two questions, one palette.

The three claims the picture makes

Each gets a section of its own below.

1. Queues are split by latency class, not by channel (Deep dive 2 fanout and what the queue is actually for).
2. The opt-out and budget gate sits in the worker, immediately before the provider call, not in the fanout service (Rate limiting fatigue and the opt out path). A bulk fan-out computed twenty minutes ago has a stale view of consent, and consent is the one thing you may not be stale about.
3. The client feeds the event stream, because the providers will not tell you the truth about delivery (Tracking and why delivered is a lie).

7. Deep dive 1: cost drives routing

The fallback ladder is the design the cost table of 3a what each channel costs forces. Try the cheapest channel that can reach the user, and escalate to a costlier one only on evidence that the cheap one did not work.

The diagram reads left to right. Diamonds are decisions, boxes are channels, and each channel box carries its price per million from 3a what each channel costs so you can see what each branch costs.

flowchart LR
    N["Notification"] --> P{"Live push token<br/>seen in 30 days?"}
    P -->|"yes"| PUSH["Push<br/>$0.36 / M"]
    PUSH --> R{"Client receipt<br/>within 5 min?"}
    R -->|"yes"| DONE["Done"]
    R -->|"no"| E{"Category allows<br/>escalation?"}
    P -->|"no"| E
    E -->|"yes, and email opted in"| MAIL["Email<br/>$100 / M"]
    E -->|"transactional only"| SMSN["SMS<br/>$7,500 / M"]
    E -->|"no"| DROP["Stop"]

    style PUSH fill:#2d6a4f,color:#fff
    style SMSN fill:#9d0208,color:#fff

Green and red mean something different in this diagram than in High level architecture’s. Here they mark unit cost: green is the channel that is nearly free, red is the one that is 20,833x dearer. They do not mark latency class or irreversibility. Nothing about the green box here relates to the green transactional queue, and nothing about the red box here relates to the red consent gate.

In words: if there is a live push token seen in the last 30 days, push, because it is nearly free. If a client receipt comes back within five minutes, stop — the message landed. Otherwise ask whether the category allows escalation. If it does, go to email when the user has email opted in, or to SMS for transactional categories only, never for anything else.

What the ladder saves

Deflection means a message that would have gone by SMS being satisfied by a cheaper channel instead. Assumption B11 puts that at 20% of SMS volume. Price it:

SMS deflected to push by the ladder, at 20% (B11)
  25,000,000 x 0.20                      =  5,000,000
saving per day
  5,000,000 / 1,000,000 x 7,500          =  37,500
saving per year
  37,500 x 365                           =  13,687,500

A 20% deflection rate is worth $13.7 M a year. That is why the “seen in 30 days” freshness check on the token and the five-minute receipt window are business logic, not plumbing.

Say where the 20% came from in the same breath as the $13.7 M. It is assumption B11 in 2a the assumptions this design rests on, it is measured nowhere in this design, and the saving is exactly linear in it. The honest form of the figure is $684,000 a year per percentage point of deflection (13,687,500 / 20 = 684,375), which also tells you what to instrument first: the deflection rate itself.

Why the ladder is ordered that way

Six properties differ across the three channels, and the table below is the whole argument for the order. Read the first row for the cost that drives everything, and the fourth and fifth rows for the reason a cheap channel cannot simply be trusted.

Three rows in that table use terms that get their own treatment later.

• An MTA is a mail transfer agent, the server that accepts and forwards email. “MTA acceptance” means the receiving mail server took the message — which is entirely compatible with filing it straight into spam.
• A DLR is a delivery receipt returned by a mobile carrier. Tracking and why delivered is a lie shows why it often means nothing.
• GSM-7 is the 7-bit alphabet SMS was designed around. The 160-versus-70 character split it creates is a real line item in the bill, derived in Templates localization and the emoji that costs 137 m.

Push is 20,833x cheaper and strictly less reliable. That is the whole tension, and the escalation rule falls out of it.

The rule: escalate on category, never on channel preference alone. An OTP escalates to SMS, because a failed login costs the business far more than three quarters of a cent. A “someone liked your post” never escalates, at any deflection rate.

8. Deep dive 2: fanout, and what the queue is actually for

The obvious answer is “use a queue,” but what the queue is actually for is not throughput; it is isolation between channels. The backlog it accumulates, once priced, forces three properties the queue must have.

Why a synchronous fan-out is impossible

A synchronous fan-out is one that runs inside the caller’s HTTP request while the caller waits. Take the worst realistic case — one account with 10 million followers posts — and time it against the push fleet from 3a what each channel costs.

followers of one large account
  10,000,000
boxes in the push fleet that actually SEND (the other 24 are the token
store, the retry queue, and the receipt pipeline -- section 3a)      6
push rate the sender fleet sustains
  6 x 2,000                              =  12,000
seconds to fan out synchronously
  10,000,000 / 12,000                    =  833

The reflex is to divide by 30, which gives 167 seconds, and 167 seconds is wrong. It multiplies the whole fleet’s box count by a per-sender throughput, and 24 of those 30 boxes never open a connection to APNs at all.

The push fleet’s real capacity is 6 x 2,000 = 12,000/s — exactly what 3a what each channel costs solved the 6 for. So one large account takes 833 seconds, nearly fourteen minutes.

The correction does not weaken the conclusion; it strengthens it by a factor of five. The reflex is to “multiply the box count by the per-box rate,” but the box count here is a cost number, not a throughput number.

No HTTP request is held open for 833 seconds, so the fan-out has to be asynchronous behind a queue.

What the queue is actually for

The more precise version prices what the queue prevents when a provider slows down.

Little’s Law is the tool. The amount of work in flight at any moment equals the arrival rate multiplied by how long each item takes: in-flight = rate x latency. It is derived in ch 01. In flight means started and not yet finished, so the answer is exactly the number of concurrent slots you must have.

Apply it twice — once with a healthy provider, once with a degraded one — at the same peak arrival rate of 15,000/s.

provider latency, healthy, seconds
  0.050
in-flight sends needed at 15,000/s
  15,000 x 0.050                         =  750
provider latency, degraded, seconds
  2.000
in-flight sends needed at the same rate
  15,000 x 2.000                         =  30,000
ratio
  30,000 / 750                           =  40

A 40x latency degradation at APNs demands 40x the concurrency, instantly.

With a fixed pool of threads, that demand cannot be met, so it turns into blocking — threads sitting idle waiting for Apple. And those threads are shared with every other channel.

Without a queue, a slow APNs takes down email and SMS, which have nothing to do with APNs. That coupling is the real argument for the queue, and it is the one candidates miss.

What the queue turns the outage into

With a queue, the mismatch becomes a backlog instead of an outage. This is backpressure: when a downstream stage cannot keep up, the excess work accumulates in a bounded, visible buffer, so the pressure is felt as a growing queue rather than as failures propagating upstream.

Size that backlog. Push arrives at its peak rate while a bounded pool of 1,000 slots drains at the degraded 2-second latency, and the difference piles up.

push arrival rate at peak, per second
  12,000
drain rate with a 1,000-slot pool at 2 s each
  1,000 / 2.000                          =  500
net accumulation per second
  12,000 - 500                           =  11,500
backlog after 10 minutes
  11,500 x 600                           =  6,900,000
bytes at 500 B per queued message
  6,900,000 x 500                        =  3,450,000,000

6.9 million messages, 3.45 GB, after ten minutes of one provider being slow. Three properties of the queue follow directly from that number.

It must be durable and off-heap. Off-heap means written to disk in a separate process, rather than held in the sender’s own memory. An in-process buffer dies with the process and takes all 6.9 million messages with it, which is a worse incident than the one that created the backlog.

It must have a TTL and a drop policy. Notifications that are ten minutes stale are mostly worthless, and dumping all 6.9 million of them the instant the provider recovers is a second incident stacked on the first.

It must be partitioned by latency class. Otherwise those 6.9 million marketing messages sit in front of the next OTP.

The drop policy is a product decision with a per-category answer:

One related mechanism deserves naming here, even though the numbers above do not need it. A dead-letter queue is a separate queue that receives messages a worker could not process after its retries are exhausted. It exists for two reasons: a single poison message — a malformed payload, a template that always throws — cannot then block the partition behind it forever, and someone can inspect the failures later instead of losing them. Every queue in this design has one.

9. Deep dive 3: exactly-once is not available

Once-and-only-once delivery cannot be built here. The task is to prove that, choose which of the two available failure modes to accept, price both, and make the surviving failure invisible to the user.

The proof

Sending is two steps: call the provider, then record locally that you called it.

Those two steps live in different failure domains — your database and Apple’s — meaning either can fail without the other. Nothing spans them: no transaction, no coordinator, and no two-phase commit, which is the protocol where a coordinator asks every participant to prepare and promise it can finish, then tells them all to commit.

Nor can you repair the gap after the fact by asking. No provider offers a query interface for “did you deliver message X.” APNs has apns-collapse-id, but that collapses several notifications into one — it deliberately loses messages rather than deduplicating them, which is the opposite of what you want. FCM has no idempotency key at all.

So after a POST /3/device/{token} — the APNs send call — there are exactly three outcomes. Look at the third row: it is unresolvable by any amount of asking, and everything in this section is a consequence of that one cell.

Unregistered in the second row is APNs’s way of saying the app has been uninstalled, and reaping the token means deleting it so nothing is ever sent to it again.

The ambiguous third outcome is the whole problem: the provider may already have delivered the message when the acknowledgement is lost.

sequenceDiagram
    participant W as Worker
    participant P as APNs / FCM
    participant D as Device
    W->>P: send(notification_id)
    P->>D: deliver
    P--xW: ack lost (timeout)
    Note over W: outcome unknown:<br/>delivered or not
    W->>P: retry (duplicate)
    P->>D: deliver again
    Note over D: client dedup on notification_id<br/>drops the repeat

Choosing which failure to accept

Since the two steps cannot be atomic, all you get to choose is their order. There are two orders, and each picks a different poison.

• Commit then send gives you at-most-once delivery. Die in between, and the notification is marked sent although it never went. That is silent, permanent loss — invisible to every dashboard you have.
• Send then commit gives you at-least-once delivery. Die in between, and it is re-sent on recovery. That produces a duplicate, which is visible, annoying, and survivable.

Price both failure classes so the choice is a number rather than a preference.

The first source of duplicates is your own crashes. Every process death during the 50 ms window between the provider’s ack and your local commit re-sends whatever was in flight.

sends per day
  500,000,000
window between the provider ack and the local state commit, seconds   0.050
process deaths across the fleet per day, deploys plus crashes         10
in-flight sends lost per death, at the 15,000/s peak
  15,000 x 0.050                         =  750
duplicates per day from crashes
  750 x 10                               =  7,500

The second source is the ambiguous third row of the outcome table. A call times out, you retry, and half the time the provider had already delivered it (assumption B9).

provider calls that end ambiguously, at 0.1%
  500,000,000 x 0.001                    =  500,000
of those, the share the provider actually delivered
  0.50
duplicates per day from retrying ambiguous calls
  500,000 x 0.50                         =  250,000
duplicate rate
  250,000 / 500,000,000                  =  0.0005
ratio to the crash-window duplicates
  250,000 / 7,500                        =  33.3

One duplicate in 2,000, and 97% of them come from ambiguous timeouts rather than crashes. The share is 250,000 / (250,000 + 7,500) = 0.971.

That share is the useful part. It says tightening the crash window — a shorter commit path, more careful shutdown handling — buys you almost nothing, because it addresses the 3%. The duplicates are inherent to the protocol, not to your code quality.

Making the duplicate invisible

So: at-least-once on the wire, dedup at the client. Every notification carries its notification_id. The device keeps a bounded set of the ids it has already displayed, and drops repeats before rendering them.

Price the memory that costs on the device:

notification ids a client retains
  1,000
bytes per id: a Snowflake id is 64 bits (section 5)
  8
client memory
  1,000 x 8                              =  8,000

8 KB per device, and note the 8 in the middle line. Data model chose a Snowflake id over a UUID precisely because it is 8 bytes rather than 16, and this dedup set is the single largest consumer of that choice. Pricing the same id at 16 here would throw away the argument made two sections earlier.

Add a server-side dedup store as a second layer. It cannot collapse duplicates the provider created, because those happened outside your system entirely — but it does collapse the ones your own retry machinery creates.

Its TTL comes from the retry ladder. The retry ladder is the schedule of retry attempts — first after a second, then a few seconds, then a minute, and so on — and its span is the wall-clock time from the first attempt to the last, here 15 minutes.

retry ladder span, seconds
  900
dedup TTL, 4x the ladder
  900 x 4                                =  3,600
notifications inside a one-hour window
  5,000 x 3,600                          =  18,000,000
bytes per entry, 16 B key as a Redis string plus per-key overhead
  66
memory
  18,000,000 x 66                        =  1,188,000,000

1.2 GB in Redis — an in-memory key-value store with built-in per-key expiry — covers the whole retry horizon.

That TTL is derived from the retry ladder, not guessed. A TTL shorter than the ladder lets the last retry through as a duplicate. A TTL longer than the ladder is memory you are paying for with nothing to show for it.

The claim you are allowed to make

This is not exactly-once delivery. It is exactly-once as observed by the user, which is the property that actually matters.

Two cases it still does not cover:

• A user with two devices sees the notification twice, unless the dedup is server-side per user and per device.
• Reinstalling the app wipes the seen-set, so the first re-send after a reinstall duplicates.

Both are acceptable. Claiming they do not exist is not.

The mechanism in code

The code below is the argument in executable form. It has three pieces: a fake provider that delivers a message and then withholds the acknowledgement, a deliver function that sends before it commits, and a client inbox that drops repeats.

Watch the two assertions at the bottom. The provider genuinely receives n-2 three times — once per ambiguous attempt — and the user still sees three notifications rather than five. Note also that deliver records "unknown", never "failed", because after a timeout you do not know which it was.

"""At-least-once on the wire, dedup at the client."""
import collections


class ProviderTimeout(Exception):
    """The call may or may not have delivered. Nothing can tell you which."""


class FakeProvider:
    """Delivers, then withholds the ack for ids listed in lose_ack."""

    def __init__(self, lose_ack=()):
        self.delivered, self.lose_ack = [], set(lose_ack)

    def send(self, notification_id, token, body):
        self.delivered.append((notification_id, token, body))
        if notification_id in self.lose_ack:
            raise ProviderTimeout(notification_id)
        return "accepted"


def deliver(provider, outbox, rec, max_attempts=3):
    """Send first, commit second. The other order loses messages silently."""
    for _ in range(max_attempts):
        try:
            provider.send(rec["id"], rec["token"], rec["body"])
        except ProviderTimeout:
            continue                      # ambiguous: retry, accept a duplicate
        outbox[rec["id"]] = "sent"
        return "sent"
    outbox[rec["id"]] = "unknown"         # never "failed" -- you do not know
    return "unknown"


class ClientInbox:
    """A bounded set of ids the device has already rendered."""

    def __init__(self, capacity=1000):
        self.seen, self.capacity, self.rendered = collections.OrderedDict(), capacity, []

    def on_receive(self, notification_id, body):
        if notification_id in self.seen:
            return False                  # duplicate: drop before rendering
        self.seen[notification_id] = None
        if len(self.seen) > self.capacity:
            self.seen.popitem(last=False)
        self.rendered.append((notification_id, body))
        return True


provider, outbox = FakeProvider(lose_ack={"n-2"}), {}
for nid in ("n-1", "n-2", "n-3"):
    deliver(provider, outbox, {"id": nid, "token": "t", "body": "hi"})

# The provider really delivered n-2 three times, once per ambiguous attempt.
assert [d[0] for d in provider.delivered] == ["n-1", "n-2", "n-2", "n-2", "n-3"]
assert outbox["n-2"] == "unknown"

inbox = ClientInbox()
accepted = [inbox.on_receive(nid, body) for nid, _, body in provider.delivered]
assert accepted == [True, True, False, False, True]
assert [r[0] for r in inbox.rendered] == ["n-1", "n-2", "n-3"]   # user sees 3

10. Retries: cite the storm, do not re-derive it

Four retry rules arrive here already derived, and two more are specific to notifications: classify failures before scheduling them, and keep the ladder inside the notification’s expiry.

The usual framing of rate limiting has you imposing limits on your callers. Here it is the other way round. Toward APNs and Twilio you are the client — the one making requests of somebody else’s service — so the whole client contract from The client contract 429 retry after and the retry storm applies to you.

The four rules inherited from chapter 04

Exponential backoff with full jitter, uniform(0, min(cap, base x 2^n)). Three words there, each doing a job. Backoff means waiting longer after each successive failure. Exponential means the wait doubles each time. Full jitter means picking a random time uniformly between zero and that doubled ceiling, instead of using the ceiling itself.

Ch 04 derives why backoff without jitter fixes nothing. A thundering herd is a crowd of clients that all retry in the same instant. If 100,000 clients are rejected inside the same 10 ms window, they double their delays together and arrive together again — so the peak stays at 100x the design point no matter how long they wait.

Honor Retry-After, but never sleep exactly it. Retry-After is the response header in which a server states how long to wait. FCM returns 429 with Retry-After; APNs signals overload with 429 TooManyProviderTokenUpdates and by resetting streams. Obeying the header literally is what builds the herd, because every blocked sender received the identical value.

Retry at exactly one layer. Three tiers each retrying three times is 3 x 3 x 3 = 27 provider calls per notification during an incident, and the provider will rate-limit you for it.

Keep a retry budget, capped as a fraction of successes. With a 10% budget, a total provider outage produces 1.1x normal load rather than 4x, because the budget is replenished by successes and during an outage there are none.

The two rules specific to notifications

Not every failure is retryable, and the classification matters more than the schedule.

Retrying a permanent failure six times is how a fleet spends its whole retry budget on devices that no longer exist.

The retry ladder must fit inside the TTL. A 15-minute ladder under a 60-second OTP TTL means every attempt after the first is guaranteed to be dropped by the consumer before it is sent — which shows up in your metrics as provider failure, when it is really a config error. So the ladders are per category: an OTP gets 3 attempts inside 60 s, order_shipped gets 6 attempts over 15 minutes.

11. Rate limiting, fatigue, and the opt-out path

The most expensive failure in the system is sending too much, and its price is what justifies three controls: a per-category budget, quiet hours with a spread release, and an opt-out check placed where it cannot go stale.

Per-user rate limiting here is not an abuse control. Ch 04 covers that mechanism, and this is not what it is for.

It is a revenue control, and the arithmetic is the argument.

Notification fatigue is the effect being measured: each extra message slightly raises the chance the user switches the channel off for good.

The block below turns that effect into dollars. It walks from daily actives, through the share who get too many messages, to opt-outs per day, to annual value destroyed — then compares that against the SMS saving from Deep dive 1 cost drives routing.

DAU
  100,000,000
share receiving more than 3 notifications a day (B12)
  0.40
extra notifications those users receive (B12)
  2
opt-out probability added per extra notification, measured by holdout
  0.0005
incremental opt-outs per day
  100,000,000 x 0.40 x 2 x 0.0005        =  40,000
annual value of one live push subscription, dollars
  5
run-rate value destroyed in a year
  40,000 x 365 x 5                       =  73,000,000
against the SMS deflection saving from section 7
  73,000,000 / 13,687,500                =  5.33

Notification fatigue costs 5.3x more than the entire SMS optimization saves, and it appears on no infrastructure dashboard. An opt-out is permanent: you cannot win the user back with a notification, because that is the channel they just closed.

The number needs a caveat. The 0.40 and the 2 are assumption B12 in 2a the assumptions this design rests on, and neither is measured anywhere in this design. Together with B5 and B6 that is four soft factors multiplied in a row, so treat $73 M as an order of magnitude rather than a figure.

The 0.0005 is the one to be most careful about. It has to come from a holdout: an experiment in which a randomly chosen slice of users is deliberately not sent the campaign, so the opt-out rate of the sent group has something to be compared against. Without a holdout it is a guess, and the whole argument rests on it (assumption B6).

The per-category budget

The budget is enforced in the worker, and the cap depends on who asked for the message.

Quiet hours, and the spike they create

Quiet hours are a second, independent gate. They also create a load problem of your own making, because suppressing traffic overnight does not delete it — it stacks it up behind a deadline.

The block below stacks one timezone’s overnight traffic, then releases it three different ways: all at once, spread over an hour, and spread over four hours. The last line re-derives the design peak in exact seconds so the comparison is like for like.

quiet window, local, hours
  10
share of the clock unavailable
  10 / 24                                =  0.417
users in the largest single timezone, at 30% of DAU
  100,000,000 x 0.30                     =  30,000,000
their notifications suppressed overnight
  30,000,000 x 5 x 10 / 24               =  62,500,000
released at the boundary, inside one minute of timers firing
  62,500,000 / 60                        =  1,041,667
against the design peak of 15,000/s
  1,041,667 / 15,000                     =  69.4
released instead across the first hour
  62,500,000 / 3,600                     =  17,361
across the first four hours
  62,500,000 / 14,400                    =  4,340
the design peak re-derived in EXACT seconds, to compare like with like
  500,000,000 / 86,400 x 3               =  17,361

Releasing the overnight backlog at the boundary is a 69x spike over the system’s own peak, from one timezone alone — 1,041,667 notifications a second, when the whole system was built for 15,000.

The fix is the same uniform jitter ch 04 applies to retries. Set the release time to quiet_end + uniform(0, W), so each message picks its own random moment inside a window W wide.

Why a one-hour window is not enough

Read the one-hour result carefully, because the obvious reading of it is a units error.

The 15,000/s design peak was computed with the 1e5 shortcut from Back of the envelope. These release rates are in exact seconds. Put both in exact seconds — 500,000,000 / 86,400 x 3 = 17,361 — and the design peak is 17,361/s, which is precisely the one-hour spread rate.

Not 1.16x it. Equal to it, to the digit, because both are the same daily volume passed through the same 3,600-versus-86,400 relationship.

So W = 1 hour buys no margin at all. It converts a 60x overload into a 1.00x one — a fleet at exactly 100% utilization with a full night’s backlog still arriving. That is the zero-headroom-by-construction failure again, the same one 3a what each channel costs flagged on the push senders.

W = 4 hours gives 4,340/s, a quarter of the peak. It leaves room for the traffic that is not deferred, and it is the setting to ship.

One consequence for the data model: storing each user’s timezone is a hard requirement, not a nice-to-have. A user whose timezone is unknown gets the sender’s quiet hours rather than no quiet hours at all.

The opt-out check, and what a bug in it costs

Opt-out is the one control where a bug is priced in statutory damages. Statutory damages are penalties fixed by law per violation, owed whether or not anyone proves they were harmed.

Under the TCPA — the United States Telephone Consumer Protection Act — damages for an SMS sent to a number that replied STOP start at $500 per message. Price a small leak against it.

SMS per day
  25,000,000
share reaching a number that has replied STOP, at 0.01%
  25,000,000 x 0.0001                    =  2,500
statutory damages per message, low end, dollars
  500
exposure per day
  2,500 x 500                            =  1,250,000

A 0.01% opt-out leak is $1.25 M a day of exposure — 6.4x the entire daily notification bill of $195,144. Three design consequences follow.

The check runs in the worker, immediately before the provider call. A bulk fan-out computed twenty minutes ago has a stale view of consent, and twenty minutes is long enough for a user to have replied STOP.

Inbound STOP, UNSUBSCRIBE, and HELP messages are a write path into the preferences store, not a support ticket for a human to action.

The email unsubscribe link must be one click. It works on a plain GET request, and it is accompanied by the List-Unsubscribe header that mail clients turn into their own unsubscribe button. A flow that demands a login gets a spam complaint instead, and spam complaints are what get your sending domain blocked.

The budget and the release in code

The code below implements the two controls from this section: the per-category budget, and the quiet-hours release.

Its assertions pin two behaviours. A fourth social notification in a day is blocked, while transactional notifications never are. And a message suppressed at 3 a.m. comes out somewhere inside the morning spread window rather than exactly at its boundary.

Three further assertions exist because the obvious implementation is wrong in a way that shows up only off the default path: a quiet window that does not wrap midnight, a category the caller invented, and a u that is not a probability. All three are inputs a real caller supplies, and none of them is exotic.

import collections


class Budget:
    """Per-user, per-category caps. Transactional is never capped, because
    the user's own action requested it; everything else is."""

    CAPS = {"transactional": None, "social": 3, "marketing": 2}
    WINDOW_S = {"transactional": 0, "social": 86_400, "marketing": 604_800}
    DEFAULT = "marketing"       # an unrecognised category is capped as marketing

    def __init__(self):
        self.log = collections.defaultdict(list)

    def allow(self, user, category, now):
        # The API takes a free-form category string (section 4), so an unknown
        # one must have an ANSWER, not a KeyError. It falls to the strictest
        # cap and shares that bucket -- an invented name cannot mint a budget.
        if category not in self.CAPS:
            category = self.DEFAULT
        cap = self.CAPS[category]
        if cap is None:
            return True
        recent = [t for t in self.log[(user, category)]
                  if t > now - self.WINDOW_S[category]]
        self.log[(user, category)] = recent
        if len(recent) >= cap:
            return False
        recent.append(now)
        return True


def release_at(hour_local, quiet_start=22, quiet_end=8, spread_h=1.0, u=0.5):
    """Quiet traffic is released across a spread window `spread_h` hours wide.
    Releasing it AT the boundary is a self-inflicted thundering herd."""
    if not 0.0 <= u <= 1.0:
        raise ValueError("u is a draw from uniform(0, 1)")
    if quiet_start < quiet_end:      # a window inside one day, e.g. 02:00-10:00
        quiet = quiet_start <= hour_local < quiet_end
    else:                            # a window that wraps midnight, e.g. 22-08
        quiet = hour_local >= quiet_start or hour_local < quiet_end
    return quiet_end + u * spread_h if quiet else hour_local


b = Budget()
assert all(b.allow("u1", "social", t) for t in (0, 10, 20))
assert not b.allow("u1", "social", 30)                 # 4th in a day: blocked
assert b.allow("u1", "social", 86_401)                 # window rolled
assert all(b.allow("u1", "transactional", t) for t in range(50))
# A category the caller invented is capped, never crashed on.
assert [b.allow("u1", "promo_v2", t) for t in (0, 1, 2)] == [True, True, False]
assert release_at(14) == 14                            # daytime: send now
assert release_at(3, u=0.0) == 8 and release_at(3, u=1.0) == 9
assert release_at(23, u=0.5) == 8.5                    # spread, not stacked
assert release_at(3, u=1.0, spread_h=4.0) == 12        # the four-hour window
# A quiet window that does NOT wrap midnight must not defer a midday message
# into a time that has already passed.
assert release_at(12, quiet_start=2, quiet_end=10) == 12
assert release_at(3, quiet_start=2, quiet_end=10, u=0.0) == 10
try:                                                   # u is a probability
    release_at(3, u=5.0)
except ValueError:
    pass
else:
    raise AssertionError("u outside uniform(0, 1) must be rejected")

Three notes on those guards, because each is a decision rather than defensive padding.

The midnight-wrap test matters most in production. 22:00-08:00 wraps past midnight; 02:00-10:00 does not. Both are quiet hours a user can legitimately set, and a single comparison chain cannot express both. The naive form treats every window as wrapping, so a message sent at noon under a 02:00-10:00 window is judged quiet and deferred to 10.5 — a moment nine and a half hours in the past.

The u check turns a documented contract into an enforced one. u is a draw from uniform(0, 1). An unvalidated 5.0 pushes the release time past midnight into the following day, silently, with no error anywhere.

The Budget default is the one to justify out loud, because this is the gate whose bug is priced in statutory damages. An unrecognised category must not raise, must not default to uncapped, and must not get a private budget of its own — a caller who invents a name would otherwise mint themselves a fresh allowance. It falls to the marketing cap of two a week and shares marketing’s bucket, which is the only one of the three options that fails safe.

12. Templates, localization, and the emoji that costs $13.7 M

Three things about templates matter here: the entire corpus is small enough to hold in memory, rendering late rather than early is the decision that counts, and a single character in an SMS template can cost as much as the whole routing optimization saves.

A template is the message with holes in it — “Hi {name}, order {id} has shipped”. Rendering is filling those holes for a specific user.

Start with how much space all the templates take, in every language, together.

templates
  500
locales
  40
rendered variants
  500 x 40                               =  20,000
bytes each
  2,000
total
  20,000 x 2,000                         =  40,000,000

40 MB. The entire template corpus therefore lives in every sender process, and a template lookup is never an RPC — a remote procedure call, a network round trip to another service dressed up as a function call. That closes the storage question in one line, which is the correct amount of interview time to spend on it.

Render late, not early

The decision that is not obvious is when to render. Render at fanout time and the queue carries finished strings. Render at send time and it carries a small reference — template id, user id, params — that the worker expands.

Compare what each choice puts in the queue per day.

rendered payload, bytes
  500
reference payload: template id, user id, params, bytes
  120
queue bytes per day if rendered at fanout
  500,000,000 x 500                      =  250,000,000,000
queue bytes per day as references
  500,000,000 x 120                      =  60,000,000,000
saving
  250,000,000,000 - 60,000,000,000       =  190,000,000,000

190 GB a day of queue traffic saved is the small reason.

The large reason is that a template fix applies to the backlog. Deep dive 2 fanout and what the queue is actually for sized a plausible backlog at 6.9 million messages. Render late, and a typo or a broken link discovered mid-incident can still be corrected for every one of them. Render early, and those 6.9 million strings are already written and unfixable.

Localization is not string interpolation

Localization is adapting the message to the user’s language and region. Three properties make it more than substitution.

The locale belongs to the user, not to the request. A locale is a language-and-region tag such as pt-BR for Brazilian Portuguese. The order-shipped event knows nothing about what language its recipient reads.

Plural and gender rules differ per language. The right tool is ICU MessageFormat — the International Components for Unicode message syntax, which encodes rules like “one file / 2 files / few files” per language — rather than %s substitution plus an if statement that only handles English.

The fallback chain is pt-BR -> pt -> en, never blank. If the Brazilian Portuguese string is missing, fall back to generic Portuguese, then to English.

One related rule: the template’s params_schema is validated at POST time (Api sketch). A missing parameter is then a 422 returned to the caller, rather than a notification that reads “Hi , your order”.

The SMS encoding trap

Now a real line on a real bill. An SMS segment is the billing unit of SMS: a message longer than one segment is split into parts and charged per part.

Two different things in this chapter are called a segment, and they appear in tables a few pages apart. Requirements defines an audience segment — a saved set of users a campaign targets. That is the one in POST /v1/notifications/bulk and in Deep dive 2 fanout and what the queue is actually for’s 10-million-user chunking argument. This section’s SMS segment is 160 characters of billed message and has nothing to do with users at all.

The collision is the industry’s, not this chapter’s, and the only defence is to say which one every time: audience segment or SMS segment, never the bare noun.

The block below prices what one emoji does to a 150-character message. Watch the third line: adding a single character takes the message from one billed segment to three.

GSM-7 characters per segment
  160
UCS-2 characters per segment
  70
a 150-character body with one emoji, segments
  3
extra segments per such message
  3 - 1                                  =  2
messages carrying a non-GSM-7 character, at 10%
  25,000,000 x 0.10                      =  2,500,000
extra cost per day
  2,500,000 x 2 / 1,000,000 x 7,500      =  37,500
extra cost per year
  37,500 x 365                           =  13,687,500

One emoji in one SMS template costs exactly what the entire fallback ladder in Deep dive 1 cost drives routing saves.

Here is the mechanism, step by step. GSM-7 is a 7-bit alphabet, so 160 of its characters fit in one segment. A single character outside that alphabet switches the whole message to UCS-2, the 16-bit Unicode encoding, which fits only 70 characters per segment — and only 67 once the segments carry the headers that let a phone reassemble them in order. A 150-character message that was one segment becomes ceil(152 / 67) = 3 segments. The carrier bills per segment, so the message triples in price.

A curly apostrophe pasted out of a word processor does this just as effectively as an emoji does.

So the segment count is a lint rule — an automated check that rejects the change rather than merely warning about it — enforced in CI, the continuous-integration pipeline that runs on every commit, for every SMS template. The template editor also shows the count live while someone types.

The lint rule needs the full alphabet, or it is itself a bug

GSM-7 is not ASCII. The GSM 03.38 basic table includes £ ¥ § ¤ ¡ ¿, the Greek capitals that have no Latin lookalike, and twenty-odd accented Latin letters: è é ù ì ò à ä ö ü ñ å Ä Ö Ü Ñ É Å Æ æ ß Ç Ø ø. The € sits in the extension table alongside the braces.

An ASCII-only approximation errs in the safe direction — it over-reports segments rather than under-charging — but it errs precisely on European currency symbols and accented templates. segments("£" + 159 more characters) answers 3 where the carrier bills 1, and a CI rule wired to that rejects a perfectly cheap German or French template.

In a section whose neighbour is about localizing across 40 locales, that is not a rounding error. It is the lint rule failing exactly the locales that localization exists for.

Counting segments the way a carrier does

The code below implements the count. Four things to watch in its assertions: 150 plain characters are one SMS segment, the same 150 plus one emoji are three, a 10% emoji share across 25 million daily messages costs exactly $37,500 a day, and the accented and currency characters that are GSM-7 are billed as GSM-7.

One line in segments is easy to skim past. len(body.encode("utf-16-le")) // 2 counts UCS-2 code units, not Python characters, which is why one emoji costs two of the 70 rather than one.

GSM7 = set("@£$¥èéùìòÇ\nØø\rÅåΔ_ΦΓΛΩΠΨΣΘΞÆæßÉ"       # GSM 03.38 basic table
           " !\"#¤%&'()*+,-./0123456789:;<=>?"
           "¡ABCDEFGHIJKLMNOPQRSTUVWXYZÄÖÑܧ"
           "¿abcdefghijklmnopqrstuvwxyzäöñüà")
GSM7_EXT = set("^{}\\[~]|€")                 # these cost two characters each


def segments(body):
    """One character outside GSM-7 drops the WHOLE message to UCS-2, from
    160 characters per segment to 70 (67 when concatenated)."""
    if all(c in GSM7 or c in GSM7_EXT for c in body):
        n = len(body) + sum(c in GSM7_EXT for c in body)
        return 1 if n <= 160 else -(-n // 153)
    n = len(body.encode("utf-16-le")) // 2   # UCS-2 code units, not codepoints
    return 1 if n <= 70 else -(-n // 67)


def daily_sms_cost(volume, emoji_share, price_per_msg=0.0075, body_len=150):
    plain = segments("a" * body_len)
    fancy = segments("a" * body_len + "\U0001f600")
    return volume * ((1 - emoji_share) * plain + emoji_share * fancy) \
        * price_per_msg


assert segments("a" * 150) == 1
assert segments("a" * 150 + "\U0001f600") == 3      # one emoji, three segments
assert segments("a" * 161) == 2
assert segments("cost: 5 {euros}") == 1             # {} are GSM-7 extended
# The false rejection an ASCII-only alphabet produces: these ARE GSM-7, and a
# carrier bills every one of them as a single segment.
assert segments("£" + "a" * 159) == 1               # £ is in the basic table
assert segments("bestätigt " * 16) == 1             # so are ä ö ü é à ñ ...
assert segments("€" + "a" * 158) == 1               # € is extended: 2 chars
assert round(daily_sms_cost(25_000_000, 0.10)
             - daily_sms_cost(25_000_000, 0.0)) == 37_500

13. Tracking, and why “delivered” is a lie

Most of what a notification dashboard reports is not measuring what its label claims — so audit the metrics one by one, then replace the broken measurement with one that works.

Three kinds of event get reported to every product team — sent, delivered, opened — and only one of them is real.

The table below takes each event, states what it actually measures, and rates it. Read the middle column, not the label in the first.

Two entries need unpacking.

An SMSC is a short message service centre, the carrier’s own store-and-forward node. A DLR that means “the SMSC accepted it” says nothing about whether the handset ever saw the message.

A tracking pixel is a 1x1 transparent image embedded in an email. Downloading it tells the sender the message was opened — which only holds if downloading the image implies a human looked.

Why the open rate is inflated, numerically

The email case is worth doing with numbers, because it is the one people still put in board decks.

Apple Mail Privacy Protection (MPP) is on by default in Apple Mail. It pre-fetches every remote image in every message, regardless of whether the human opened anything. So every Apple Mail recipient registers as an open.

Split the base into the Apple half, which reports 100% opens, and the rest, which reports the true rate.

true open rate, measurable only on non-Apple clients
  0.25
Apple Mail share of the recipient base
  0.55
reported open rate under pixel pre-fetching
  0.55 x 1.00 + 0.45 x 0.25              =  0.6625
inflation factor
  0.6625 / 0.25                          =  2.65

A 66.25% headline open rate against a real 25%: inflated 2.65x. Worse, the inflation moves whenever Apple’s share moves, so a year-over-year comparison of open rates is measuring device market share rather than campaign quality.

Report the non-Apple cohort and its coverage, or do not report opens.

The metric that does work

The fix for delivery is to stop asking the provider and ask the device.

The client posts a receipt when it renders the notification. It goes over the same channel that already carries the dedup key from Deep dive 3 exactly once is not available, so it costs nothing extra to add.

3b storage and the tracking load priced that receipt stream at 320 M a day and 3,200 writes a second. It is one of the three events per notification, inside a total event load of 15,000 writes a second — 3x the 5,000/s send rate.

That gives a true rendered metric for the part of the population that is able to report. Publish the coverage of that population next to the rate, because “80% of reporting devices rendered it” and “80% of notifications were delivered” are different claims, and only the first one is supported.

Dead tokens, the other silent corruption

Every uninstalled app leaves behind a token you will keep pushing to forever. At a 1% monthly uninstall rate, compound over a year: 0.99 ^ 12 is the share still alive, so 1 - 0.99 ^ 12 is the share dead.

registered devices
  200,000,000
monthly uninstall rate
  0.01
share of tokens dead after 12 months without reaping
  1 - 0.99 ^ 12                          =  0.114
pushes per day aimed at devices that no longer exist
  400,000,000 x 0.114                    =  45,600,000

11.4% of push volume goes nowhere, and every rate computed against sent is therefore 11% wrong — because the denominator includes 45.6 million daily pushes that could never have arrived.

Reaping — deleting a token the provider has declared dead — is not housekeeping. When APNs returns 410 Unregistered or FCM returns NotRegistered, delete the token immediately, in the same code path that got the response.

There is a second class of dead token that no provider will ever tell you about: an app the user simply stopped opening. That is why the fallback ladder in Deep dive 1 cost drives routing checks “seen in 30 days” rather than “token exists.”

14. Bottlenecks and scaling

Every resource the system consumes appears below with the number that bounds it, which makes the one axis that actually matters visible against the eight that do not.

The rows are ordered by how much they should worry you. The first row is the design’s real constraint; the last two are there so you can say why they are not constraints and move on.

One row uses a term worth naming. Approximate counting means each node keeps its own per-user counter and syncs it periodically, instead of coordinating with every other node on every send. The cap is then enforced within a small error — which is fine here, because the cap itself (“three social notifications a day”) is a product heuristic with a far wider error bar than the counter’s.

The scaling axis people reach for is sender throughput. The axis that matters is the queue partition count.

A partition is one independently-consumed slice of a queue. A single partition is drained in order, so a category’s entire latency budget is decided by how much of the previous category is sitting in front of it. Adding senders does not fix that; adding partitions does.

15. Failure modes

This design fails in production in nine ways, each row giving the trace an operator would actually see, the signal that catches it, and the guard that prevents it.

Read down the Detection column specifically. Almost none of these are caught by an error rate — they are caught by a ratio, a per-category percentile, or a count of something that should be zero. A system whose only alarm is “errors went up” catches none of the nine.

Two guards in that table use terms worth restating.

To circuit-break at a threshold is to stop calling a failing provider entirely for a cooling-off period, once its error rate crosses a line. The point is to make sure your own retries are not what keeps it down.

A bounded pool per provider means each provider gets its own fixed allocation of concurrent slots. Exhausting Apple’s allocation then cannot consume the slots email is using — which is the coupling Deep dive 2 fanout and what the queue is actually for priced.

16. Alternatives rejected

Eight designs a candidate is likely to propose, each rejected on a number or a named impossibility, never on taste — what is genuinely good about the alternative first, then the figure that kills it.

Synchronous send inside the caller’s request. Good: the caller learns the outcome immediately. Rejected because the outcome is not knowable (Deep dive 3 exactly once is not available) and because an 833-second fan-out cannot live in an HTTP request. The caller gets 202 and a notification_id, and looks it up if it cares.

One queue for everything. Good: one thing to operate. Rejected because a marketing backlog then sits in front of a 60-second OTP; queue depth is shared, so the cheapest, least urgent traffic sets the latency of the most urgent. Partition by latency class, not by channel — the channel is already separated by the worker pool.

Exactly-once via a distributed transaction with the provider. Good: it would be genuinely better. Rejected because it does not exist: no two-phase commit is on offer, neither APNs nor FCM accepts an idempotency key, and neither provides a way to ask afterwards what happened to an ambiguous send. The alternative is not a worse protocol, it is an imaginary one.

Server-side dedup only, no client dedup. Good: nothing to ship in the app. Rejected because the duplicates that dominate — 250,000/day against 7,500 — are created outside your system by ambiguous provider calls, and no server-side store can see them. The client is the only place that observes the actual delivery.

Retry until success on every failure. Good: maximum delivery. Rejected on two counts: Unregistered and PayloadTooLarge are permanent, so retrying spends the budget on garbage; and retries without a budget cap turn a provider outage into a 4x self-amplified load (ch 04).

Render templates at fan-out time. Good: the worker gets to be simple, and the processing cost is paid once per notification either way. Rejected because it puts 250 GB/day of rendered strings in the queue instead of 60 GB, and — the real reason — a template fix can no longer reach a 6.9 M-message backlog.

Store tracking events in the same database as the outbox. Good: one store, joins are free. Rejected on shape and on blast radius — how far a single failure spreads: 27.4 TB/year of append-only analytical rows against a small, frequently-updated operational table, and one heavy analytics query that saturates the store also stops the sending. Events go to a columnar store off a stream (sql/03 on why the storage layout differs).

A single global per-user rate limiter with strict consistency. Good: exact caps. Rejected because the cap is a product heuristic with an error bar far wider than the limiter’s — approximate counting with periodic sync (ch 04) is free and indistinguishable in outcome. Spend strictness on the opt-out check, which has statutory damages attached, not on “3 per day.”

17. Interviewer pushback

Six questions, each with the answer and what it is really testing. If a paragraph here does not make sense, the section it compresses is the one to reread.

“Can you guarantee exactly-once delivery?” Testing: whether you will overclaim.

No, and nobody can. The send crosses into APNs, which shares no transaction with my database and offers no idempotency key and no query API for “did you deliver this.” After a timeout there are two possibilities and no way to distinguish them, so I choose which error I want: commit-then-send loses messages silently and permanently, send-then-commit duplicates them visibly. I take duplicates.

Then I make them invisible to the user: every notification carries its id, the client keeps the last thousand ids in 8 KB and drops repeats, and a Redis set with a TTL of 4x the retry ladder collapses the ones my own retries create. The rate is about one in 2,000, and 97% of those come from ambiguous provider calls rather than my crashes — which tells me that hardening my shutdown path buys almost nothing and the client dedup buys everything.

What I get is exactly-once as observed by the user. Two holes I would name before you do: two devices, and a reinstall that wipes the seen-set.

“Why not just send everything over SMS? It always works.” Testing: whether you have priced the channels.

Because at my volume push costs $0.36 per million loaded with the fleet, email costs $100, and SMS costs $7,500. Today SMS is 5% of my messages and 96% of my $195,144 daily bill. Moving everything to SMS would be about $3.75 M a day, or $1.4 B a year.

The design that falls out is a fallback ladder: push first if there is a token seen in the last 30 days, escalate only on evidence — no client receipt within five minutes — and escalate to SMS only for transactional categories. Twenty percent deflection is $13.7 M a year.

And I would flag the second-order one: a single emoji in an SMS template flips the encoding to UCS-2 and takes a 150-character message from one SMS segment to three, which at a 10% template share is $37,500 a day. Same order as the entire routing optimization, and it is a lint rule.

“APNs starts taking two seconds per call. Walk me through what happens.” Testing: whether you know why the queue is there.

Little’s Law: in-flight work is rate times latency, so at 15,000/s the concurrency demand goes from 750 to 30,000 — 40x, instantly. If sends are synchronous on a shared thread pool, the pool fills, and the threads that fill it were also serving email and SMS, so a problem entirely inside Apple takes down two channels that have no dependency on Apple.

With a per-channel queue it becomes a backlog instead: 12,000/s arriving against 500/s draining is 11,500/s of accumulation, 6.9 million messages and 3.45 GB after ten minutes.

That forces three things. The queue is durable and off-heap, because losing 6.9 million messages to a process restart is worse than the original incident. Every message has a TTL, because delivering a ten-minute-old “your ride is here” is worse than dropping it, and because dumping all 6.9 million the instant Apple recovers is a second outage. And the queues are split by latency class, so the marketing backlog is not sitting in front of the next OTP.

“Your dashboard says 99% delivered. Is it?” Testing: whether you audit your own metrics.

Almost certainly not, and I would not have shipped that number. For push, APNs never tells me about handset delivery at all — a 200 means “accepted,” so a “delivered” metric on push is measuring my own API call. For SMS, the carrier DLR means handset receipt on some routes, SMSC acceptance on others, and on international aggregator routes it is sometimes synthesized. For email, “delivered” means the receiving MTA accepted the transaction, which is compatible with going straight to spam.

The one I would show the room is email opens: with Apple Mail Privacy Protection pre-fetching every pixel, a 25% real open rate on the measurable half of the base reports as 66.25% — inflated 2.65x, and the inflation tracks Apple’s market share rather than anything about the campaign.

What I do instead is have the client post a receipt when it renders, over the same channel that already carries the dedup id. That is 320 million receipts a day, 3,200 writes a second — and with three events per notification the whole event stream is 15,000 writes a second against 5,000 sends, so tracking writes at 3x the rate of the thing it tracks. A larger system than the send path, which is why it is in the architecture diagram. And I publish the coverage of the reporting population next to the rate, because 80% receipts and 80% delivery are different claims.

“How do you stop bombarding users?” Testing: whether fatigue is a number or a vibe.

With a per-category budget enforced in the worker, and I would justify the caps with the cost of not having them. If 40% of my 100 million daily actives get two notifications past the third, and each marginal notification adds 5 basis points of opt-out probability — which is measurable with a holdout — that is 40,000 opt-outs a day. At $5 a year of incremental value per live push subscription, a year of that destroys $73 M of run rate, which is 5.3x what the entire SMS routing optimization saves. Opt-outs are also irreversible, because the channel you would use to win them back is the one they closed.

So: transactional uncapped, social three a day, marketing two a week and every campaign carries a holdout arm.

Quiet hours are a separate gate, and they create their own problem — suppressing 22:00 to 08:00 stacks 62.5 million messages for my largest timezone, and releasing them all at 08:00 is 1,041,667/s. I spread the release with the same uniform jitter I use for retries — and I would spread it over four hours, not one, because in consistent seconds my design peak is 500 M / 86,400 x 3 = 17,361/s and a one-hour spread is 17,361/s exactly, which is 100% utilization rather than any margin at all.

“Where does the opt-out check go, and why does it matter?” Testing: whether you know which control is legally load-bearing.

In the channel worker, in the last few milliseconds before the provider call. Not in the fanout service, because a bulk fanout over a 10 million-user audience segment takes minutes and consent can change during it, and not in the API, because the API runs before any of that.

The reason it is worth the extra lookup on the hot path is the price of being wrong: TCPA statutory damages start at $500 per SMS to a number that replied STOP, so a 0.01% leak against 25 million SMS a day is 2,500 messages and $1.25 M of daily exposure — 6.4x my entire daily notification bill.

That also makes STOP, UNSUBSCRIBE and HELP a write path into the preferences store rather than something support handles, and it makes the email unsubscribe a one-click endpoint plus List-Unsubscribe, because a flow that requires a login gets a spam complaint instead.

Cheat sheet

Every result in the chapter, one line each, in the order you would say them at a whiteboard. If you can reconstruct the derivation behind each right-hand cell, you are ready for the round.

Related: 04 — Rate Limiter owns backoff, jitter, and the retry storm this chapter cites; 07 — Unique ID Generator supplies the notification_id that is also the dedup key; 09 — Web Crawler is the other chapter where politeness toward someone else’s server is the binding constraint.