Design a Notification System
Push, email and SMS at scale. Provider failover, preference enforcement, deduplication and the digest logic that stops you spamming your own users.
Covers: Multi-channel delivery, templating, user preferences, deduplication, rate limiting, retries, third-party providers
Notifications look trivial and are not. The delivery is easy; the hard parts are respecting user preferences correctly, not sending the same thing twice, surviving a provider outage, and rate limiting so a runaway loop does not email a user four hundred times.
Notification pipeline
One entry point, a preference and dedup gate, then per-channel workers with independent retry behaviour.
Flow
- Event sourcesNotification API
- Notification APIPreference & dedup gate
- Preference & dedup gatePer-channel queues
- Per-channel queuesChannel workers
- Channel workersProviders
- Channel workersDelivery log
30-second answer
Route every notification through a single gate that owns both concerns. Deduplicate on a caller-supplied idempotency key stored with a TTL, so retries and duplicate events collapse into one send. Enforce preferences as a layered check: global opt-out, then per-category, then per-channel, then quiet hours in the user's own time zone, then a frequency cap. Critical transactional messages - password resets, security alerts, payment failures - bypass marketing preferences but must still be deduplicated and rate limited.
def should_send(user, notif) -> tuple[bool, str]:
# 1. Hard opt-out beats everything except legally required messages.
if user.unsubscribed_all and notif.category != "transactional":
return False, "global_opt_out"
# 2. Category preference (marketing / social / product / security).
if not user.prefs.category_enabled(notif.category):
return False, "category_disabled"
# 3. Channel preference within the category.
if notif.channel not in user.prefs.channels_for(notif.category):
return False, "channel_disabled"
# 4. Quiet hours - in the USER's timezone, not the server's.
if notif.category != "critical" and user.prefs.in_quiet_hours(now_in(user.tz)):
return False, "quiet_hours:defer" # defer, do not drop
# 5. Frequency cap.
if rate_limiter.count(user.id, notif.category, window="1h") >= user.prefs.hourly_cap:
return False, "rate_limited:digest" # roll into a digest instead
# 6. Deduplicate last: it is the only check that mutates state.
if not dedup.claim(notif.idempotency_key, ttl=86_400):
return False, "duplicate"
return True, "ok"| Category | Respects opt-out | Respects quiet hours | Rate limited |
|---|---|---|---|
| Marketing | Yes | Yes | Aggressively |
| Social (likes, comments) | Yes | Yes | Yes - digest above threshold |
| Product updates | Yes | Yes | Yes |
| Transactional (receipt, shipping) | No | Yes | Light |
| Security (login, password reset) | No | No | Light, but must not be blocked |
Say these points
- A single gate owns preferences, quiet hours, frequency caps and deduplication.
- Deduplicate on a caller-supplied idempotency key with a TTL.
- Quiet hours use the user's time zone, and defer rather than drop.
- Security and transactional messages bypass marketing preferences, not deduplication.
- Only the notification service holds provider credentials - no bypass path exists.
Avoid these mistakes
- Dropping notifications that arrive during quiet hours.
- Quiet hours evaluated in server time.
- Services sending directly and bypassing preference logic.
- Rate limiting security alerts into oblivion.
Expect these follow-ups
- A bug enqueues the same notification 400 times. What stops the user seeing 400 pushes?
- How do you handle a user who changes time zone mid-flight?
- How would you design a daily digest that is not just a dump of everything?
30-second answer
Each channel needs at least two providers with automatic failover driven by a circuit breaker on error rate, not just on hard failures. Retries must distinguish transient errors - timeouts, 5xx, provider rate limits - which are retried with exponential backoff and jitter, from permanent errors like an invalid address or an unregistered device token, which must not be retried at all and should update the user's contact record. Queue depth grows during an outage, which is exactly what the queue is for, but you need a maximum age after which time-sensitive notifications are dropped rather than delivered uselessly late.
| Provider response | Class | Action |
|---|---|---|
| Timeout / connection error | Transient | Retry with backoff; count towards the circuit breaker |
| 429 rate limited | Transient | Retry honouring Retry-After; slow the worker |
| 5xx | Transient | Retry with backoff; failover if the rate is sustained |
| 400 malformed | Permanent | Do not retry - this is a bug; alert |
| Invalid recipient / bounce | Permanent | Do not retry; mark the address invalid |
| Unregistered device token | Permanent | Delete the token; the app was uninstalled |
| Spam complaint | Permanent | Suppress that address globally and immediately |
PROVIDERS = {"email": [ses, sendgrid], "sms": [twilio, messagebird]}
def send(channel, message):
last_error = None
for provider in PROVIDERS[channel]:
if breakers[provider].is_open():
continue # skip a provider we know is failing
try:
result = provider.send(message)
breakers[provider].record_success()
return result
except PermanentError as e:
suppression.add(message.recipient, reason=str(e))
raise # never retry, never failover
except TransientError as e:
breakers[provider].record_failure()
last_error = e
continue # try the next provider immediately
raise AllProvidersFailed(last_error) # requeue with backoffOrdering and priority
- Separate queues per priority, not a priority field on one queue - otherwise a marketing backlog delays password resets.
- Separate queues per channel, so a slow SMS provider cannot block push notifications.
- Cap the retry budget per notification - typically 3–5 attempts over a few hours - then dead-letter it.
- Alert on the dead-letter queue, because it is where genuinely broken things accumulate silently.
Say these points
- Two providers per channel with circuit-breaker-driven failover.
- Classify errors: transient retries with backoff, permanent never retries.
- Hard bounces and complaints go to a permanent suppression list.
- Attach a max age; drop stale notifications rather than delivering them late.
- Separate queues per channel and per priority.
Avoid these mistakes
- Retrying hard bounces and destroying sender reputation.
- One queue for all channels and priorities.
- Delivering a two-hour-old 'your driver is here' after an outage.
- No dead-letter queue or no alerting on it.
Expect these follow-ups
- How do you test failover without sending real notifications?
- How would you throttle a 10-million-recipient campaign so it does not starve transactional traffic?
- How do you measure whether a notification was actually useful?
Showing 2 of 2 questions for design-a-notification-system.
Check your understanding
3 questions · no sign-up, nothing stored