Designing for scale, lessons from 10M+ events a day

Scale rarely breaks you with a single dramatic failure. It breaks you with a slow accumulation of small assumptions that were true at a thousand events a day and quietly false at ten million.

Idempotency is not optional

At volume, every message will be delivered twice eventually. Design every consumer so that processing the same event twice produces the same result. A deterministic idempotency key derived from the event payload, stored with a TTL, removes an entire category of incident.

Backpressure beats bigger machines

When a downstream dependency slows down, the instinct is to scale consumers up. That usually makes things worse by increasing pressure on the thing that is already struggling. Bounded queues, concurrency limits per dependency, and circuit breakers keep a slow dependency from becoming a total outage.

Make the boring path observable

• Track queue lag as a first-class SLO, not CPU.
• Emit per-stage timing so you can see where a pipeline degrades before customers do.
• Alert on rate of change, not just thresholds.

Simplicity compounds

The systems that survived longest in my career were not the cleverest ones. They were the ones a new engineer could reason about at 3am. Every piece of accidental complexity is a tax paid by everyone who touches the system after you.