Latency vs Throughput

Two distinct performance dimensions. Latency is the time to complete a single operation, end to end, measured in percentiles (p50, p99, p999), not averages. Throughput is the rate of operations the system sustains: requests/sec, MB/sec, messages/sec. They are related but independent: a highway's speed limit is latency; its number of lanes is throughput.

Forces you to be precise about what 'fast' means. A user staring at a spinner cares about latency. A batch pipeline processing a billion rows overnight cares about throughput. Conflating them leads to the wrong optimization: batching improves throughput but adds latency; tiny per-request work minimizes latency but wastes throughput.

Latency is reduced by caching, doing less work per request, moving computation closer to the user (CDN), and avoiding serial round-trips. Throughput is increased by parallelism, batching, pipelining, and adding nodes (horizontal scaling). The tension: batching N requests amortizes fixed costs (great for throughput) but each request now waits for the batch to fill (worse latency). Little's Law ties them together: concurrency = throughput × latency.

• Measuring both separately tells you which optimization actually helps your users
• Throughput optimizations (batching, pipelining) are often cheap capacity wins
• Latency optimizations (caching, CDN) directly improve perceived user experience

• They trade off: batching for throughput adds latency; minimizing latency wastes throughput on per-request overhead
• Averages lie about latency, so you must track tail percentiles (p99/p999) because that's what users feel
• Optimizing the wrong metric burns engineering time without moving the number users care about

• Stock Exchange (Matching Engine) →

  Microsecond latency is the entire product; they sacrifice almost everything else to shave tail latency off order matching

• Apache Kafka →

  Built for throughput: batches and pipelines records, accepting some per-message latency to push millions of messages/sec

• Netflix →

  CDN edges cut video start latency; the encoding pipeline is throughput-optimized batch work where latency doesn't matter

• Never report latency as an average. p99 and p999 are what users actually feel; averages hide the tail where everyone gets angry.
• Batching is the classic latency-for-throughput trade. Tune batch size and max-wait to the SLA, and don't make latency-sensitive requests wait for a batch to fill.
• Little's Law (concurrency = throughput × latency) is the cheat sheet: if latency rises and throughput is flat, in-flight requests are piling up, which is where queues back up and systems fall over.
• Higher throughput doesn't imply lower latency. A system can be saturated (max throughput) while every individual request crawls.