Latency and Throughput

Fast and high-volume are different properties

When discussing performance, people often say, “the system must be fast.” But that phrase hides at least two different properties — latency and throughput. If you mix them up, you can optimize the wrong problem.

Latency: how long one request waits

Latency is the delay from the start of an operation until its result is received. For example, a user clicks a button, the request goes to the server, and the response arrives 80 ms later. For that request, latency is approximately 80 ms.

Request ──────────────→ Response
        <--- 80 ms --->

In a real system, latency is made up of several parts: network time, waiting in queues, application work, storage calls, and other operations. That is why speeding up one component does not always noticeably improve the final response time.

Throughput: how much work passes through the system

Throughput is the amount of work a system processes per unit of time. For an API, this is often requests per second; for a queue, messages per second; for storage, read or write operations per second.

1 second
| Req | Req | Req | Req | Req | Req |
          throughput = 6 req/s

High throughput does not guarantee low latency. A system can process thousands of requests per second while an individual user still waits noticeably longer for a response.

An average can hide the problem

If most requests complete in 40 ms but a small portion takes several seconds, a simple average may not reflect the real user experience. That is why latency is often viewed through percentiles: for example, p95 means that 95% of measured requests completed no slower than the stated value.

p50 = 45 ms
p95 = 180 ms
p99 = 700 ms

This picture immediately reveals the “long tail”: almost all requests are fast, but a small share performs much worse.

Optimization depends on the goal

For an interactive interface, latency is usually especially noticeable. For background processing of millions of events, throughput may matter more. Often both properties are important, but the architectural decisions used to improve them can differ.