In any stable system, the average number of items inside it is the arrival rate multiplied by the time each item spends there:
L = λ × W
- L: items in progress at the same time (requests, orders, people in line).
- λ (lambda): arrival rate (requests per second).
- W: average time each item spends in the system.
Example: a bakery serves 2 customers per minute and each one stays inside for 5 minutes. On average, there are 10 customers in the bakery.
In servers: if 1,000 requests arrive per second and each takes 200 ms, there are 200 requests in flight. If each request holds a thread, you need 200 threads. To handle twice the traffic with the same response time, you need twice the threads.
That is why, when response time is dominated by waiting (database, network), the throughput limit is usually how many concurrent tasks the system can hold, not the CPU. It is the core argument for Java virtual threads.
The law holds on average and for stable systems (the queue doesn’t grow forever), and it does not depend on the arrival distribution.
- To dig deeper: Little’s law (Wikipedia)