To sum up: Variance is the enemy of performance and the source of much of the latency we encounter when using software.
To keep latency to a minimum:
- As a rule of thumb, target utilization below 75%,
- Steer slower workloads to paths with lower utilization,
- Limit variance as much as possible when utilization is high,
- Implement backpressure in systems where it is not built-in,
- Use throttling and load shedding to reduce pressure on downstream queues.