Book Notes / Systems Performance: Enterprise and the Cloud (2nd Edition)

Brendan Gregg / Reading notes

A slow operation can be waiting in different ways

Start with the operation whose completion matters and the workload under which it is slow. Request rate alone is insufficient: request size, concurrency, cache state, and access pattern change the work performed. A performance claim needs a defined population and a defined interval before its measurements can be compared.
An operating-system thread can be running on a CPU, runnable but waiting for one, or blocked until some event makes it runnable again. These states imply different interventions. Expensive execution suggests less computation; runnable delay suggests a scheduling or available-capacity constraint; blocking points toward a dependency such as I/O, a lock, or another thread.
Leaving a CPU does not establish which of those constraints applies. A thread can block voluntarily or be descheduled while still ready to run. Even a large amount of blocked time may be harmless: a worker awaiting its next request is idle by design. Attribute the waiting to the operation being investigated.

A CPU profile samples execution. The largest stack may identify expensive computation, but it does not account for a request's entire elapsed time. Optimizing that stack can have little effect when the request spends most of its time waiting.
Off-CPU stacks locate where threads stopped running. Scheduler timing distinguishes the delay after a thread becomes runnable from the delay before it can proceed. The stack, the state transition, and the duration answer different questions; none alone establishes the complete cause.
Resource measurements supply another view. Utilization asks how much a resource is used; saturation asks whether demand is waiting for service; errors expose failed work. A busy resource need not delay this operation, and an apparently idle machine can still contain its constraint. One fully occupied core on an eight-core machine produces only 12.5% aggregate CPU use.
Connect the two views: show that the operation waits during the interval when the relevant resource cannot serve it. If a counter moves without a corresponding path to the delayed work, it remains a candidate explanation. Missing instrumentation leaves that part of the explanation unresolved.

Consider a hypothetical serial request that spends 20 ms executing and 80 ms waiting for a required dependency. If the dependency time stays fixed, doubling execution speed reduces completion time from 100 ms to 90 ms. Eliminating execution entirely still leaves 80 ms. The largest execution hotspot and the largest opportunity to shorten the request need not coincide.
Parallel requests require the dependency graph as well as the durations. If two necessary branches take 20 ms and 80 ms concurrently, completion takes at least 80 ms. Halving the short branch saves no elapsed time under those conditions. Halving the long branch reduces the bound to 40 ms. Summing overlapping spans would conceal which change can matter.
Those bounds are conditional. Speeding up one stage can change arrival pressure at another, and removing one bottleneck can expose the next. Re-measure the original operation after the intervention; the first diagnosis is not a permanent property of the system.

Before tuning, state what should change if the explanation is right. If CPU scheduling delay dominates, making more CPU time available should reduce runnable delay under the same workload. It need not reduce time blocked on an external service. If blocking remains unchanged while completion improves by the expected amount, the measurements support that account.
Inspect the result alongside alternative explanations. A quieter load, a warmer cache, or fewer completed requests can improve a latency number without validating the tuning change. Keep workload and measurement boundaries comparable, and repeat enough to distinguish the effect from ordinary variation.
Count failures and timeouts alongside successful completions. A service can improve the latency distribution of its successes by failing its slow requests earlier. That changes which requests remain in the measured population; it does not establish that users received a faster service.