Finding 4-7x throughput in a pipeline running at 15% CPU

The situation

A production forecasting pipeline was throttled to about 75 work items per hour while the machines ran at 15% CPU and tens of thousands of dollars of LLM provider capacity sat more than 85% idle. The maddening part: latency didn't correlate with load. The same concurrency level produced a p90 of 6 seconds in one window and 146 seconds in another. Every obvious dial had already been turned: raising the concurrency cap had provably done nothing.

Ruling things out, with numbers

We ran a structured investigation: five independent investigation tracks, three adversarial verification passes cross-checked against the platform's observability data and the client's own latency measurements. The goal was not just to find the cause but to disprove the plausible wrong answers, because each of them had an expensive "fix" attached:

• Not thread-pool starvation. Worst-case queue math put that delay at 5.4 seconds against 130 seconds observed, off by 24x.
• Not the concurrency cap. Little's Law put effective concurrency at ~13 against a cap of 40. Raising it had been inert, and now we knew why.
• Not HTTP connection pools, DNS, the GIL, the database, or provider rate limits. Each ruled out with a measurement, not a hunch.

The real problem: synchronized waves

Each work item makes three LLM calls in sequence. Staggered, each call returns in 7 to 40 seconds. But when a dozen items hit the same model endpoint within a second of each other, the whole wave queues server-side and collapses to the 120-second timeout. The system wasn't slow; it was phase-locked. That's why latency ignored load: what mattered was whether requests arrived staggered or synchronized.

Four mechanisms were manufacturing the waves: a timeout cascade that released batches of work simultaneously, a dispatch stagger far too small to break phase alignment, worker instances churning (five restarts in twelve hours, each one re-releasing all in-flight work at once), and retry amplification that logged 442 fallbacks and 2,729 timeouts on a single instance.

The fix

One reversible pull request, ranked smallest blast-radius first: a diagnostic span shipped ahead of everything to confirm the analysis in production, an explicitly pinned thread pool, parallelized database writes so post-model persistence stopped re-batching items into waves, a shorter per-request timeout to truncate stalls, per-provider concurrency caps sized to the measured wave, randomized dispatch jitter, and a minimum-instances setting to stop the worker churn.

The result

A quantified path from 75 items per hour to a realistic 330 to 540 (theoretical ceiling 700 to 860), with no new infrastructure, no bigger machines, and no additional provider spend. The most valuable deliverable was arguably the ruled-out list: the client stopped planning capacity purchases that would not have helped.