The problem
A marketing campaign needed a real-time inference service to hold up in front of a live audience, at a scale the existing one could not reach. Single-request inference was taking roughly 2.8 seconds, and p50 latency climbed with concurrency until the experience stopped being real-time in any sense a user would recognise.
The instinct in the room, mine included, was that a GPU workload that cannot serve users needs a bigger GPU, or more of them. That instinct had a price tag attached, which is the only reason it was worth being careful about.
Three explanations that did not survive
Rather than argue about it, I built a sweep: seven configurations, one variable moved at a time, each one designed so that a specific explanation would produce a specific number if it were true.
Three of the hypotheses I went in with were falsified by it, including the expensive one. I want to be precise about what that means: they were not merely unsupported, they predicted results the sweep did not produce, and I threw them out. The value of the exercise was almost entirely in what it killed.
What survived was an observation the sweep kept reproducing and I kept trying to explain away:
Under normal load, GPU utilisation sat between 0% and 13%, reaching 83% only at 150 users. Raw inference was 8 ms per model. Around each of those 8 ms sat roughly 2,400 Python function calls of preprocessing and postprocessing.
That ratio is the diagnosis. The accelerator was doing a rounding error's worth of the wall-clock time, and the work wrapped around it could only ever run one thread at a time. Buying a bigger GPU makes the 8 ms smaller. It does nothing to the rest.
The diagnosis
The service was Python, and the ceiling was the Global Interpreter Lock. The GPU work was genuinely fast and genuinely parallel; the Python-level work wrapped around it (request handling, pre- and post-processing, serialisation, the orchestration between them) could only ever run one thread at a time. So concurrency did not queue on the accelerator, it queued on the interpreter, and every additional user made the line longer while leaving the GPU exactly as idle as it had been.
This is why the utilisation figure was the whole diagnosis and not a supporting detail. It is the single number that distinguishes "we need more compute" from "we cannot reach the compute we already have", and those two readings point at opposite budgets.
A second, independent finding came out of the same instrumentation.
The realtime transport was fanning every message out to every
connected client: a broadcast=True on a path that only
ever needed to reach one. That makes per-message cost scale with the
number of people connected, which is the worst possible shape for a
system whose entire goal is to hold more people. Removing it took
p50 from 871 ms to 59 ms on its own.
What it moved
| Before | After | |
|---|---|---|
|
Concurrent users (load test) GCP 1× L4, one process → AWS 4× L40S, 48 containers |
~120 | 500 |
|
Best-tuned density (load test) same 4× L40S, 80 containers |
n/a | 700 |
| Inference | 2,800 ms | 60 ms |
| p50, realtime path | 871 ms | 59 ms |
The comfortable ceiling is 500 concurrent at 97.9% delivery and 259 ms p50, about $0.015 per user-hour. The best-tuned configuration reached 700 at 99.4% and 132 ms, at $0.0106 per user-hour, but 800 collapses to 87.7%, so 500 is the number I would plan against.
That 500 to 700 is worth separating out, because no hardware changed. It is the same box and the same four L40Ses, run at a different container density: 48 containers gives 700 at 98.6% and 245 ms, 64 gives 94.4%, and 80 gives 99.4% at 132 ms. Same GPUs, same code, better packing.
Being precise about where the capacity came from, because the before and after are not the same machine. The baseline was a GCP L4 running one Python process behind 32 gunicorn workers: about 120 users at 99% delivery, 871–1215 ms p50. The result is an AWS box with four L40Ses.
The chip is the smallest part of it. Holding the fixed architecture constant and varying only the GPU, one L4 carries about 150 users and one L40S about 200, which is 1.78× the throughput (84 to 150 fps) and well under the 3× the spec sheets imply for this workload. Going from one L40S to four accounts for a 2.5× jump. The rest is the code: before the fan-out and pinning fixes, adding hardware did not help at all, because the constraint was a single Python process, not the GPU.
What I will not claim about this
500 is proved capacity, not served load. It is a load-test figure, and production never rose to meet it. I am confident about the ceiling because the test was built to find it, but a load generator is not an audience: it has no think time, no unusual clients, no pathological network, and none of the ways real traffic is creative. The honest statement is that we proved headroom and then never needed it. Part of that headroom was bought with three extra GPUs, and part of the baseline gap is an L4 against an L40S rather than anything I wrote: at constant code that swap alone is worth roughly 150 users to 200. The hardware was necessary: a single L40S with more vCPU and more RAM was tested at 500 users and delivered 8.2%. It was not sufficient either, which is the whole point of the case: four GPUs behind one Python process would have idled the way one did. And the last stretch, 500 to 700, came from container density on unchanged hardware.
The cost figure carries the same asterisk: $0.0106 per user-hour is the campaign configuration at campaign scale, and it would not survive being quoted as a general unit economic for the platform.
And the thing I would keep from this over any of the numbers: the first three explanations were mine, they were confident, and one of them would have been expensive. What separated the right answer from the wrong ones was not insight, it was that the sweep was designed so a wrong hypothesis would visibly fail rather than quietly go unchallenged. GPU utilisation was on the dashboard the entire time. It just took building the experiment to make me believe it.