November 3, 2026 · from Distributed AI Inference
How idle threads burned two thirds of my inference capacity
Model workers ran at a third of their measured speed inside containers. The CPU was busy waiting, not computing.
Called directly, the embedding model handled about 55 texts per second in its container. Inside the HTTP worker, under load, it managed 17. Same model, same container, same 2-CPU limit.
Following the CPU
The container's cgroup statistics showed it was throttled in about 3 of every 4 scheduling periods. Per-thread CPU time showed two ONNX Runtime threads doing almost all the work, about three times more CPU per text than in the direct test.
ONNX Runtime's thread pool spin-waits after each call, ready for the next one. Under a container CPU quota that spinning is billed like real work, so the worker spent its quota waiting and was then throttled when real work arrived.
Two fixes
- Turn spinning off (session.intra_op.allow_spinning = 0). The library used to load the model didn't expose the option, so the worker sets it while the model is constructed.
- Encode responses with orjson straight from the numpy array. The framework's generic JSON encoder was costing about 13 ms of CPU per request on the event-loop thread.
The lesson is to profile inside the real environment. Neither problem shows up in a notebook.