P95 Matters: The Hidden Tail Latency of AI Providers
Median latency hides the requests your users actually complain about.
- 12 of 13 providers (92%) show p95 TTFT above 2× their median.
- The widest tail in the pool: Nebius at 3.7× (p50 288 ms → p95 1061 ms).
- The tightest: Cortecs at 1.9× — proof that a narrow tail is an engineering choice, not luck.
The ratio table
p95/p50 is the honest tail signal: it is scale-free, so a fast and a slow provider can be compared directly on how much they degrade.
| Provider | p50 TTFT | p95 TTFT | Ratio |
|---|---|---|---|
| Nebius | 288 ms | 1061 ms | 3.68× |
| Together | 401 ms | 1452 ms | 3.62× |
| OVHcloud | 372 ms | 1345 ms | 3.62× |
| Nextbit | 341 ms | 1158 ms | 3.40× |
| Cerebras | 148 ms | 501 ms | 3.39× |
| SiliconFlow | 587 ms | 1986 ms | 3.38× |
| Mistral | 298 ms | 963 ms | 3.23× |
| Groq | 179 ms | 530 ms | 2.96× |
| Fireworks | 312 ms | 918 ms | 2.94× |
| OpenRouter | 384 ms | 1106 ms | 2.88× |
| Scaleway | 356 ms | 1025 ms | 2.88× |
| DeepInfra | 512 ms | 1337 ms | 2.61× |
| Cortecs | 525 ms | 1003 ms | 1.91× |
- For interactive workloads, set SLOs on p95, not p50 — and pick providers with ratios near the pool's best.
- A low median with a wide tail often indicates aggressive batching; verify under your own concurrency.
InferenceBench (2026). P95 Matters: The Hidden Tail Latency of AI Providers. InferenceBench Research.