The P95 Problem
Median latency is marketing; the tail is what your users experience.
- Measured p95/p50 TTFT ratios span 1.5× to 7.9× across providers.
- Together shows the heaviest measured tail: 309 ms median vs 2426 ms p95.
- Cerebras serves the tightest distribution — its p95 stays within 1.5× of median.
Distributions, not points
Two providers with similar medians can ship completely different user experiences: one in twenty requests lives at p95, and for interactive products that request defines perceived quality. The ratio, not the median, is the engineering number.
| Provider | p50 | p95 | p95/p50 |
|---|---|---|---|
| Together | 309 ms | 2426 ms | 7.9× |
| Scaleway | 183 ms | 690 ms | 3.8× |
| Mistral | 386 ms | 1087 ms | 2.8× |
| OpenAI | 1021 ms | 2359 ms | 2.3× |
| Groq | 283 ms | 418 ms | 1.5× |
| Cerebras | 251 ms | 365 ms | 1.5× |
- Set SLOs on p95, benchmark on p95, and treat median-only comparisons as incomplete by construction.
InferenceBench (2026). The P95 Problem. InferenceBench Research.