PERFORMANCELlama 3.3 70B → Scaleway: throughput ↑ 53%6hPERFORMANCEGLM-5.2 → Scaleway: TTFT ↓ 30%6hPERFORMANCEMistral Medium 3.5 → Scaleway: throughput ↑ 22%6hPERFORMANCEQwen3-235B → Scaleway: throughput ↑ 67%6hPERFORMANCEGPT-5.6 Sol → OpenAI: TTFT ↓ 22%6hPERFORMANCEMistral Large 3 → Mistral AI: TTFT ↓ 21%6hPERFORMANCEGPT-5.6 Luna → OpenAI: throughput ↑ 21%6hPERFORMANCEDeepSeek V4 Flash → Scaleway: throughput ↑ 147%6hPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↓ 71%6hPERFORMANCEGPT-OSS 120B → Scaleway: TTFT ↓ 46%6hPERFORMANCEGPT-OSS 120B → Together AI: TTFT ↓ 40%6hPERFORMANCEKimi K3 → Together AI: TTFT ↓ 20%6hPERFORMANCEDeepSeek V4 Flash → Scaleway: throughput ↑ 36%30hPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↓ 19%30hPERFORMANCEQwen3-235B → Scaleway: throughput ↑ 19%2dPERFORMANCEDeepSeek V4 Flash → Scaleway: throughput ↑ 58%2dPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↑ 21%2dPERFORMANCEGPT-5.6 Sol → OpenAI: throughput ↑ 16%2dPERFORMANCEGPT-OSS 120B → Together AI: TTFT ↑ 22%2dPERFORMANCEKimi K3 → Together AI: throughput ↑ 62%2dPERFORMANCEKimi K3 → Together AI: TTFT ↓ 42%2dPERFORMANCEQwen3-235B → Scaleway: throughput ↑ 30%3dPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↓ 56%3dPERFORMANCELlama 3.3 70B → Scaleway: throughput ↑ 34%3dPERFORMANCELlama 3.3 70B → Scaleway: TTFT ↓ 26%3dPERFORMANCEKimi K3 → Together AI: throughput ↑ 128%3dPERFORMANCEGPT-OSS 120B → Together AI: throughput ↑ 29%3dPERFORMANCEGPT-OSS 120B → Together AI: TTFT ↓ 21%3dPERFORMANCEGPT-OSS 20B → Groq: TTFT ↑ 66%3dPERFORMANCEGLM-5.2 → Together AI: throughput ↑ 47%3dPERFORMANCEMistral Medium 3.5 → Scaleway: TTFT ↑ 18%4dPERFORMANCEGLM-5.2 → Scaleway: throughput ↓ 27%4dPERFORMANCEGLM-5.2 → Scaleway: TTFT ↑ 49%4dPERFORMANCEDeepSeek V4 Flash → Scaleway: throughput ↓ 71%4dPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↑ 190%4dPERFORMANCEGPT-5.6 Luna → OpenAI: throughput ↓ 23%4dPERFORMANCELlama 3.3 70B → Scaleway: TTFT ↑ 60%4dPERFORMANCEGLM-5.2 → Together AI: TTFT ↓ 25%4dPERFORMANCEGPT-OSS 120B → Together AI: throughput ↓ 16%4dPERFORMANCEGPT-OSS 120B → Together AI: TTFT ↓ 17%4d
ENGINEERING STUDY

The P95 Problem

Median latency is marketing; the tail is what your users experience.

Key findings
  • Measured p95/p50 TTFT ratios span 1.5× to 7.9× across providers.
  • Together shows the heaviest measured tail: 309 ms median vs 2426 ms p95.
  • Cerebras serves the tightest distribution — its p95 stays within 1.5× of median.

Distributions, not points

Two providers with similar medians can ship completely different user experiences: one in twenty requests lives at p95, and for interactive products that request defines perceived quality. The ratio, not the median, is the engineering number.

Providerp50p95p95/p50
Together309 ms2426 ms7.9×
Scaleway183 ms690 ms3.8×
Mistral386 ms1087 ms2.8×
OpenAI1021 ms2359 ms2.3×
Groq283 ms418 ms1.5×
Cerebras251 ms365 ms1.5×
Measured TTFT distribution by provider (aggregated paths) · ● MEASURED
What this means for engineers
  • Set SLOs on p95, benchmark on p95, and treat median-only comparisons as incomplete by construction.
Dataset

Figures recompute from the live rolling window on every visit; the snapshot and versions above are the provenance of the regime that produced them. View the current benchmark →

Cite this researchInferenceBench (2026). The P95 Problem. InferenceBench Research.
← All research
Related research
The P95 Problem · InferenceBench