PRICE~moonshotai/kimi-latest cached_input_per_mtok decreased from 0.80 to 0.292mPRICE~moonshotai/kimi-latest output_per_mtok decreased from 13.00 to 11.362mPRICE~moonshotai/kimi-latest input_per_mtok decreased from 0.99 to 0.752mPRICE~deepseek/deepseek-pro-latest cached_input_per_mtok increased from 0.0042 to 0.102mPRICE~deepseek/deepseek-pro-latest output_per_mtok increased from 0.40 to 4.202mPRICE~deepseek/deepseek-pro-latest input_per_mtok decreased from 0.13 to 0.132mPRICE~moonshotai/kimi-latest cached_input_per_mtok increased from 0.29 to 0.8033mPRICE~moonshotai/kimi-latest output_per_mtok increased from 11.36 to 13.0033mPRICE~moonshotai/kimi-latest input_per_mtok increased from 0.75 to 0.9933mPERFORMANCEGLM-5.2 → Scaleway: TTFT ↑ 20%1hPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↑ 35%1hPERFORMANCEMistral Medium 3.5 → Scaleway: TTFT ↓ 21%1hPERFORMANCEDeepSeek V4 Flash → Scaleway: throughput ↓ 20%1hPERFORMANCEGPT-OSS 120B → Scaleway: throughput ↑ 31%1hPERFORMANCEGPT-OSS 120B → Scaleway: TTFT ↓ 24%1hPERFORMANCEGPT-OSS 120B → Together AI: throughput ↑ 20%1hPERFORMANCEKimi K3 via OpenRouter: TTFT ↓ 49%1hPERFORMANCEGLM-5.2 via OpenRouter: reliability recovered 96.8% → 100.0%1hPERFORMANCEKimi K3 via Cortecs: throughput ↑ 184%1hPERFORMANCEKimi K3 via Cortecs: TTFT ↓ 42%1hPERFORMANCEGLM-5.2 via OpenRouter: throughput ↓ 49%1hPERFORMANCEGLM-5.2 via OpenRouter: TTFT ↑ 121%1hPERFORMANCEMiniMax M3 via OpenRouter: reliability recovered 96.8% → 100.0%1hPERFORMANCEDeepSeek V4 Pro via Cortecs: throughput ↑ 27%1hPERFORMANCEGPT-OSS 120B via OpenRouter: throughput ↓ 22%1hPERFORMANCEGLM-4.7 via OpenRouter: TTFT ↓ 78%1hPERFORMANCEGPT-OSS 20B → Groq: TTFT ↑ 24%1hPERFORMANCEKimi K3 → Together AI: TTFT ↓ 34%1hPERFORMANCEDeepSeek V4 Flash via OpenRouter: throughput ↓ 17%1hPERFORMANCELlama 3.3 70B via Cortecs: TTFT ↓ 51%1hPERFORMANCEQwen3 235B via OpenRouter: throughput ↑ 37%1hPERFORMANCEQwen3 235B via OpenRouter: TTFT ↓ 29%1hPERFORMANCEDeepSeek V4 Flash → Together AI: throughput ↑ 19%1hPERFORMANCEGLM-5.3 via OpenRouter: throughput ↓ 17%1hPERFORMANCEGPT-OSS 20B via Cortecs: throughput ↑ 28%1hPERFORMANCELlama 3.3 70B → Together AI: TTFT ↓ 33%1hPRICE~moonshotai/kimi-latest cached_input_per_mtok decreased from 0.80 to 0.291hPRICE~moonshotai/kimi-latest output_per_mtok decreased from 13.00 to 11.361hPRICE~moonshotai/kimi-latest input_per_mtok decreased from 0.99 to 0.751hPRICE~deepseek/deepseek-v4-flash-latest cached_input_per_mtok decreased from 0.0077 to 0.00111h
ENGINEERING STUDY

The P95 Problem

Median latency is marketing; the tail is what your users experience.

Aug 22, 20269,095 benchmark runs8 providers15 modelsMethodology v0.13-dev● MEASURED
Key findings
  • Measured p95/p50 TTFT ratios span 1.5× to 7.2× across providers.
  • Scaleway shows the heaviest measured tail: 172 ms median vs 1234 ms p95.
  • Groq serves the tightest distribution — its p95 stays within 1.5× of median.

Distributions, not points

Two providers with similar medians can ship completely different user experiences: one in twenty requests lives at p95, and for interactive products that request defines perceived quality. The ratio, not the median, is the engineering number.

Providerp50p95p95/p50
Scaleway172 ms1234 ms7.2×
OpenRouter656 ms3027 ms4.6×
Mistral307 ms1009 ms3.3×
Cortecs425 ms1254 ms3.0×
Together572 ms1625 ms2.8×
OpenAI1040 ms2453 ms2.4×
Cerebras184 ms323 ms1.8×
Groq262 ms384 ms1.5×
Measured TTFT distribution by provider (aggregated paths) · ● MEASURED
What this means for engineers
  • Set SLOs on p95, benchmark on p95, and treat median-only comparisons as incomplete by construction.
Dataset
9,095 results7-day rolling window · canonical vantage eu-falkensteinsnapshot 2026-08-24-eu-falkensteinMethodology v0.13-dev● MEASURED

Figures recompute from the live rolling window on every visit; the snapshot and versions above are the provenance of the regime that produced them. View the current benchmark →

Cite this researchInferenceBench (2026). The P95 Problem. InferenceBench Research.
← All research
Related research
ENGINEERING STUDY

Tool Calling Is an Infrastructure Property

Agentic workloads live or die on tool-call reliability — and measurement shows it is set as much by serving infrastructure as by the model.

Aug 22, 2026ENGINEERING STUDY

Structured Output: Trust, but Validate

Machine-readable output is a contract. This table is the measured conformance rate per provider — spread or uniformity, whichever the window shows.

Aug 22, 2026MARKET REPORT

The State of AI Inference — August 2026

The flagship market snapshot, computed exclusively from independent measurements taken from the canonical European vantage: the provider ranking, the price–performance frontier, and the spread between infrastructures serving identical models.

Aug 22, 2026
The P95 Problem · InferenceBench