PERFORMANCELlama 3.3 70B → Scaleway: throughput ↑ 53%6hPERFORMANCEGLM-5.2 → Scaleway: TTFT ↓ 30%6hPERFORMANCEMistral Medium 3.5 → Scaleway: throughput ↑ 22%6hPERFORMANCEQwen3-235B → Scaleway: throughput ↑ 67%6hPERFORMANCEGPT-5.6 Sol → OpenAI: TTFT ↓ 22%6hPERFORMANCEMistral Large 3 → Mistral AI: TTFT ↓ 21%6hPERFORMANCEGPT-5.6 Luna → OpenAI: throughput ↑ 21%6hPERFORMANCEDeepSeek V4 Flash → Scaleway: throughput ↑ 147%6hPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↓ 71%6hPERFORMANCEGPT-OSS 120B → Scaleway: TTFT ↓ 46%6hPERFORMANCEGPT-OSS 120B → Together AI: TTFT ↓ 40%6hPERFORMANCEKimi K3 → Together AI: TTFT ↓ 20%6hPERFORMANCEDeepSeek V4 Flash → Scaleway: throughput ↑ 36%30hPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↓ 19%30hPERFORMANCEQwen3-235B → Scaleway: throughput ↑ 19%2dPERFORMANCEDeepSeek V4 Flash → Scaleway: throughput ↑ 58%2dPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↑ 21%2dPERFORMANCEGPT-5.6 Sol → OpenAI: throughput ↑ 16%2dPERFORMANCEGPT-OSS 120B → Together AI: TTFT ↑ 22%2dPERFORMANCEKimi K3 → Together AI: throughput ↑ 62%2dPERFORMANCEKimi K3 → Together AI: TTFT ↓ 42%2dPERFORMANCEQwen3-235B → Scaleway: throughput ↑ 30%3dPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↓ 56%3dPERFORMANCELlama 3.3 70B → Scaleway: throughput ↑ 34%3dPERFORMANCELlama 3.3 70B → Scaleway: TTFT ↓ 26%3dPERFORMANCEKimi K3 → Together AI: throughput ↑ 128%3dPERFORMANCEGPT-OSS 120B → Together AI: throughput ↑ 29%3dPERFORMANCEGPT-OSS 120B → Together AI: TTFT ↓ 21%3dPERFORMANCEGPT-OSS 20B → Groq: TTFT ↑ 66%3dPERFORMANCEGLM-5.2 → Together AI: throughput ↑ 47%3dPERFORMANCEMistral Medium 3.5 → Scaleway: TTFT ↑ 18%4dPERFORMANCEGLM-5.2 → Scaleway: throughput ↓ 27%4dPERFORMANCEGLM-5.2 → Scaleway: TTFT ↑ 49%4dPERFORMANCEDeepSeek V4 Flash → Scaleway: throughput ↓ 71%4dPERFORMANCEDeepSeek V4 Flash → Scaleway: TTFT ↑ 190%4dPERFORMANCEGPT-5.6 Luna → OpenAI: throughput ↓ 23%4dPERFORMANCELlama 3.3 70B → Scaleway: TTFT ↑ 60%4dPERFORMANCEGLM-5.2 → Together AI: TTFT ↓ 25%4dPERFORMANCEGPT-OSS 120B → Together AI: throughput ↓ 16%4dPERFORMANCEGPT-OSS 120B → Together AI: TTFT ↓ 17%4d
MARKET REPORT

The State of AI Inference — August 2026

The first fully measured snapshot: who leads, what a frontier API buys, and how far apart identical models sit.

Key findings
  • Cerebras leads the measured provider ranking at 99.0 under the General profile.
  • On GPT-OSS 120B — the widest-covered model (4 providers) — identical weights span 161–291 ms median TTFT, while tool-calling reliability from 0.0% to 100.0%.
  • 2 of 6 measured providers sit on the price–performance frontier; every other provider is dominated on both axes at once.

The measured ranking

Aggregated across every measured model with coverage weighting and freshness shading, Cerebras holds #1 at 99.0, ahead of Mistral (99.0) and Groq (95.6). The gap between #1 and the median measured provider is 7.4 points — the market is not close.

Frontier APIs enter the table measured like everyone else, under the same request shape, the same exclusion rules and the same window statistics as every open-weight serving specialist.

889296100$4.37$8.74$13.10$17.47Output price $/M →Score ↑Cerebras — $0.75/M out · score 99.0CerebrasMistral — $4.50/M out · score 99.0Groq — $0.45/M out · score 95.6Together — $4.40/M out · score 91.6OpenAI — $15.60/M out · score 88.4Scaleway — $1.81/M out · score 85.6
Score vs $/M output across measured providers · the frontier is where no provider is better and cheaper at once · ● MEASURED

Identical weights, different products

GPT-OSS 120B is served by 5 measured providers. The same weights produce median TTFT from 161 to 291 ms (1.8×) and output price from $0.60 to $0.75 per million depending on whose infrastructure executes them — tool-calling reliability from 0.0% to 100.0% across the set. Infrastructure is not a commodity layer — it is the product.

Scaleway161 msTogether201 msCerebras251 msGroq291 ms
Median TTFT on GPT-OSS 120B across measured providers — identical weights, best first · ● MEASURED
What this means for engineers
  • Model selection and provider selection are separate decisions of comparable weight — benchmark both.
  • Frontier pricing is not a proxy for serving quality; measure the path you intend to ship.
Dataset

Figures recompute from the live rolling window on every visit; the snapshot and versions above are the provenance of the regime that produced them. View the current benchmark →

Cite this researchInferenceBench (2026). The State of AI Inference — August 2026. InferenceBench Research.
← All research
Related research
The State of AI Inference — August 2026 · InferenceBench