BENCHMARKFirst benchmarks recorded: MiniMax M3 → Cortecs → ?3hBENCHMARKFirst benchmarks recorded: Kimi K3 → Cortecs → ?3hBENCHMARKFirst benchmarks recorded: DeepSeek V4 Pro → Cortecs → ?4hBENCHMARKFirst benchmarks recorded: GLM-4.7 → Cortecs → ?4hBENCHMARKFirst benchmarks recorded: GLM-5.2 → Cortecs → ?4h
ENGINEERING STUDY

Tool Calling Is an Infrastructure Problem Too

Identical models disagree on tools depending on who serves them.

Aug 15, 2026103,921 benchmark runs13 providers8 modelsMethodology v0.10-devDEV DATA
Key findings
  • Qwen3-235B tool success spans 94.7%–99.2% (4.5 points) across 12 providers under the Agents workload.
  • Llama 3.3 70B shows the same pattern: a 4.7-point spread on identical weights.
  • At agent chain lengths of 10+ calls, the per-call gap compounds into materially different task-completion rates.

Same weights, different agents

A 2-point per-call difference sounds small. Across a 10-call agent chain it is the difference between an 82% and a 90% chance of a flawless run — visible directly in production error budgets.

Fireworks99.2OpenRouter99.0Cerebras98.8Nebius98.1Groq97.5Together96.9Nextbit96.7Scaleway96.6DeepInfra95.7Mistral95.7SiliconFlow95.2OVHcloud94.7
Tool-call success by provider · Agents workload · DEV DATA
What this means for engineers
  • Agent teams should benchmark tool success per provider, not per model, and re-verify after provider-side serving updates.
Dataset
103,921 runs30-day rolling window · Aug 2026snapshot 2026-08-15-v1Methodology v0.10-devDEV DATA

Published figures are pinned to this snapshot and are not recalculated retrospectively. View the current benchmark →

Cite this researchInferenceBench (2026). Tool Calling Is an Infrastructure Problem Too. InferenceBench Research.
← All research
Related research
ENGINEERING STUDY

Same Model, Different Infrastructure: How Large Is the Gap?

The core InferenceBench thesis, measured: for the same model, TTFT varies by multiples, throughput by an order of magnitude, and tool reliability by whole points.

Aug 15, 2026ENGINEERING STUDY

P95 Matters: The Hidden Tail Latency of AI Providers

Tail analysis across the provider pool: p95/p50 ratios, who degrades under the tail, and why medians alone are insufficient for interactive workloads.

Aug 15, 2026ENGINEERING STUDY

Structured Output Reliability Across Providers

JSON validity and schema compliance measured across the provider pool: who enforces structure, and how large the gap is on identical models.

Aug 15, 2026